Using Bosh to Bootstrap Cloud Foundry via Stark & Wayne Consulting

I finally sat down and really started to take a stab at Cloud Foundry Bosh. Here’s the quick lowdown on installing the necessary bits and getting an initial environment built. Big thanks out to Dr Nic @drnic, Luke Bakken & Brain McClain @brianmmcclain for initial pointers to where the good content is. With their guidance and help I’ve put together this how-to. Enjoy…  boshing.

Prerequisites

Step: Get an instance/machine up and running.

To make sure I had a totally clean starting point I started out with an AWS EC2 Instance to work from. I chose a micro instance loaded with Ubuntu. You can use your local workstation if you want to or whatever, it really doesn’t matter. The one catch, of course is you’ll have to have a supported *nix based operating system.

Step: Get things updated for Ubuntu.

sudo apt-get update

Step: Get cURL to make life easy.

sudo apt-get install curl

Step: Get Ruby, in a proper way.

\curl -L https://get.rvm.io | bash -s stable
source ~/.rvm/scripts/rvm
rvm autolibs enable
rvm requirements

Enabling autolibs sets up so that rvm will install all the requirements with the ‘rvm requirements’ command. It used to just show you what you needed, then you’d have to go through and install them. This requirements phase includes some specifics, such as git, gcc, sqlite, and other tools needed to build, execute and work with Ruby via rvm. Really helpful things overall, which will come in handy later when using this instance for whatever purposes.

Finish up the Ruby install and set it as our default ruby to use.

rvm install 1.9.3
rvm use 1.9.3 --default
rvm rubygems current

Step: Get bosh-bootstrap.

bosh-bootstrap is the easiest way to get started with a sample bosh deployment. For more information check out Dr Nic’s Stark and Wayne repo on Github. (also check out the Cloud Foundry Bosh repo.)

gem install bosh-bootstrap
gem update --system

Git was installed a little earlier in the process, so now set the default user name and email so that when we use bosh it will know what to use for cloning repositories it uses.

git config --global user.name "Adron Hall"
git config --global user.email plzdont@spamme.bro

Step: Launch a bosh deploy with the bootstrap.

bosh-bootstrap deploy

You’ll receive a prompt, and here’s what to hit to get a good first deploy.

Stage 1: I select AWS, simply as I’ve no OpenStack environment. One day maybe I can try out the other option. Until then I went with the tried and true AWS. Here you’ll need to enter your access & secret key from the AWS security settings for your AWS account.

For the region, I selected #7, which is west 2. That translates to the data center in Oregon. Why did I select Oregon? Because I live in Portland and that data center is about 50 miles away. Otherwise it doesn’t matter which region you select, any region can spool up almost any type of bosh environment.

Stage 2: In this stage, select default by hitting enter. This will choose the default bosh settings. The default uses a medium instance to spool up a good default Cloud Foundry environment. It also sets up a security group specifically for Cloud Foundry.

Stage 3: At this point you’ll be prompted to select what to do, choose to create an inception virtual machine. After a while, sometimes a few minutes, sometimes an hour or two – depending on internal and external connections – you should receive the “Stage 6: Setup bosh” results.

Stage 6: Setup bosh

setup bosh user uploading /tmp/remote_script_setup_bosh_user to Inception VM Initially targeting micro-bosh… Target set to microbosh-aws-us-west-2′ Creating initial user adron… Logged in as admin’ User adron’ has been created Login as adron… Logged in as adron’ Successfully setup bosh user cleanup permissions uploading /tmp/remote_script_cleanup_permissions to Inception VM Successfully cleanup permissions Locally targeting and login to new BOSH… bosh -u adron -p cheesewhiz target 54.214.0.15 Target set to microbosh-aws-us-west-2′ bosh login adron cheesewhiz Logged in as adron’ Confirming: You are now targeting and logged in to your BOSH

ubuntu@ip-yz-xyz-xx-yy:~$

If you look in your AWS Console you should also see a box with a key pair named “inception” and one that is under the “microbosh-aws-us-west-2” name. The inception instance is a m1.small while the microbosh instance is an m1.medium.

That should get you going with bosh. In my next entry around bosh I’ll dive into some of Dr Nic & Brian McClain’s work before diving into what exactly Bosh actually is. As one may expect, from Stark & Wayne we can expect some pretty cool stuff, so keep an eye over there on Stark & Wayne.

Write the Docs, Proper Portland Brew, Hack n’ Bike and Polyglot Conference 2013

Blog Entry Index:

I just wrapped up a long weekend of staycation. Monday kicked off Write the Docs this week and today, Tuesday, I’m getting back into the saddle.

Write the Docs

The Write the Docs Conference this week, a two day affair, has kicked off an expanding community around document creation. This conference is about what documentation is, how we create documentation as technical writers, writers, coders and others in the field.

Not only is it about those things it is about how people interact and why documentation is needed in projects. This is one of the things I find interesting, as it seems obvious, but is entirely not obvious because of the battle between good documentation, bad documentation or a complete lack of documentation. The later being the worse situation.

The Bloody War of Documentation!

At this conference it has been identified that the ideal documentation scenario is that building it starts before any software is even built. I do and don’t agree with this, because I know we must avoid BDUF (Big Design Up Front). But we must take this idea, of documentation first, in the appropriate context of how we’re speaking about documentation at the conference. Just as tests & behaviors identified up front, before the creation of the actual implementation is vital to solid, reliable, consistent, testable & high quality production software, good documentation is absolutely necessary.

There are some situations, the exceptions, such as with agencies that create software, in which the software is throwaway. I’m not and don’t think much of the conference is about those types of systems. What we’ve been speaking about at the conference is the systems, or ecosystems, in which software is built, maintained and used for many years. We’re talking about the APIs that are built and then used by dozens, hundreds or thousands of people. Think of Facebook, Github and Twitter. All of these have APIs that thousands upon thousands use everyday. They’re successful in large part, extremely so, because of stellar documentation. In the case of Facebook, there’s some love and hate to go around because they’ve gone between good documentation and bad documentation. However whenever it has been reliable, developers move forward with these APIs and have built billion dollar empires that employ hundreds of people and benefit thousands of people beyond that.

As developers that have been speaking at the conference, and developers in the audience, and this developer too all tend to agree, build that README file before you build a single other thing within the project. Keep that README updated, keep it marked up and easy to read, and make sure people know what your intent is as best you can. Simply put, document!

You might also have snarkily asked, does Write the Docs have docs,why yes, it does:

http://docs.writethedocs.org/ <- Give em’ a read, they’re solid docs.

Portland Proper Brew

Today while using my iPhone, catching up on news & events over the time I had my staycation I took a photo. On that photo I used Stitch to put together some arrows. Kind of a Portland Proper Brew (PPB) with documentation. (see what I did there!) It exemplifies a great way to start the day.

Everyday I bike (or ride the train or bus) in to downtown Porltand anywhere from 5-9 kilometers and swing into Barista on 3rd. Barista is one of the finest coffee shops, in Portland & the world. If you don’t believe me, drag your butt up here and check it out. Absolutely stellar baristas, the best coffee (Coava, Ritual, Sightglass, Stumptown & others), and pretty sweet digs to get going in the morning.

I’ll have more information on a new project I’ve kicked off. Right now it’s called Bike n’ Hack, which will be a scavenger style code hacking & bicycle riding urban awesome game. If you’re interested in hearing more about this, the project, the game & how everything will work be sure to contact me via twitter @adron or jump into the bike n’ hack github organization and the team will be adding more information about who, what, where, when and why this project is going to be a blast!

Polyglot Conference & the Zombie Apocalypse

I’ll be teaching a tutorial, “Introduction to Distributed Databases” at Polyglot Conference in Vancouver in May!  So it has begun & I’m here for you! Come and check out how to get a Riak deployment running in your survival bunker’s data center. Zombies or just your pointy hair boss scenarios of apocalypse we’ll discuss how consistent hashing, hinted handoff and gossipping can help your systems survive infestations! Here’s a basic outline of what I’ll cover…

Introducing Riak, a database designed to survive the Zombie Plague. Riak Architecture & 5 Minute History of Riak & Zombies.

Architecture deep dive:

  • Consistent Hashing, managing to track changes when your kill zone is littered with Zombies.
  • Intelligent Replication, managing your data against each of your bunkers.
  • Data Re-distribution, sometimes they overtake a bunker, how your data is re-distributed.
  • Short Erlang Introduction, a language fit for managing post-civil society.
  • Getting Erlang

Installing Riak on…

  • Ubuntu, RHEL & the Linux Variety.
  • OS-X, the only user centered computers to survive the apocolypse.
  • From source, maintained and modernized for humanities survival.
  • Upgrading Riak, when a bunker is retaken from the zomibes, it’s time to update your Riak.
  • Setting up

Devrel – A developer’s machine w/ Riak – how to manage without zombie bunkers.

  • 5 nodes, a basic cluster
  • Operating Riak
  • Starting, stopping, and restarting
  • Scaling up & out
  • Managing uptime & data integrity
  • Accessing & writing data

Polyglot client libraries

  • JavaScript/Node.js & Erlang for the zombie curing mad scientists.
  • C#/.NET & Java for the zombie creating corporations.
  • Others, for those trying to just survive the zombie apocolypse.

If you haven’t registered for the Polyglot Conference yet, get registered ASAP as it could sell out!

Some of the other tutorials that are happening, that I wish I could clone myself for…

That’s it for updates right now, more code & news later. Cheers!

Deploycon, PaaS & the pending data tier gravity fallout…

For a quick recap of last years Deploycon & related talks, check out my “Day #3 => DeployCon && Enterprise && Data Gravity” entry from last year.

PaaS Systems aren’t always effectively distributed. Heroku has fallen over every time east-1 has gone down at AWS. Not that I’m saying they’ve done bad, just pointing that out. With Cloud Foundry, there’s several key SPOFs (Single Points of Failure), and with all PaaS Systems the data tier is often the neglected pairing of the system. I’ve been wanting to write about this for a few months now and Deploycon has lit a fire for me to do just that.

Deploycon – “Platform Services and Developer Expectations” **

I’m on a panel at Deploycon titled “Platform Services and Developer Expectations” and this leads right back around to that. This SPOF issue is concerning to me as PaaS Providers talk up the offerings more and more with little light actually shone on this issue. In some ways each is moving away form their respective SPOFs, but overall they’re all pretty prevalent throughout. For security, each has a non-distributed database, which technically needs backed up still – no clear replication or other mechanisms setup to ensure data integrity in a failure situation. Of course, the huge saving grace with a PaaS, is that if the overall system goes down or a SPOF blows up, all the existing deployed applications will generally continue to run. Unless of course the routing and networking are also SPOF. This is the largest glaring concern with PaaS Systems that I see today.

One of the other things about PaaS that has always led to a ton of questions is “what about my PostGresql/mysql/Riak/mongodb/database thing and how do I do X, Y, Z with it to ensure scalability in my PaaS.” In almost every case it ends with a simple and unfortunate answer, “…when it comes to data, a PaaS doesn’t really do a damn thing for ya…” This is obviously not very helpful. The entire reason to put a PaaS into place is to simplify life, the sad fact that it barely does a thing for the data tier isn’t very helpful.

Now, hold on a second before you start screaming at me about “but a PaaS does X, Y and Z and isn’t even supposed to touch that aspect of things…” let me elaborate a bit more. The panel at Deploycon states “…Developer Expectations” and when things are getting simplified in the way a PaaS does, developers assume that if it does all this fancy magic for an application it ought to simplify the data side of things too! Right? Well no, and it isn’t going to for the foreseeable future. But no matter what, it doesn’t change the fact that developers often have that expectation.

Now, I could write at length about all the reasons that PaaS doesn’t really do anything for the data tier. I could wax poetic about how a distributed database (re: Riak, Cassandra, etc) just doesn’t lend itself to a cookie cutter approach to deployment under a PaaS or an RDBMS has umpteen different configurations for stability, scaling, hot swappable services, and other such complexities around the data tier. But instead I’m going to skip all, maybe cover some of those things another day, and jump right into some of the things that are actually moving forward to fill this gap.

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

The most obvious reason there isn’t a simple turn key solution to the data side of things with a PaaS ecosystem is that data is complex and extremely diverse. There’s distributed key/value stores (Riak, Cassandra), there’s sort of kind of distributed databases (Mongo), graph databases (Neo4j), the age old RDBMS (DB2, SQL Server, Oracle’s Stuff, etc) and the million solutions around that, there’s key/value in memory styled databases that are insanely fast, like Redis. Expanding just slightly you have software that works around these systems such as Hadoop & Riak CS & the list goes on. All of it focused on the data tier and maintaining one, two or some form of the three points around CAP Theorem (http://en.wikipedia.org/wiki/CAP_theorem), atomicity and other key capbilities.

All of the PaaS Systems, including public and private often have some sort of plug-in style architectures for data. Whether it is Apprenda which is closed to community and closed source or an ongoing open to community PaaS like OpenShift or Cloud Foundry, things still fall almost entirely to the developers or database team to build an architecture around the data. When looking at solutions to simplify data in PaaS Systems the closed source solutions we have no idea what they’re up to in this regard. The one’s that are open source or in large part public and involved in the community PaaSes, like EngineYard, Heroku, Cloudbees and others we can really see the directions and efforts around creating real PaaS style solutions to the data tier problem.

BOSH, Vagrant, etc…One of the best solutions I’ve seen so far is the ability of Bosh, which was created by the Cloud Foundry team while at VMware, to spool up an environment that includes such things as a Riak Cluster (or other cluster). Currently Brian McClain & Dr Nic have worked to put together such Bosh + Vagrant scripts & get things rolling. I myself will be spending some considerable time on just that. But beyond that this is a good start in enabling data tier back ends.

How to close the gap, between absurdly simple application deployment and still arduous and difficult data tier deployment? For the next several years I think we’ll have cumbersome deployment practices around the data tier. There won’t be anything as elegantly simple as Cloud Foundry’s single line deployment or AppFog’s one click deployment of a web application. The best we can do at this time, is to streamline around pieces and architectures, and at least get them into a kind of simple 3 step deployment.

Please drop a comment or two on how you think we might simplify the data side of the PaaS toolchain. Also drop a few tweets in the twitterverse too, I’m sure that’ll be exploding as usual. I’m @adron, ping me.

Cheers, happy data architecting.

** the Deployconpanel will be at 4:30pm in Santa Clara on April 2nd. Come check it out.

Who is Node PDX for? Who should attend? What’s going on? Where are we at?

Node PDX is a conference that is for programmers (ticket link below!!) both new to the industry and stalwarts, ladies and gentlemen, and simply the curious or already seasoned pros of JavaScript. Our goal is to have great speakers, great topics, encourage and support intelligent and forward thinking conversation. All of this and we’ll strive to move the industry forward as a whole in whatever way we can. Node PDX is for those that want to help do just that!

What is Node PDX? What is Portland?

Node PDX is a conference held yearly in Portland, Oregon. It’s driven by the passionate tech industry located in Portland and around the country and world. At the conference we’ll dive into technologies surrounding JavaScript, Node.js, NPM, hardware and beyond. Other topics may include patterns, design practices, new libraries, and technology add ons.

Portland is a coder’s paradise! Nuff’ said. 🙂

Where is Node PDX?

PDX is the airport code for Portland, Oregon. It’s not at the airport, but if you fly in to our fair city for Node PDX, take the light rail Red Line MAX to downtown and enjoy our coffee, beer and some of the finest food and the most amazing cart collections you’ve ever seen in your life. Promise!

If you’ve never heard of Portland, Oregon here’s a documentary starter…

When is Node PDX?

First off, it’s going to be every year. This year, the grand year of two thousand and thirteen, is going to be on the 16th and 17th of May. That’s a Thursday and Friday on the 3rd week of the fifth month of 2013. Come for Thursday or Friday or Thursday and Friday and enjoy the weekend afterwards. However you pick your days, we hope you come and join us for Node PDX 2013 on May 16-17, 2013.

Why Node PDX?

This is easy. Node PDX is a conference that Troy Howard and I put together out of a frustration from two things: we wanted a conference without the high prices and had a desire to contribute to the local JavaScript and Node.js community, and any other interested coder community. Thus, Node PDX was born in a few quick weeks of hectic hustle. This year we’re back at it with Luc Perkins hacking the program along with us.

Node PDX, How is This Happening?

Mostly magic, with the help of some druids and unicorns. But for some of the heavy lifting we’re looking for sponsors, but we’re throwing an interesting little twists into the fray.

Last year we worked hard to be inclusive and bring together great talks. We did this in a various ways. The first was, we used a Github repository for the site and pull requests as the method to submit talks. We figured, what better way to submit a talk to a developer’s conference than to throw down a commit and pull request to Github. Needless to say, it worked great!

In 2012 to insure we had proper party groove and vittles for the attendees we had volunteers throw down some chef skills and some great sponsors including New Relic, Mozilla and others. This year we’re aiming for a slightly different angle and putting a little more crowd sourcing into the ingredient list.

So here’s how you can get involved!

  • First and foremost: Come to Node PDX – Get your tickets here!!  Oh yeah, there’s a 200 limit, and only so many early birds so do NOT wait!!  🙂
  • Second, throw a talk into the mix! We’ve got two tracks lined up so we’re looking for a lot of material, beginner to advanced!

That’s it! To keep up with the latest, subscribe to my blog here and/or read up on the Node PDX Site! We’re looking forward to seeing you in Portland (even if we see you in Portland every day!)  Cheers!

Distributed Coding Prefunc: Chicago Boss, Rails Based Erlang Power

Troy Howard and I sat down on a Friday night to do some straight up thrashing of Erlang and Chicago Boss. It seemed like a framework that has a lot of promise. Thus, we wanted to hack at it a bit and see what we could come up with.

First things first, I running OS-X and Troy hacking with the latest Ubuntu, we dove in. I first downloaded from the main Chicago Boss Site, which I immediately decided not to use and went straight for a fork and cloning of the repo.

git clone git@github.com:Adron/ChicagoBoss.git
cd ChicagoBoss
make
make app PROJECT=bigonion
cd ../bigonion
./init-dev.sh

The ‘./init-dev.sh’ will launch the server to view whatever is available on the site. Now when I went to the localhost at port 8001, it shows some general mess which is kind of nondescript. But one can identify that the server is running if this works.

Hitting http://localhost:8001

Hitting http://localhost:8001

Boardwalk Explosion!

Boardwalk Explosion!

Oh dear, then the explosion…

At this point Troy started running into a number of problems on Ubuntu. Ranch wasn’t building based on what was included. He went through fighting and attempting to manually install this dependency but no go. He had originally downloaded RB14 of Erlang which should have worked. However it did not. It caused all sorts of madness.

I spooled up my VM of Ubuntu and went through the same steps above, even did the quick intro to setup a hello world, and it worked. We then checked out Erlang versions and I was running with R15B01. Troy went with installing that version and lo’ n’ behold, magic of the druids! Mayor Daley kicked this thing and made it run. So if you are running anything besides R15B01 on Ubuntu, we can’t attest to this stuff working. Just sayin’.

So Back to the Chicago Boss

So go back to the bigonion project above that we created and throw in a controller file. Be sure to use the exact naming, because this is seriously like rails, follow the convention or you’ll spend the rest of your life in the jail of convention-less hell!

Add a file at /src/controller/bigonion_greeting_controller.erl and add the following code.

-module(bigonion_greeting_controller, [Req]).
-compile(export_all).

hello('GET', []) ->
    {output, "<strong>Daley will tell you how to run the Big Onion!</strong>"}.

Now execute the code, or if you have the server running with ‘./init-dev.sh’ still then just navigate to ‘http://localhost:8001/greeting/hello&#8217; and you’ll get a hello world reality check about Chicago history. 😉

On to next things, let’s get a conformance test in there. Add a file called ‘src/test/functional/bigonion_test.erl’ with the following code snippet of a test in it.

-module(bigonion_test).
-compile(export_all).

start() ->
    boss_web_test:get_request("/greeting/hello", [],
        [ fun boss_assert:http_ok/1,
            fun(Res) -> boss_assert:tag_with_text("strong",
                "Daley will tell you how to run the Big Onion!", Res) end ], []).

Now build the project to production.

./rebar compile
./rebar boss c=test_functional

…which left me with

$ ./rebar boss c=test_functional
==> bigonion (boss)
FATAL: Config file "boss.test.config" not found.

Joy, another completely random error. But hey, that’s software development right. So that’s coming later… for now, I’m out. Keep hacking the Erlang bits. If you know what happened with the last command to run the test above, please leave a comment and help out! 🙂