OSCON : Conversations, Deployments, Architecture, Docker and the Future?

I wrote about my first day of OSCON “OSCON : Day 1, Windows Just Doesn’t Do Cloud Foundry… but, there’s a fix for that…“. The rest of the week was most excellent. I caught up with friends and past coworkers. I heard about people working on some amazing new projects. Some things I will try to write up in the coming days, as I’m sure some of it will be making the tech news (if not the regular people news too).

Conversations

Had some great conversations about the direction of enterprise and paas uptake. It’s great to hear that there is some movement in that space finally. As one would expect however, there is still a lot of distance for the enterprise to catch up on, but they’ll get there – or fall apart in the meantime.

There were also tons of conversation about the Indiegogo Ubuntu Edge mobile device. This device is a great looking and sounds like a solid idea. The questions arise in the fact that they’re working to make this a purely crowd funded project. This wouldn’t be a concern if they were trying to just get a few million in capital, but they’re aiming for $32 million! Overall though, with 128 GB, Dual LTE Antennas for Europe and the US, a top tier screen in quality and design, a metal body and also multiple other features that put this phone ahead of anything out there. I hope it’s successful, but I must admit my own hesitance. What’s your take on the device?

Deployments

Over the course of the conference I talked to and worked with a number of other individuals playing around with Cloud Foundry and also OpenShift. The primary aspect that we worked on was strategies around deployment of these PaaS Technologies.

We also worked with Iron Foundry to extend Cloud Foundry to support .NET. If you love .NET or hate .NET, wherever in that spectrum, it has an absolutely huge user base still. Primarily because .NET spent the last decade and a few years going head to head against Java in the Enterprise, and we all know the enterprise is slow to shift anything. So for now and the foreseeable future .NET is an extremely large part of the development world. Having it work in your PaaS is fundamental to gaining significant enterprise share. Cloud Foundry is the only open source, internally usable PaaS on the market today. There are closed source options available, but that obviously doens’t come up at OSCON.

While at OSCON, I also got to discuss architecture and deployment of Riak with a number of people. The usage of Riak continues to grow and the environments, use cases and tooling that people are using Riak with and for is always an interesting space for me. I also got to discuss deployment of Cassandra and even some Neo4j, Redis and Riak side by side deployments. People have used an interesting mix of NoSQL solutions out there to pull their respective data together for their needs.

Among all these deployments, conversations regularly returned to a known topic of mine. Cloud computing and who is capable of what, where and when. AWS is still an easy leader in cloud computing, not just in customers but in technology. This also brought up the concerns and apathy that some have around OpenStack (hat tip to Ben Kepes for the write up) working more homogeneously with AWS. Whatever the case might be, the path for OpenStack needs to be clarified regularly. I imagine the next movement is going to be away from being too concerned with infrastructure and increased concern with portability of applications and development of applications.

Another growing topic of discussion was around building applications for, on and with Windows Azure. Microsoft has actually become dramatically more involved in open source in an honest and more integrity based way. I’m honestly amazed at how far they’ve come from the declaration years ago that “open source is a cancer” and the all too famous, “linux is communism“. Whatever that was supposed to mean, they didn’t seem to get it back then. Now however, they regularly contribute to open source projects on codeplex but also github and other places. Microsoft has even contributed to the Linux kernal a few months ago.

That leads me to the next topic that came up a number of times…

Architecture

There’s been a lot of discussion about architecture around PaaS, containers (more on that in a moment), distributed systems in general and distributed databases. As I wrote about recently, “Architectural PaaS Cracks or Crack PaaS” the world of distributed systems and distributed databases has more than a few issues when working together in a PaaS environment. This brought up the discussion about what solutions exist today, solutions I look forward to writing and building in the coming months.

The most immediate solution to scalable data sources is still to run your operational data sources such as Neo4j, Redis, Riak or other database autonomously but residing close to your PaaS System. The current public PaaS Providers do exactly this and in some cases extend that to offer the databases and data sources as services through add-ons. These are currently great solutions, but require time, effort and custom development work when setting up internally.

This leads me to the last topic…

The Story of a Container – Docker

Well, not just Docker, but containers in general and Docker specifically. First some context about what a container is.

Container – In this particular context I’m writing about a container, or more specifically a runtime-container, that isolates resources for applications or services. Containers are common in PaaS technologies to help isolate the specific services or applications when they’re on a single physical machine or instance. For each of the respective PaaS systems that came up at OSCON we have dotCloud from the same team that created Docker, Cloud Foundry has Warden and OpenShift has gears and Red Hat Enterprise Linux OS specific containers.

I’ve studied Warden a little in the past while I was working with AppFog and Tier 3 around Cloud Foundry. Warden is a great piece of technology. However the star at OSCON was clearly Docker. I jumped into a number of conversations around Docker. This conversation would then take the direction to containers becoming the key to PaaS tooling and systems growth and increasing capabilities. That leads me back to my previous blog entry “Architectural PaaS Cracks or Crack PaaS” and one of the key solutions to the data tier issue.

Containers, A Solution for Scaling the Data Tier

One of the issues that comes up when trying to scale any distributed database in a PaaS Environment is how to provide multi-tenancy without spooling up new instances for each and every single installation of a node within that distributed database. Here’s an example diagram of the requirements behind a scalable distributed database.

Masterless, Distributed Cluster of Nodes
Masterless, Distributed Cluster of Nodes

In a default configuration you’d want each node to be running on a physical machine or dedicated virtual instance. This is for performance reasons as well as reasons for load balancing, security, data integrity and a host of others. This is the natural beginning state of a highly available distributed database or distributed system.

Trying to deploy something like this into a PaaS environment is tricky. Take into account that there is no such thing in application or service speak as an instance, and especially not anything such as a physical server. The real division between process and resources are containers. These containers are what actually needs to run the distributed system node. This becomes possible, if a distributed system node can be deployed to and executed from within a container.

Enter Docker

After reviewing Docker, the capabilities around it and the requirements of a distributed database, it looks like an ideal marriage of the two technologies. Already Docker has Redis and other database technologies running on it. The Container technology around Docker looks like an ideal fit to extend distributed systems to run autonomously of a single physical machine or single instance per node. This would enable nodes to be deployed as resources are available to provide a more seamless and PaaS style deployment for systems like Cassandra, Riak and related distributed systems. Could this be the next evolution of affordable distributed systems, containers to the rescue?

I’ll be reporting back on my progress, this could be cool!

Stay tuned for a write up on Docker in the near future. For more information now check out http://www.docker.io.

Lenovo X1 Carbon Touch :: Opening, Setup and Failure

X1 Carbon with all the standard parts that come in box. (Click for full size image)
X1 Carbon with all the standard parts that come in box. (Click for full size image)

Yesterday I received my X1 Carbon Touch from Amazon. First part of this whole adventure is that I sent it to my old address in one part of town so that led to a little sleuth action to track it down. After a short bike ride up the street I arrived and the office staff had my X1 Carbon. Whew, disaster averted.

I went down to Ace Hotel were one of the local Stumptown locations is to open it up and see what I was in store for. Nothing like a good macchiato while I unpackaged the new machine. When I arrived I ran into Nathan Aschbacher and Eric Redmond. Two of my fellow Basho comrades. We all grabbed coffee and headed up to the roost for some hacking and conversation.

Unpacking

In the package, laptop sitting on the table at Ace Hotel's Stumptown Coffee.
In the package, laptop sitting on the table at Ace Hotel’s Stumptown Coffee. (Click for large image)

Unpacking the Lenovo X1 Carbon is a straight forward process. A simple box, no elegance, just a box with some labels and logos on it. Pulling the laptop out of the box, still just the bare minimum. No bells, no whistles, even the documentation is a 2-3 page pamphlet. Personally, I’m totally cool with this approach. I find Apple’s packaging to be an experience of sorts, however extensively wasteful.

One of the applications I found not available for Windows 8 was a native HipChat client. This actually makes sense, since most of their customers are likely using Linux or OS-X. It really shows how Windows has seriously lost the edge with developers.

Nathan and Eric both give a feel to see how light and strong the laptop is. Nobody actually threw the laptop, but we all wanted to, just to see how it would hold up. Maybe with somebody else’s hard earned Lenovo purchase. 😉

Nathan gives it a look.  (Click for full size image)
Nathan gives it a look. (Click for full size image)
Gleefully smiling at the laptop, Eric proposes we throw it over the guardrail to the first floor below. (Click for full size image)
Gleefully smiling at the laptop, Eric proposes we throw it over the guardrail to the first floor below. (Click for full size image)
No start? (Click for full size image)
No start? (Click for full image)

After Nathan and Eric threaten the poor laptop, I set her down and try and get her booted up. First thing I notice, it doesn’t start. I’m puzzled? Why doesn’t it start? I pick at my PC Tech experience and think, “oh yeah, probably gotta do something stupid an unintuitive like plug it in for some magically arbitrary amount of time first”.

Lenovo lives! (Click for full size)
Lenovo lives! (Click for full size)

So I plug it in and try again. A small light around the power button, kind of a halo, lights up and immediately I get the happiness. The machine is coming to life. A bright Lenovo logo pops on the screen with the notorious Windows 8 swirly working image below.

Move ya mouse! (click for full size image)
Move ya mouse! (click for full size image)

Windows 8 then shifts into a preparing windows workflow which basically means you fill out a few things and it does something to the OS to make it ready to run. I sit through a solid 7-10 minutes of these screens, these fluctuating colors. It’s rad, in a psychedelic waste of time kind of way. However, I’ll admit, my Mac Book Air is sitting beside me running just fine that I’m using to do work while I wait for all this process to finish. I’m no amateur at loading operating systems, I come prepared. 😉

A Problem Arises

I relocate to Bailey’s Taproom after setting up some basic things and installing Visual Studio 2012 on the machine. While working through updates and installing patches my track pointer (the little red button thingy in the middle of the keyboard, that Lenovo is famous for) stops working.

I toy around with the settings and see why the track pointer is shadowed out in the settings. I battle with Windows 8 trying to find the easiest way into the settings and out of the settings and to the desktop and to the start screen and back and forth. It’s somewhat tumultuous but in the end it’s helping me get used to the new system and where everything is. But still, I’ve no idea why the track pointer thingy doesn’t work. I consult the great Google.

Apparently the drivers that it ships with are the suck. I get pointed to this video by Jesse Anderson.

After I get the drivers installed, everything is working flawlessly again. Onward!

Flakiness o’ Windows 8

As I’m working on Windows 8 setting up some of the cool applications for the start menu (or whatever the metro dealio is called now) I get a really flaky behavior. This is the kind of behavior that screams “we don’t really pay attention to usability” or maybe it screams “we’ve no idea what we shipped” or maybe it’s just a simple example of “oh shit we shipped that stupid user experience“. Whatever the case is, this is it…

Notice something redundant here?
Notice something redundant here?

Yup, on a laptop with a HARDWARE LAPTOP ATTACHED Windows 8 is showing me the keyboard. WTF kind of pure idiocy of a UX is this? My mind is blown. After years of the iPad having this problem figured out (and Apple doesn’t even sell keyboards themselves). When you have a HARDWARE keyboard NEVER show anybody the stupid SOFTWARE keyboard EVER. Seriously, this has to be one of the dumbest UX situations that I’ve seen in ages. This is a total failure of logical flow. Note also, this screen doesn’t fold all the way around, this is a laptop pure and simple, not in any way a tablet. But there’s the SOFTWARE keyboard that one should only see on a tablet! Oh well, it aint the end of the world, it’s just DUMB.

I get everything else setup, zonk for the night after working through all the software installations and patches. All is right. All is cool.

Loading Ubuntu Linux

Loading Ubuntu (Click for full size)
Loading Ubuntu (Click for full size)
Ubuntu (Click for full size)
Ubuntu (Click for full size)

The next morning I rise early and get to working on the next phase of my installation. I don’t, by any means, intend to use Windows 8 all the time on this machine. I want to have a dual boot of Ubuntu and Windows 8 on this laptop so that I can have every OS (OS-X, Windows 8 and Ubuntu) running natively on at least one machine that I have.

I do a little research and find this information about making a bootable USB Stick for Ubuntu from Windows. That information points me to this application that makes it a no brainer to get a bootable USB stick ready for use via Linux Live USB.

Just As I Got Ubuntu Installed…

I shut down the computer after getting all of these things installed. Windows 8 was finally fully patched, Ubuntu was installed and running with all patches too. The X1 seemed to hang on the shutdown. So I held down the power button for about 8 seconds to hard reboot the machine. Thinking that it would startup no problem at a later time I packed it in my messenger bag and headed off to a meeting I had scheduled. I arrived at the meeting and went to start the laptop again.

…nothing.

So I tried to hold the button down for 7 seconds to start it back up.

…nothing.

I packed it back up and returned to a place I could plug it in and try to start it. I swung into Backspace and found an electrical jack. Plugged in, counted a few seconds just for good luck. I then held down the power button for 8 seconds to see if it would start plugged in.

…nothing.

SOS SOS SOS !!!!!
SOS SOS SOS !!!!!

(Click for large image)

I then sporadically pressed the button. I then used morse code to spell S.O.S. on the power button.

…nothing.

I resigned myself to now owning a large paper weight. Albeit a much lighter paper weight than what laptops traditionally weighed. My X1 Carbon Touch was dead. I called tech support.

Impressive Support

First thing that happens, I navigate through support quickly. The automated voice tells me I am now being connected to Lenovo Support in Atlanta, Georgia. At this point I was impressed. I’m getting to speak to someone in the country where I’ve bought the machine. That is cool.

I get connected to Tom in support. I fill him in on my sitrep. We walk through some basic troubleshooting. Such as the “pin in the battery reset hole” trick“.

…nothing.

Tom wastes no time as I’ve already laid out everything I have in this blog entry. He declares it dead and gets a box on its way to me for returning it to Lenovo. With a promised 7 day return after I ship it to them. Well hot damn, my laptop is dead by I’m stoked to have support like this. I don’t recall support this good since the late 90s!!

As I tweeted about this I got a lot of responses like this. I concur and I already have a mac, this is an machine specifically NOT for using the mac.  😛

So in the meantime, it’s back to Windows 8 via VMware Fusion on a good ole’ mac!

Windows 8 via VMware Fusion on OS-X. (Click for full size)
Windows 8 via VMware Fusion on OS-X. (Click for full size)

…so stay tuned, for the ongoing saga of Windows 8 & Ubuntu Linux Development on a Lenovo!

Alright Stop Waiting, CorrugatedIron v1.0 .NET Client released for Riak!

I’ll kick right off with all the specifics:  Jeremiah did a blog entry on today’s release titled “Just one more thing…  Introducing Corrugated Iron v1.0“.

Send a congrats out to the team duo of OJ @TheColonial & Jeremiah @peschkaj via Twitter. Check out the .NET Rocks Podcast with Jeremiah talking with Carl and Richard about storing data in Riak. Also check out Adrian Hills’ article on getting up and running via Nuget with Corrugated Iron in Visual Studio, ping him on Twitter @AdaTheDev.

So no excuses in .NET land to write some apps that are hard core data centric and capable based on the power of Riak! There are tons of features. You can read about them yourself via the README.md file on the CorrugatedIron Repository, but I wanted to post the features right here so you get an idea of the feature rich capabilities of the library. In addition, it does indeed work on Linux & OS-X with Mono. So don’t let Windows get in your way! 🙂

Current Features

♥: denotes availability of both blocking and asynchronous APIs
«: denotes availability of both streaming and non-streaming APIs

  • Riak cluster support:
    • One or more nodes in the cluster.
    • Load-balancing and pooling of connections across the nodes.
      • Currently only round-robin is supported, more strategies to come later.
    • Per-node configuration for:
      • Host Name (purely used for identification).
      • Host Address.
      • PBC Port.
      • HTTP/REST Port.
      • Pool Size.
      • Timeout parameters.
  • Server ping.    ♥
  • Get server information/version.    ♥
  • Simple Get/Put/Delete operations.    ♥
  • Bulk Get/Put/Delete operations.    ♥
  • List buckets.    ♥
  • List keys.    ♥  «
  • Semi-fluent Map/Reduce.    ♥  «
  • Link walking.    ♥
  • Delete buckets.    ♥
  • Set/Get bucket properties.    ♥
  • Batch operations on a single connection.
    • Each time a Client function is called resulting in communication with the Riak cluster, a connection is pulled from a pool on a given node. In most use-cases this functionality is fine as it is often single-shot calls that are made. There are, however, cases where many operations will happen at once. Rather than forcing the user to make multiple calls to the client, resulting in multiple connection acquisitions behind the scenes, the user can use the Batch interface to make many calls on a single connection. This also reduces the overhead of setting the client ID on each call.
    • Because a batch operation reuses a single connection only a subset of the client API is available for batch actions. The functions that are excluded are the asynchronous functions.
  • Graceful degrades to HTTP/REST API when the request isn’t supported via Protocol Buffers.
  • Configurable via web.configapp.config or a custom configuration file.

CorrugatedIron works with .NET 4.0 on Windows and Mono on Linux and OSX.

Thor Project Opens Up, Building the Cloud Foundry Ecosystem with the Community

The Iron Foundry Team are big advocates of open source software. We write code across all sorts of languages, just like many of the development shops out there do. Sometimes we’re heavy on the .NET, other times we’re all up in some Java, Ruby on Rails, spooling up a Node.js Application or something else. So keeping with our love of open source and our polyglot nature we’ve created the Thor Project with three distinct apps.

Before jumping into the applications though, a little context for what and where Thor is in the grand scheme of things. We need to roll back to the Cloud Foundry Project to get into that. The Cloud Foundry Project is an open source project built around software for PaaS (Platform as a Service) which can be used to build your own PaaS internally or externally, in a cloud provider or directly on hardware. It’s your choice how, when and where you want to use it. For more context on PaaS check out my previous entry “The Confusions of IaaS, PaaS and SaaS“.

Thor Project

Cocoa for OS-X

Thor Odinson
Thor Odinson, God of Thunder

You know who Thor is right? He’s this mythic Norse God, also known as the God of Thunder. Since we’re all about bringing the hamma we welcomed Thor into our team’s stable of applications. So starting immediately we’ve released Thor into the realms for contributions and fighting the good open source software battle! If you’d like to join the effort, check out the github project and feel free to join us!

Technically, what is the Thor Application? This is a Cocoa Application built for OS-X that is used for managing, deploying and publishing applications to Cloud Foundry enabled and or Iron Foundry extended PaaS Environments.

.NET for Windows 7

The .NET Metro version of the Thor Application is also released via github with a provided installer. We’ve almost taken the same path, except of course for the very different UX and UI queues with Windows 7 and the Metro UX design guidelines.

WinRT for Windows 8

I wasn’t really sure what to call this version. Is it Metro or WinRT or Windows 8 or something else? Anyway, there is a project, it is albeit empty at this point, but it is the project where the Windows 8 version of Thor will go! For now get the Windows 7 version and install it on Windows 8, it won’t have touch interface support and things, but should work just like a regular application on Windows 8.

The Code

To get started with these, generally you’d just clone the repo and do a build, then get started checking out the code. There is one catch, for the OS-X version you’ll want to pull down the sub-modules with the following command.

[sourcecode language=”bash”]
git clone git@github.com:YourForkHere/Thor.git
git submodule update –init –recursive
[/sourcecode]

Once you do that in XCode just make sure to then select the right project as the starting build project.

…then when the application is launched…

Thor Running in OS-X
Thor Running in OS-X

I’ll have more in the coming days and weeks about Thor & Iron Foundry. For now, check out the blog entry on the Iron Foundry Blog and subscribe there for more information.

Ways to Interact Asynchronously with C#

NOTE: All of this code is available at my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.

While working on the Thor Project there have been numerous situations where I need to fire off an asynchronous callback in C# while maintaining good responsiveness in the actual user interface. Benjamin (@bvanderveen) has been using Reactive Extensions with subscriptions to do this in the Objective-C code for the Cocoa based OS-X Thor user interface. See my previous blog entry for an example of what he’s doing.

For a good example of asynchronous calls against Cloud Foundry I setup the following project using the Iron Foundry Project VCAP Client Library. The first thing I setup was a static class with a few constants to use across the examples for the URI, username and password for the Cloud Foundry Account.

[sourcecode language=”csharp”]
public static class YourSecrets
{
public const string Username = "youremail@someplace.com";
public const string Password = "AnAwesom3HardPassw0rd!";
public const string Uri = "http://api.yourpaas.com";
}
[/sourcecode]

Next step was to setup the delegate and method I’d use for calling out to the Cloud Foundry environment and retrieving data in parallel to my active console or user interface. That code snippet looked like this. I also added a private variable _finished for use in tracking when the request was completed in the begin and end invoke example below.

[sourcecode language=”csharp”]
private bool _finished;

IEnumerable TheMethodToConnectThatWillTakeLongTime(string uri)
{
var client = new VcapClient(uri);
client.Login(TheSecretBits.YourSecrets.Username, TheSecretBits.YourSecrets.Password);

_finished = false;

return client.GetApplications();
}

delegate IEnumerable MethodDelegate(string uri);
[/sourcecode]

Once I had that setup I was ready to create my baseline method that would make a synchronous call. A synchronous call is one that makes the call as if it just called the method directly. There’s no real reason to create one like I’ve done here, but I was just using it to provide a basic example of calling the delegate.

[sourcecode language=”csharp”]
public void SynchronousCall()
{
var starting = DateTime.Now.ToLongTimeString();

var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);
var returnedBits = delegateMethod(TheSecretBits.YourSecrets.Uri);

var ending = DateTime.Now.ToLongTimeString();

Console.WriteLine(string.Format("The delegate call returned \n\n{0}\n\nstarting at {1} and

ending at {2} which takes a while of waiting.",
returnedBits, starting, ending));

_finished = false;
}
[/sourcecode]

That gets us a baseline. If you run a synchronous call against anything with a console application or a windows app, WPF or whatever it will lock up the calling thread while it is waiting for a response. In any type of user interface that is unacceptable. One of the best options is to fire of an asynchronous callback. The way I did this, which is an ideal way to make calls with the Iron Foundry Client Library against a Cloud Foundry Environment, is shown below.

This is my asynchronous call.

[sourcecode language=”csharp”]
public void DemoCall()
{
Console.WriteLine("Callback:");
var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);

var callbackDelegate = new AsyncCallback(MyAsyncCallback);

Console.WriteLine(" starting…{0}", DateTime.Now.ToLongTimeString());
delegateMethod.BeginInvoke(TheSecretBits.YourSecrets.Uri, callbackDelegate, delegateMethod);
Console.WriteLine(" ending…{0}", DateTime.Now.ToLongTimeString());
}
[/sourcecode]

Now the simple callback.

[sourcecode language=”csharp”]
public void MyAsyncCallback(IAsyncResult ar)
{
Console.WriteLine("Things happening, async state calling.");

var delegateMethod = (MethodDelegate)ar.AsyncState;

Console.WriteLine(" called…{0}", DateTime.Now.ToLongTimeString());

var returnedBits = delegateMethod.EndInvoke(ar);

Console.WriteLine(" end invoked…{0}", DateTime.Now.ToLongTimeString());

foreach (Application application in returnedBits)
{
Console.WriteLine("Application {0} is in {1} state…",
application.Name, application.State);
Console.WriteLine(" with {0} running instances, {1} memory per instance, {2} disk allocated…",
application.RunningInstances, application.Resources.Memory, application.Resources.Disk);
Console.Write(" hosted at ");
foreach (var uri in application.Uris)
{
Console.Write(uri + " ");
}
Console.WriteLine(".");
}
}
[/sourcecode]

That’ll get the call running on a parallel thread and when it is wrapped up it returns the data.

The User Interface Interaction Issue

This is all fine and dandy for the command console. But if you want to give control back to the UI thread in a UI application and make sure that the background thread can actually update a control when fired off, do the same thing as I’ve discussed here except set the control up to invoke the dispatcher, so that the “threads don’t cross” when trying to return information to a control that needs updated. In order to do this take the control that needs updated and set the Dispatcher Invoke method as shown below.

[sourcecode language=”csharp”]
private void Write(string updateText)
{
UpdatingTextBlock.Dispatcher.Invoke(
System.Windows.Threading.DispatcherPriority.Normal,
new Action(delegate
{
UpdatingTextBlock.Text += updateText;
}
));
}
[/sourcecode]

For more on the Iron Foundry Project and the library I’ve used here, check out the Iron Foundry Blog & Site. For more information on Thor and to follow or get involved check out the Thor Project Site (Hosted with Cloud Foundry at Tier 3!).

All of this code is available in my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.