Month: July 2013

OSCON : Conversations, Deployments, Architecture, Docker and the Future?

I wrote about my first day of OSCON “OSCON : Day 1, Windows Just Doesn’t Do Cloud Foundry… but, there’s a fix for that…“. The rest of the week was most excellent. I caught up with friends and past coworkers. I heard about people working on some amazing new projects. Some things I will try to write up in the coming days, as I’m sure some of it will be making the tech news (if not the regular people news too).

Conversations

Had some great conversations about the direction of enterprise and paas uptake. It’s great to hear that there is some movement in that space finally. As one would expect however, there is still a lot of distance for the enterprise to catch up on, but they’ll get there – or fall apart in the meantime.

There were also tons of conversation about the Indiegogo Ubuntu Edge mobile device. This device is a great looking and sounds like a solid idea. The questions arise in the fact that they’re working to make this a purely crowd funded project. This wouldn’t be a concern if they were trying to just get a few million in capital, but they’re aiming for $32 million! Overall though, with 128 GB, Dual LTE Antennas for Europe and the US, a top tier screen in quality and design, a metal body and also multiple other features that put this phone ahead of anything out there. I hope it’s successful, but I must admit my own hesitance. What’s your take on the device?

Deployments

Over the course of the conference I talked to and worked with a number of other individuals playing around with Cloud Foundry and also OpenShift. The primary aspect that we worked on was strategies around deployment of these PaaS Technologies.

We also worked with Iron Foundry to extend Cloud Foundry to support .NET. If you love .NET or hate .NET, wherever in that spectrum, it has an absolutely huge user base still. Primarily because .NET spent the last decade and a few years going head to head against Java in the Enterprise, and we all know the enterprise is slow to shift anything. So for now and the foreseeable future .NET is an extremely large part of the development world. Having it work in your PaaS is fundamental to gaining significant enterprise share. Cloud Foundry is the only open source, internally usable PaaS on the market today. There are closed source options available, but that obviously doens’t come up at OSCON.

While at OSCON, I also got to discuss architecture and deployment of Riak with a number of people. The usage of Riak continues to grow and the environments, use cases and tooling that people are using Riak with and for is always an interesting space for me. I also got to discuss deployment of Cassandra and even some Neo4j, Redis and Riak side by side deployments. People have used an interesting mix of NoSQL solutions out there to pull their respective data together for their needs.

Among all these deployments, conversations regularly returned to a known topic of mine. Cloud computing and who is capable of what, where and when. AWS is still an easy leader in cloud computing, not just in customers but in technology. This also brought up the concerns and apathy that some have around OpenStack (hat tip to Ben Kepes for the write up) working more homogeneously with AWS. Whatever the case might be, the path for OpenStack needs to be clarified regularly. I imagine the next movement is going to be away from being too concerned with infrastructure and increased concern with portability of applications and development of applications.

Another growing topic of discussion was around building applications for, on and with Windows Azure. Microsoft has actually become dramatically more involved in open source in an honest and more integrity based way. I’m honestly amazed at how far they’ve come from the declaration years ago that “open source is a cancer” and the all too famous, “linux is communism“. Whatever that was supposed to mean, they didn’t seem to get it back then. Now however, they regularly contribute to open source projects on codeplex but also github and other places. Microsoft has even contributed to the Linux kernal a few months ago.

That leads me to the next topic that came up a number of times…

Architecture

There’s been a lot of discussion about architecture around PaaS, containers (more on that in a moment), distributed systems in general and distributed databases. As I wrote about recently, “Architectural PaaS Cracks or Crack PaaS” the world of distributed systems and distributed databases has more than a few issues when working together in a PaaS environment. This brought up the discussion about what solutions exist today, solutions I look forward to writing and building in the coming months.

The most immediate solution to scalable data sources is still to run your operational data sources such as Neo4j, Redis, Riak or other database autonomously but residing close to your PaaS System. The current public PaaS Providers do exactly this and in some cases extend that to offer the databases and data sources as services through add-ons. These are currently great solutions, but require time, effort and custom development work when setting up internally.

This leads me to the last topic…

The Story of a Container – Docker

Well, not just Docker, but containers in general and Docker specifically. First some context about what a container is.

Container – In this particular context I’m writing about a container, or more specifically a runtime-container, that isolates resources for applications or services. Containers are common in PaaS technologies to help isolate the specific services or applications when they’re on a single physical machine or instance. For each of the respective PaaS systems that came up at OSCON we have dotCloud from the same team that created Docker, Cloud Foundry has Warden and OpenShift has gears and Red Hat Enterprise Linux OS specific containers.

I’ve studied Warden a little in the past while I was working with AppFog and Tier 3 around Cloud Foundry. Warden is a great piece of technology. However the star at OSCON was clearly Docker. I jumped into a number of conversations around Docker. This conversation would then take the direction to containers becoming the key to PaaS tooling and systems growth and increasing capabilities. That leads me back to my previous blog entry “Architectural PaaS Cracks or Crack PaaS” and one of the key solutions to the data tier issue.

Containers, A Solution for Scaling the Data Tier

One of the issues that comes up when trying to scale any distributed database in a PaaS Environment is how to provide multi-tenancy without spooling up new instances for each and every single installation of a node within that distributed database. Here’s an example diagram of the requirements behind a scalable distributed database.

Masterless, Distributed Cluster of Nodes
Masterless, Distributed Cluster of Nodes

In a default configuration you’d want each node to be running on a physical machine or dedicated virtual instance. This is for performance reasons as well as reasons for load balancing, security, data integrity and a host of others. This is the natural beginning state of a highly available distributed database or distributed system.

Trying to deploy something like this into a PaaS environment is tricky. Take into account that there is no such thing in application or service speak as an instance, and especially not anything such as a physical server. The real division between process and resources are containers. These containers are what actually needs to run the distributed system node. This becomes possible, if a distributed system node can be deployed to and executed from within a container.

Enter Docker

After reviewing Docker, the capabilities around it and the requirements of a distributed database, it looks like an ideal marriage of the two technologies. Already Docker has Redis and other database technologies running on it. The Container technology around Docker looks like an ideal fit to extend distributed systems to run autonomously of a single physical machine or single instance per node. This would enable nodes to be deployed as resources are available to provide a more seamless and PaaS style deployment for systems like Cassandra, Riak and related distributed systems. Could this be the next evolution of affordable distributed systems, containers to the rescue?

I’ll be reporting back on my progress, this could be cool!

Stay tuned for a write up on Docker in the near future. For more information now check out http://www.docker.io.

Johnny 5 yo! Nodebots Day PDX

Today the first Nodebots Day took place worldwide! Portland held its own event here at the old Urban Airship garage space. With the garage door wide open and the glorious day gleaming daylight into the garage 50+ hardware and coding beginners, learners, coders and hackers of all backgrounds came together. Breadboards were wired up, code was slung, robots moved and twitched to life.

This all started with a super quick organizing effort by some local JavaScript hackers to join the worldwide Nodebots Day (check the repo for infoz). After a meet, some sponsors jumping to our aid, and some hustle by some great people, the event in Portland came together. The turnout was great!

The bits everybody got...
The bits everybody got…
Bits up close...
Bits up close…

My Own Robot Battle

Oh dear, I was dead zonked when I arrived. It’s been a super long and hard week. I’ve had deadlines to meet, code to write and OSCON to attend, needless to say that leaves basically a few hours each night of the last week to actually recharge. Things were definitely catching up with me…  and I’ll admit I made almost zero progress, however I was super excited to see what many others accomplished!

Panoramic View of Nodebots Day PDX.

There was the quad copter that Carter @CarterRabasa got up and flying with some aerobatic acrobatic flips.

There was an erector set wheeled robot that was primed for deployment. Nothing like combining the quality and build endurance of erector set gear with that of modern machine and robot automation for fun!

Troy going mad scientist on his bot.
Troy going mad scientist on his bot.

Troy @thoward37 was building a walking bot for world domination… which if he didn’t finish it after my departure I’m looking forward to see it walking and doing full auto-deploy in the near future.

Serial Port for a head!
Serial Port for a head!
Wires, Connections, Devices & More...
Wires, Connections, Devices & More…

There was musical linkages being made to device and computer alike. With code combining to form knew methods of interaction between device, human and music.

Along with these bots there was much progress among breadboards laced with ideas and blinking LEDs amidst us all. I do believe everybody had a blast and learned a lot.

Erector Robot
Erector Robot
Edit Post ‹ Composite Code — WordPress
Working together… creating the robot society!
Codez! Arduino! Wires! Brains!
Codez! Arduino! Wires! Brains!

Johnny 5 and I say “THANKS AND HIGH FIVES!!!”

A huge thanks to so many, I might have missed a few people, apologies (and let me know, I’ll add you to the list of thanks!!)  —  and also, thanks to EVERYBODY who came out and worked and learned about robots!

Also if anybody has any questions about robots, javascript, node.js, robots in Portland, Portland Bridges or the event or about coming to Portland to hack, code, beer, food or just move here. Let me know and I’ll be more than happy to hook you up with appropriate resources!  Peace!

  • @HackyGoLucky – Cuz yeah, you kicked some ass and herded all us cats together for this! Thanks Tracy!
  • @nexxylove – Thanks for the code repo o’ lights! I’m sure you enjoyed San Francisco, we missed you in Portland! We’ll hack when you’re back!
  • @BlaineBublitz – Welcome to Portland again, thanks for traveling into town! I’ll ping you next I’m in Phoenix and we’ll hack the light rail. 😉
  • @s5fs – Yo, ok, you just front loaded my cortex full of ideas on that last brunch convo. Thanks for helping, being kick ass, and coming out to Nodebots day!
  • @nickniemeir – Thanks for coming up to Nodebots day PDX!
  • @thoward37 – What did you do again? You keep showing up everywhere… are you a robot?
  • @CarterRabasa – Thanks for coming down to the Stumptown from the Emerald City.
  • @_jden – Yo, Palo Alto to bridge city (another PDX nick name)… welcome back to PDX and to the future with our Robot Overlords!

Resources:

Architectural PaaS Cracks or Crack PaaS

Over the last couple years there have been two prominent open source PaaS Solutions come onto the market. Cloud Foundry & OpenShift. There’s been a lot of talk about these plays and the talk has slowly but steadily turned into traction. Large enterprises are picking these up and giving their developers and operations staff a real chance to make changes. Sometimes disruptive in a very good way.

However, with all the grandeur I’m going to hit on the negatives. These are the missing parts, the serious pain points beyond just some little deployment nuisance. Then a last note on why, even amidst the pain points, you still need to make real movement with PaaS tooling and technologies.

Negative: The Data Story is Lacking

Both Cloud Foundry and OpenShift have a way to plug into databases easily.

Cloud Foundry provides ways to build a Cloud Foundry Service that becomes the bound and hooked in SQL Server, MySQL, Postgresql, Redis or whatever data storage service you need. For more details on building a service, check out the echo example on the vcap sample github project.

OpenShift has what are called Cartridges which provide the ability to add databases and other services into the system. For more information about the cartridges check out Red Hat’s OpenShift Documentation and also the forums.

Cloud Foundry and OpenShift however have distinctive weak spots when it comes to services that go beyond a mere single instance database. In the case of a true distributed database such as Cassandra, HBase or Riak, it is inordinately difficult to integrate a system that any PaaS inter-operates with well. In some cases it’s irrelevant to even try.

The key problem being that both of the PaaS systems assume the mantle of master while subjugating the distributed database a lower tier of coordination. The way to resolve this at the moment is to do an autonomous installation of Riak, Cassandra, Neo4j or other database that may be distributed, stored hot swappable, or otherwise spread across multiple machine or instance points. Then create a bound connection between it and the PaaS Application that is hosted. This is the big negative in PaaS systems and tooling right now, the data story just doesn’t expand well to the latest in data and database technologies. I’ll elaborate more about this below.

Negative: Deployment is Sometimes Easy, Maintenance is Sometimes Hard

Cloud Foundry is extremely rough to deploy, unless you use Bosh to deploy to either VMware Virtualized instances or AWS. Now, you could if resources were available get Bosh to deploy your Cloud Foundry environment anywhere you wanted. However, that’s not easy to do. Bosh is still a bit of a black box. I myself along with others in the community are working to document Bosh, but it is slow going.

OpenShift is dramatically easier to deploy, but is missing a few key pieces once deployed that draw some additional operational overhead. One of those is that OpenShift requires more networking management to handle routing between various parts of the PaaS Ecosystem.

Overall, this boils down to what you need between the two PaaS tool chains. If you want Cloud Foundry’s automatic routing and management between nodes. This is a viable route, but if your team wants to manage the networking tier more autonomous from the PaaS environment then maybe OpenShift is the way to go. In the end, it’s negative bumpy territory to determine which you may or may not want based on that.

Negative: Full Spectrum Polyglot, Missing Some

Cloud Foundry has a wider selection of languages and frameworks with community involvement around those with groups like Iron Foundry. OpenShift I’m sure will be getting to parity in the coming months. I have no doubt between both of these PaaS Ecosystems that they’ll expand to new languages and frameworks over time. Being polyglot after all is a no brainer these days!

Why PaaS Is, IMHO, Still Vitally Important

First toss out the idea that huge, web scale, Facebooks and Googles need to be built. Think about what the majority of developers out there in the world work on. Tons and tons and tons of legacy or greenfield enterprise applications. Sometimes the developer is lucky enough to work on a full vertical mix of things for a small business, but generally, the standard developer in the world is working on an enterprise app.

PaaS tooling takes the vast majority of that enterprise app maintenance from an operational side and tosses it out. Instead of managing a bunch of servers with a bunch of different apps the operations team manages an ecosystem that has a bunch of apps. This, for the enterprises that have enough foresight and have managed their IT assets well enough to be able to implement and use PaaS tooling, is HUGE!

For companies working to stay relevant in the enterprise, for companies looking to make inroads into the enterprise and especially for enterprises that are looking to maintain, grow or struggling to keep ahead of the curve – PaaS tooling is something that is a must have.

Just ask a dev, do they want to spend a few hours configuring and testing a server?  Do they want to deploy their application and focus on building more value into that application?

…being I’ve spent a few years being the developer, I’ll hedge on the side of adding value.

What’s Next?

So what’s next? Two major things in my opinion.

1. Fill the data gap. Most of the PaaS tooling needs to bridge the gap with the data story. I’m working my part with testing, development and efforts to get real options built into these environments, but this often leads back to the data story of PaaS being weak. What’s the solution here? I’m in talks, ongoing, planning sessions ongoing, and we’ll eventually get a solid solution around the data side.

2. Fix deployments & deployment management. Bosh isn’t straight forward or obvious in what it does, Cloud Foundry is easily the hardest thing to deploy with many dependencies. OpenShift is easier to deploy and neither of them actually have a solid management story over time. Bosh does some impressive updates of Cloud Foundry, and OpenShift has some upgrade methods, but still over time and during day to day operations there hasn’t been any clear cut wins with viewing, monitoring and managing nodes and data within these environments.

OSCON : Day 1, Windows Just Doesn’t Do Cloud Foundry… but, there’s a fix for that…

 The day before yesterday was day one, for me, of OSCON. I’d been out of town on business meet on Monday, so skipped out on the intro day. However the second day, my first, was a good time. There was already a good dose of “oh dear, I can’t attend ALL of the sessions I want to – BLASTED CONCURRENCY ISSUES!” problem. I was pondering the Intro to Erlang, then the backbone.js session, but in the end settled on Dr Nic’s @drnic session on how to deploy Cloud Foundry with BOSH.

Windows Just Doesn’t Do It

The first issue we ran into was actually the issue of prerequisites. About 30% of the audience was running Windows. To clarify the Windows question, there is no PaaS Solution that meets the following requirements:

– All Services Running on Windows
– Open Source Software
– Free or Cost

For those of you running Windows, the closest thing you can get – and I might add it’s a damn good solution – is Iron Foundry. But you’ll have to accept that there will still be some Linux involved for the Cloud Foundry parts that don’t run on Windows.

OSCON Ongoing

After the session I footed it over to the booths were a food & beer crawl of sorts was occurring, which I think might have been the first booth crawl, of two booth crawls. This was a good time, as the booth crawls usually are. It’s also fun seeing and learning about all the companies that are participating. Since everybody involved is ideally open sourced 100%, and most are at least a large percent open sourced, I always like hearing about the business models that are being used around the various products and services.

With that, this is day 1 coverage, I’ll leave you with a few photos of my first day:

The Chalk Art Wall o' Companies & Messages (Click for full size)
The Chalk Art Wall o’ Companies & Messages (Click for full size)
ESRI hanging out below the Samsung Sign... or is that perception? (Click for full size)
ESRI hanging out below the Samsung Sign… or is that perception? (Click for full size)
Riot Games just before the deluge! (Click for full size)
Riot Games just before the deluge! (Click for full size)

…and with that, I’ll have a follow up post on the following days following this post. Cheers!

PIE’s Third Class, You Better Keep an Eye on These Companies…

There are a number of new startups that have joined the third PIE Class. However there are a few that have stood out to me.

The first startup has to do with the IoT. IoT stands for Internet of Things. I’m a MASSIVE fan of what is being done with IoT. Personally I think it should be the space to watch in regard to the next big moves and big shifts in technology. From a market perspective, there’s some legitimate reasons to watch the IoT space from that view too.

Smart Mocha

With that, Smart Mocha caught my eye immediately. The description reads “Connects monitoring/measurement devices to the Internet of Things, enabling greater and more efficient access to critical data.” Their first product is Sense Simple, which is an “out of box” sensor network. This is interesting, being that existing systems that do what their Sense Simple offering does, are:

  • Dramatically more expensive, easily 10x or more.
  • Complexity in existing systems introduces vastly more points of failure, maintenance issues and other concerns.
  • Often not as capable for integration into other systems, Sense Simple already has “cloud control” – which is a control and device diagnostic tool to provide remote views of the sensor network.
  • All this, via a cellular gateway preconfigured and ready for logging data , multiple sensors, around temperature, humidity, barometric pressure, vibration, sound and more.

As I mentioned above, integration with existing industry standard sensors and the ability of the company to expand this product in the future already exceeds most of the existing offerings in the space. An example, just based on the cell gateway and cloud based control, provides a prime avenue to expand into the API space to provide even more ways to track, report and log data.

Orchestrate.io

The tag line says it all, “Add Features, Not Databases”. Orchestrate.io have designed a simple API, idiomatic client drivers as their site states. All of which enables you to get started trying tout Orchestrate.io rapidly. The goal of Orchestrate.io is to remove the need to manage a disparate array of databases and to instead focus on the data, what you want to do with the data and to develop solutions against that data. Being that it is offered as a Service akin to PaaS, IaaS and other styles of offerings, it provides the ability for you to pay for only what you use.

In today’s marketplace this is extremely ideal for a number of companies and becoming even more ideal for existing companies, legacy data and more. Got data? Check out Orchestrate.io and see if it works for you.

Summary

IoT:  As I was writing above, IoT is definitely shaping up to be a huge deal in the near future. Many industries are moving back to make progress in the physical realm akin to the migrations from ‘foot travel’ to ‘horse travel’ to ‘rail travel’ to ‘air travel’. We’re going to see some huge leaps here, maybe something along the lines of ‘human vision’ to ‘augmented vision’ to ‘perceptual planes vision’. Do you even know what ‘perceptual planes vision’ is? If not, get ready for the future, things might get bumpy! Smart Mocha looks to be positioned in a good place for impact.

Big Data, Data and more data: I’m under the impression I don’t need to elaborate on the notions of big data, but I will. Data has become a major differentiator, more so than even 5-10 years ago. Data has also become an even greater pain while becoming this major advantage. From genomic research to full tracked telemetry data to high volume high scale high quality printing, our new world of big data is here to stay. Orchestrate.io can help you wrap this realm up.

Disclosure: I don’t work for either of these companies, nor am I paid by the city of Portland, but they’re on my radar as I watch Portland’s startup scene and culture. I also live and breath the culture here, I am a Portlandian. Stay tuned for more in the coming weeks as other incubators and startups keep rocking and rolling here in the city of Portland, OR.

Resources:

Indirect Resources:

For good coverage of Portland’s artistic side, video quality and some of our current startups and companies, give this Techtown Portland video a watch.

Tech Town Portland from Uncage the Soul Productions on Vimeo.