A few days ago Troy Howard, Jeremiah Peschka and I all traveled via Amtrak Cascades up to Seattle. The mission was simple, Jeremiah was presenting “Riak in a .NET World”, I was handling logistics and Troy was handling video.
So I took the video that Troy shot, I edited it, put together some soundtrack to it and let Jeremiah’s big data magic shine. He covers the basics around RDBMSes, SQL Server in this case but easily it applies to any RDBMS in large part. These basics bring us up to where and why an architecture needs to shift from an RDBMS solution to a distributed solution like Riak. After stepping through some of the key reasons to move to Riak, Jeremiah walks through a live demo of using CorrugatedIron, the .NET Client for Riak (Github repo). During the walk through he covers the specific characteristics of how CorrugatedIron interacts with Riak through indexs, buckets and during puts and pulls of data.
Toward the end of the video Joseph Blomstedt @jtuple, Troy Howard @thoward37, Jeremiah Peschka @peschkaj, Clive Boulton @iC and Richard Turner @bitcrazed. Also note, I’ve enabled download for this specific video since it is actually a large video (1.08GB total). So you may want to download and watch it if you don’t have a super reliable high speed internet connection.
Today I’ve officially kicked off a new project from my notebook of projects based around building a Riak admin, data manipulation, reporting and news tool for Windows 8. If you want to jump right to the project, here’s the Github Pages Site, the Github Junction Repo and eventually I’ll have it listed in the Windows 8 Store for download. Yes, it’ll be free as in beer, it’ll all be Apache 2.0 Licensed and the project is open to contributors and others that want to jump into things. There’s also a quick intro for how I setup the “Windows 8 Logos, Badges & Splash Screens of Riak“.
So now that I’ve provided the links, here’s a quick intro to each of the application sections, what this application is for, where the workflow for contributions will be and what the next steps are. Trust me, I roll easy, I’ll be working as hard as I can to make pull requests easy peasy, keep the issues down to workable contributions and the whole “this is a good OSS project”.
Riak Junction Application rocking on the Windows 8 desktop with a full tile!
Juncture Divisions
The juncture application should be split into several key components, or application divisions of functionality. I’ve broken each out with a basic description. If you just want to watch a video where I outline each division, play the video below for a quick 5 minute intro to the application and the idea behind it all.
A quick run through of the first sample UI.
Call the Doctor! (Administration & Maintenance)
This part of the application would provide an interface for all the general administration and maintenance needs around individual nodes and around the overall cluster of nodes. The ability to add, remove and generally administer everything that is available via the riak-admin command line interface.
Time Travel That Data (Performance Benchmarking)
This section of the application will provide the ability to benchmark the timing of data in and out of a cluster. In addition it should show standard benchmarking similar to that which is offered with the basho_bench project.
Love of the Data (Reporting)
This division of the application would be focused on reporting. I’m not sure what exactly that would entail, but something with charts, graphs and pulling together trending points of some sort. If you have ideas and want to work on this part of the application, weigh in!
Golfing With Your Data (Query, Put, Deletes, Etc. Handling the CRUD)
The application will have an interface to provide access to add and remove data, as well as viewing the data that is available within a cluster. The primary means for implementing this part of the application will be with the CorrugatedIron Project. It’s a library available via Nuget that @peschkaj and @TheColonial have put together.
News! News! News! (News… RSS Feed Reader)
The idea is that this will provide a quick and easy way to get familiar with Windows 8 dev and the project overall. I’m aiming to eat the Basho blog feed and provide it as key highlights for the application with future abilities around mining other RSS feeds or such and having those fed into a ?? Riak Cluster? Again, everything is open to change, addition or removal! So jump into the project and let me know your thoughts.
So no excuses in .NET land to write some apps that are hard core data centric and capable based on the power of Riak! There are tons of features. You can read about them yourself via the README.md file on the CorrugatedIron Repository, but I wanted to post the features right here so you get an idea of the feature rich capabilities of the library. In addition, it does indeed work on Linux & OS-X with Mono. So don’t let Windows get in your way! 🙂
Current Features
♥: denotes availability of both blocking and asynchronous APIs «: denotes availability of both streaming and non-streaming APIs
Riak cluster support:
One or more nodes in the cluster.
Load-balancing and pooling of connections across the nodes.
Currently only round-robin is supported, more strategies to come later.
Per-node configuration for:
Host Name (purely used for identification).
Host Address.
PBC Port.
HTTP/REST Port.
Pool Size.
Timeout parameters.
Server ping. ♥
Get server information/version. ♥
Simple Get/Put/Delete operations. ♥
Bulk Get/Put/Delete operations. ♥
List buckets. ♥
List keys. ♥ «
Semi-fluent Map/Reduce. ♥ «
Link walking. ♥
Delete buckets. ♥
Set/Get bucket properties. ♥
Batch operations on a single connection.
Each time a Client function is called resulting in communication with the Riak cluster, a connection is pulled from a pool on a given node. In most use-cases this functionality is fine as it is often single-shot calls that are made. There are, however, cases where many operations will happen at once. Rather than forcing the user to make multiple calls to the client, resulting in multiple connection acquisitions behind the scenes, the user can use the Batch interface to make many calls on a single connection. This also reduces the overhead of setting the client ID on each call.
Because a batch operation reuses a single connection only a subset of the client API is available for batch actions. The functions that are excluded are the asynchronous functions.
Graceful degrades to HTTP/REST API when the request isn’t supported via Protocol Buffers.
Configurable via web.config, app.config or a custom configuration file.
CorrugatedIron works with .NET 4.0 on Windows and Mono on Linux and OSX.
The Iron Foundry Team are big advocates of open source software. We write code across all sorts of languages, just like many of the development shops out there do. Sometimes we’re heavy on the .NET, other times we’re all up in some Java, Ruby on Rails, spooling up a Node.js Application or something else. So keeping with our love of open source and our polyglot nature we’ve created the Thor Project with three distinct apps.
Before jumping into the applications though, a little context for what and where Thor is in the grand scheme of things. We need to roll back to the Cloud Foundry Project to get into that. The Cloud Foundry Project is an open source project built around software for PaaS (Platform as a Service) which can be used to build your own PaaS internally or externally, in a cloud provider or directly on hardware. It’s your choice how, when and where you want to use it. For more context on PaaS check out my previous entry “The Confusions of IaaS, PaaS and SaaS“.
Thor Project
Cocoa for OS-X
Thor Odinson, God of Thunder
You know who Thor is right? He’s this mythic Norse God, also known as the God of Thunder. Since we’re all about bringing the hamma we welcomed Thor into our team’s stable of applications. So starting immediately we’ve released Thor into the realms for contributions and fighting the good open source software battle! If you’d like to join the effort, check out the github project and feel free to join us!
Technically, what is the Thor Application? This is a Cocoa Application built for OS-X that is used for managing, deploying and publishing applications to Cloud Foundry enabled and or Iron Foundry extended PaaS Environments.
.NET for Windows 7
The .NET Metro version of the Thor Application is also released via github with a provided installer. We’ve almost taken the same path, except of course for the very different UX and UI queues with Windows 7 and the Metro UX design guidelines.
WinRT for Windows 8
I wasn’t really sure what to call this version. Is it Metro or WinRT or Windows 8 or something else? Anyway, there is a project, it is albeit empty at this point, but it is the project where the Windows 8 version of Thor will go! For now get the Windows 7 version and install it on Windows 8, it won’t have touch interface support and things, but should work just like a regular application on Windows 8.
To get started with these, generally you’d just clone the repo and do a build, then get started checking out the code. There is one catch, for the OS-X version you’ll want to pull down the sub-modules with the following command.
Once you do that in XCode just make sure to then select the right project as the starting build project.
…then when the application is launched…
Thor Running in OS-X
I’ll have more in the coming days and weeks about Thor & Iron Foundry. For now, check out the blog entry on the Iron Foundry Blog and subscribe there for more information.
I was speaking with Krishan Subramanian (@krishnan) and Adam Seligman (@adamse) today. I love talking to these guys. They’re both smart, intelligent and upbeat guys. They see the positive things we’re all working toward and accomplishing in the technology space, specifically around PaaS, Cloud Computing and around the cultural implications of stronger technology communities, involvement of individuals. We all can see the positives, of how the industry is moving forward so that corporations aren’t the only enablers that are juxtaposed against developers or consumers but instead act to serve consumers based on the progress that individuals make themselves. There’s so much to do and so much progress to be made, the venders can simply follow the community and step up to provide points of leadership.
Absolutely great talking with these guys…
On that topic, what is it that we discussed that has me so motivated? Well there’s a few things that I’m done with and I’m going to make every effort to just throw away the trash. Here’s a few of these things that we discussed and I challenge everybody out there, drop the trash talk and let’s move forward because there is a LOT of awesome things to accomplish. Here’s the two things I’m just dropping… cold. No reason to discuss them anymore.
Toss the language and framework religious wars. It is far simpler than it is sometimes perceived. We have a polyglot industry now where we can easily use the right tool for the job, the right framework, or the language that handles our particular domain the best. There is literally no reason to argue about this anymore. Of course we can talk semantics, debate best use cases, and of course we’ll talk accomplishments and what various things do well. That’s exactly what the focus should be on, not the harping on my X is better than your Y nonsense.
The culture war is basically over. Sure there are the hold outs that haven’t gotten a clue yet. But it’s an open source world at this point. Even the dreaded and horrible Oracle has generally conceded this and is frantically waving its marketing arms around trying to get attention. But at the core, mysql, java and the other things that they’ve purchased they’re keeping alive. They’re active participants in the community now, albeit in a somewhat strange way. Considering that even Oracle, Microsoft, Apple and so many others contribute back to the open source community in massive ways, that war can be considered won. Victory, the community and every individual in that community!
Lockin is basically dead. The technological reasons to lock in are gone, seriously. There’s some issues around data gravity that are to be overcome, but that’s where a solid architecture (see below) comes in. Anything you need can be contributed to and derived from the development community. Get involved and figure out how technology can be a major piece of your business in a positive way. If you design something poorly, lock in becomes a huge issue. Use the rights tools, don’t get into binding contracts, because in the polyglot world we’re in now there’s no reason to be permanently locked in to anything. Be flexible, be where you need to be, and make those decisions based on the community, your support systems, and your business partners. Don’t tie yourself to vendors unless there is mutual reasons to do exactly that. Lock in is a dead conversation, just don’t, time to move on.
So what are the key conversations today?
Ecosystem Architecture – If you’re deploying to AWS, Heroku, Tier 3, AppFog or Windows Azure it all boils down to something very specific that will make or break you. Your architecture. This is where the real value add in the cloud & respective systems are, but there are many discussions and many elements of the technology to understand. This is a fundamentally key conversation topic in the industry today. Pick this one up and drop the other trash.
Movement & Data Gravity – How do you access your data, how do you store it, where and how do you derive insight from that data? This is one of the topics that came up in our discusssion and it is huge. The entire computer industry basically exists for the reason of insight. What should we eat today, how do I shift my investments, how is my development team doing, what’s the status of my house being built, where is my family today and can I contact them! All of these things are insights we derive from computer systems. These are the fundamental core reason that computers exist. As an industry we’re finally getting to a point were we can get some pretty solid insightful, intelligent and useful information from our systems. The conversation however continues, there is so much more we can still achieve. So again, drop the wasteful convo and jump on board the conversations about data, information and insights!
Community Involvement – I’ve left the key topic for last. This is huge, companies have to be involved today. Companies aren’t dictating progress but instead the community is leading as it should. The community is providing a path for companies to follow or lead, but the community, the individuals are the ones that are seen and known to be innovating. This is so simple it’s wild that it is only now becoming a known reality – companies don’t innovate, people do. Companies don’t involve, people do. Individuals are the drivers of companies, the drivers of Governments, they’re the ones driving innovation and progress. The focus should now and should have always been on the individuals and what they’re working toward to accomplish. So get involved, get the companies involved as a whole and keep the semantic ideal of individuals and the progress they can make core to the way you think of communities. The idea of the “company” innovating is silly, let’s talk and build community with the people that are working around and innovating with these technologies.
Of course there are more, I’d love to hear your take on what the conversations of today should be about. What do we need to resolve? How do we improve our lives, our work and the efforts we’re working toward on a day to day basis?
You must be logged in to post a comment.