A Few Notes on Riak 1.3 RC

Full context – Riak 1.3 RC came out just a couple dozen hours ago. RC stands for release candidate, which in turn basically means that version 1.3 is complete and any other additions will be for quick fixes or any issues that crop up. I’ve just started rolling a few new systems myself with this new version and hope you’ll join me in taking a hack at it. Let’s jump into a few reason why you’d want to leap into 1.3. You can read about the features below via the release notes also, but I’ve turned them into smaller bit size chunks below.

  • Giddyup in action!
    Giddyup in action!

    The first thing with the latest v1.3 has been the massive effort put into testing via the riak_test and the giddyup repos. Ongoing there will be a much easier way to move forward in features & quality. This is one of the reasons I love working for Basho, the whole team isn’t about smoke and mirrors with testing, they readily and diligently work on testing. Which to add context, remember we’re talking about distributed systems here, which aren’t exactly the easiest thing to test. One doesn’t just merely walk in and write unit tests and assume a distributed systems is tested. This moves us forward, and those that want to contribute and get involved more heavily in Riak now have a platform to dive in confidently when using these testing repositories.

  • Active Anti-Entropy – Alright, now we’re getting to the features with bad ass sounding names. Also referred to as AAE, this feature grabs bad replica data and begins a correction through read repair to protect data. It’s one more layer of protection against any type of data loss, disaster, bit rot, etc).
  • MapReduce Sink Backpressure – This one reminds me of tuning when setting up forced induction, AKA a turbo on a car. But I digress, I’ve snagged a description from the release notes for this feature, “Riak Pipe brought inter-stage backpressure to Riak KV’s MapReduce system. However, prior to Riak 1.3, that backpressure did not extend to the sink. It was assumed that the Protocol Buffers or HTTP endpoint could handle the full output rate of the pipe. With Riak 1.3, backpressure has been extended to the sink so that those endpoint processes no longer become overwhelmed. This backpressure is tunable via a soft cap on the size of the sink’s buffer, and a period at which a worker should check that cap. These can be configured at the Riak console by setting application environment variables” ….suffice it to say this helps out with map reduce in certain situations.
  • Additional IPv6 Support – Riak Handoff and Protocol Buggers listen ala IPv6 now. Nuff’ said.
  • Luke removal – Luke is completely and utterly gone now. Dead. Don’t look for Luke here.
  • Riaknostic – This is now part of the default featureset instead of separate tooling.
  • SmartOS 1.8 Packages – They’re available.
  • Health Check – This is a pretty awesome system that’s been added. Basically it watches the system and enables and disables services based on conditions. It’s super easy, just flick the switch in the app.config.
    [sourcecode language=”yml”]
    {enable_health_checks, true}
    [/sourcecode]
  • Reset Bucket Properties – A quickie definition from the release notes “The HTTP interface now supports resetting bucket properties to their default values. Bucket properties are stored in Riak’s ring structure that is gossiped around the cluster. Resetting bucket properties for buckets that are no longer used or that are using the default properties can reduce the amount of gossiped data.”

There were also a lot of PRs and more that you can check out on Github. These are the main key features that are now available and ready for use in 1.3. Check em’ out, feel free to contact me or any of the team to ask questions, let us know your 2 cents or otherwise banter about. Cheers! Sometime in the coming days I’ll have a quick start, akin to what’s in the docs, but with some specific ops on some IaaS Providers. So keep reading, coming up soon.

Happy hacking!  \m/   \m/

Riak is… A Whole Big List of Things

What is Riak? Who builds it? Who maintains it? Can I download it? How does it work? What are the features?

Here’s the start of answers to these questions and more.

First, the basic high level description:

Riak is an open source, highly scalable, fault-tolerant distributed database.”

That’s the first line you’ll read when checking out the product via the Basho product link. It provides good information, but here I’m going to add more to the definition without the need to dig around yourself. Maybe I can save you some time & provide some links directly to solid information in the docs. Kind of a “Cliff Notes” of Riak. Let’s take this feature by feature which will in turn get us to a definitive definition of what exactly Riak is.

Riak is Open Source.

Riak is built and contributed to by the community, with Basho being the steward and an active member that extends, builds and provides support for additional products. The avenues to reach the Riak Open Source Community members is pretty straight forward, following known avenues of communication. Hit us up on the email list, especially feel free to contribute & ask questions via the Github Basho organization, there is the Basho Riak Blog, the weekly recap and jump into the IRC chat room #riak on freenode. Oh, and there’s a twitter feed @basho.

So what exactly does this get you, when you become a user or contributor of Riak? The entire community is behind you, will help you get started using Riak and provide help whenever you run into problems. If you want SLAs or 24 hour support Basho can provide this for you. But for bugs, issues, queries, searching and all sorts of other related development questions there is the community. An open source community like this is passionate, which means you’ll have support like no closed source company will ever provide you, and absolutely no closed source product’s community will provide you. We’re talking about a different level of interest, passion and levels of personal involvement.

Riak is a key value based database store.

Riak is a key value store. What exactly is a key value store? It’s pretty simple and you’re probably already familiar with what a key value store is. A key value is made up of two pieces of data, the first is the identifier for the second element within the data structure. This gives a system or developer using key value storage a schema-less way of working with data.

Riak is designed for highly distributed environments.

This type of distributed isn’t the “we put one database over here and one database over here and you gotta figure out how they work together” type of distribution. So this isn’t some of that oddball pretend stuff Oracle keeps hoisting on people. This is the honest to goodness distribution of the sort, when one node goes down you don’t blink, you don’t stop eating dinner, you don’t sweat it. You just continue onward with life knowing full well that you’ll just spool up another node when you need to.

Riak is master-less, with no single point of failure.

This is one of self explanatory features. But what does a master-less system provide us? One thing is no single point of failure. Being that all nodes can act autonomously to work around the loss of one or more nodes it also helps add to the high availability of the system.

Riak is fault tolerant, like a disk drive you wish was real.

Ever have a backup disk drive? What? You don’t have one of those? Ugh. Ok, so imagine you had a backup disk drive that had an unfortunately high failure rate. Well, why, because you know, they have an oddly high failure rate. If you do backups like good practices dictate, eventually you’ll end up with some dead drives.

RAID, both software and hardware, are built specifically to deal with this type of failure. With a distributed system like Riak, it bumps the level of abstraction above software or hardware RAID, enabling another level of even greater fault tolerance. Not to remove the relevance of RAID capabilities, but with a multi-node system like Riak, you can easily remove nodes and swap them out as needed, keeping costs down by using simple drives in simple machines. If you want to, you could indeed get higher I/O machines and faster drives, but it isn’t necessary to insure fault tolerance in a Riak Database System.

Riak scales, with hot swappable nodes enabling zero downtime.

The ability to commit hot swappable changes while in the midst of operating starts at a very low level for Riak. The language used to build Riak, Erlang has the ability to change pieces of an application system in realtime built into the precepts of the language. This provides, at the core, the inherent capability to change out systems, and by proxy of architectural design, the ability for nodes in Riak to be changed out simply by removing them from a cluster ring. Once that is done it is just as simple to add another node or nodes back into the cluster ring, enabling a number of additional practices around upgrades, hot swaps for failures, or even version changes.

Riak can be used as a building block for distributed (aka cloud) infrastructure.

The concepts and contractual components that Riak Database is built on are available for use via the Riak Core Project. If you’re looking into starting a project around distributed systems this is a great place to get start. Also be sure to do a general web engine (re: google) search for “riak core” and you’ll find lots of material around the project, and projects people have started with the project as a base. I’m currently in the process of putting together one of these projects myself.

Riak is eventually consistent.

The term eventually consistent is becoming more and more common place. Riak is one of the many systems, that inherently often apply to distributed systems, that use the concepts of eventual consistency. The idea, is that even though all nodes may not immediately receive a new piece of data, or updated piece of data, they eventually will receive that update and by synchronized with the cluster ring of nodes. This goes back to the equality of nodes and removal of the master-less concepts, providing the availability and other capabilities, with some trade off in the synchronization of data through eventual consistency.

In Summary

That’s round one for the many features of Riak. I’ll be adding more in the future, but for now this is a good starting point in knowing about and knowing what Riak is, what it can be used for, and how it might help you extend, maintain or invent the next great piece of technology.

Alright Stop Waiting, CorrugatedIron v1.0 .NET Client released for Riak!

I’ll kick right off with all the specifics:  Jeremiah did a blog entry on today’s release titled “Just one more thing…  Introducing Corrugated Iron v1.0“.

Send a congrats out to the team duo of OJ @TheColonial & Jeremiah @peschkaj via Twitter. Check out the .NET Rocks Podcast with Jeremiah talking with Carl and Richard about storing data in Riak. Also check out Adrian Hills’ article on getting up and running via Nuget with Corrugated Iron in Visual Studio, ping him on Twitter @AdaTheDev.

So no excuses in .NET land to write some apps that are hard core data centric and capable based on the power of Riak! There are tons of features. You can read about them yourself via the README.md file on the CorrugatedIron Repository, but I wanted to post the features right here so you get an idea of the feature rich capabilities of the library. In addition, it does indeed work on Linux & OS-X with Mono. So don’t let Windows get in your way! 🙂

Current Features

♥: denotes availability of both blocking and asynchronous APIs
«: denotes availability of both streaming and non-streaming APIs

  • Riak cluster support:
    • One or more nodes in the cluster.
    • Load-balancing and pooling of connections across the nodes.
      • Currently only round-robin is supported, more strategies to come later.
    • Per-node configuration for:
      • Host Name (purely used for identification).
      • Host Address.
      • PBC Port.
      • HTTP/REST Port.
      • Pool Size.
      • Timeout parameters.
  • Server ping.    ♥
  • Get server information/version.    ♥
  • Simple Get/Put/Delete operations.    ♥
  • Bulk Get/Put/Delete operations.    ♥
  • List buckets.    ♥
  • List keys.    ♥  «
  • Semi-fluent Map/Reduce.    ♥  «
  • Link walking.    ♥
  • Delete buckets.    ♥
  • Set/Get bucket properties.    ♥
  • Batch operations on a single connection.
    • Each time a Client function is called resulting in communication with the Riak cluster, a connection is pulled from a pool on a given node. In most use-cases this functionality is fine as it is often single-shot calls that are made. There are, however, cases where many operations will happen at once. Rather than forcing the user to make multiple calls to the client, resulting in multiple connection acquisitions behind the scenes, the user can use the Batch interface to make many calls on a single connection. This also reduces the overhead of setting the client ID on each call.
    • Because a batch operation reuses a single connection only a subset of the client API is available for batch actions. The functions that are excluded are the asynchronous functions.
  • Graceful degrades to HTTP/REST API when the request isn’t supported via Protocol Buffers.
  • Configurable via web.configapp.config or a custom configuration file.

CorrugatedIron works with .NET 4.0 on Windows and Mono on Linux and OSX.

SITREP – Stoking Iron Foundry, Thor Hammering & Joining… ?

What have I been up to? Here’s a quick recap. You may want to get involved with some of these projects!

Iron Foundry & Tier 3 Web Fabric

Iron FoundryBack when I left the kick ass team at Russell Investments in Seattle I stepped directly to bat as team lead at Tier 3. My job, get a PaaS built on Cloud Foundry and extending that with Iron Foundry. It was an ambitious effort that would provide the most extensive framework and language support available from any PaaS Provider on the market.

Well, we did it, thanks to the capabilities of Cloud Foundry Community, the great minds of Jared Wray @jaredwray, Luke Bakken, Eric Lee @saintgimp, Cale Hoopes & the rest of the Tier 3 Team! I was able to add this to my list of successes. We had some bumps, some collisions, a brick wall or two and other scheduling problems – ya know, the standard things that happen on a project. But in spite of it all, we got the Web Fabric released – and it continues to be the only PaaS available with such a wide framework and language support. It ranges from Ruby on Rails, Erlang, Node.js & Java to .NET! If you’d like to check out the open source PaaS of Cloud Foundry & Iron Foundry both projects are always looking for participation & contributions!

Thor Brings the Hamma!

After the release of the Tier 3 Web Fabric I started the search for a wicked smart and capable OS-X / Cocoa Coder – it seems their availability is pretty limited these days! Well I finally lucked out and found Benjamin van der Veen (@bvanderveen, thanks for the intro Selena @selenamarie!) to help me get started on the Thor Projects. There’s the Cocoa Thor Project & the Windows 7 WPF Metro based Thor .NET Project that we’re wrapping up with v1 releases coming really soon. To check out more on these projects that I’ve lead & coded on check out the code bases & information, all linked on ironfoundry.org. The projects are open source, so feel free to jump in and help out or fork & submit pull requests. The team will be happy to review & discuss ASAP.

While we’re wrapping these projects up right now, I’ll actually be continuing on and supporting the projects: Thor & Thor .NET. I will continue to be involved, as I was saying, in a number of ways in the PaaS space. So don’t think I’m disappearing form that realm!

Basho Sings my Song

I’ve been keeping track of Basho for a while now. Riak caught my interest many months ago as a really well built, well thought out & advanced distributed database. As you might guess, being into the whole “cloud computing” industry, I’m just ever so slightly interested in distributed systems. The other thing that I’m a huge fan of, which Basho does, is heavily support and involve itself in the open source software community and movement. The icing on the cake, was their diverse use of systems and language use around Erlang. All things that are massive wins.

Bailey's Taproom
Bailey’s Taproom

Well during a random conversation with Eric @coderoshi at Bailey’s Tap Room & then attending the RICON 2012 Conference (article here and pictures) I spoke to some of the team and found out they were looking for some particular skill sets. Well it just happened that I was keenly interested in meeting those skill set requirements! So December 1st I’ll be joining the Basho team full time as developer advocate, evangelist, messenger or such for the northwest working with a few people you may know such as Mark Phillips @pharkmillups, Andy Gross @argv0 (thanks for the intro James @wattersjames), Eric Redmond @coderoshi, Shanley Kane @shanley, Casey Rosenthal @caseyrosenthal and many others. Simply, I’m freaking stoked.

How This Helps You Help Me Help You

Alright, so it’s great but how can I help you in your day to day? What data do you work with? Do you work with a data scientist? Are you a data scientist? Do you work with huge sets of data, many objects, large objects? I want to know about your data usage and data problems, because there’s a good chance we’ll have more than few things to discuss. Here’s some ways I can help you, help me, help you. Ping me if you’re interested in…

  • talking about your data usage at the monthly Riak user group.
  • coding, pairing & otherwise learning Erlang and the monthly Erlang group.
  • interested in coding, deploying and inventing new paradigms and patterns of data storage.
  • interested in pairing up to learn how to deploy, migrate, upgrade or otherwise use NoSQL solutions – namely Riak.
  • interested in Ruby on Rails, Node.js, Map Reduce, .NET, Java, PHP and how these things can and do work against data in everything from relational databases to the new echelon of NoSQL databases.

I hope to hear from you soon and see you at an upcoming user group, cheers!