One of the core tools in the belt of the distributed database is consistent hashing. In Riak this is especially true, as it stands at the core of a Riak Cluster. Hashing, using a hash function, is an algorithm that maps data to variable length to data that’s fixed. In other words, odd things like the name of things mapped to integers. Consistent hashing is a special kind of hashing that provides the pattern for mapping keys and all related functionality around a cluster ring in Riak.

Consistent hashing was originally devised by David Karger, a professor of computer science at MIT (Massachusetts Institute of Technology). He’s also known for Karger’s Algorithm, a Monte Carlo method that computes the minimum cut in a connected graph (graph theory related stuff). Along with these developments he’s been part of many other efforts and contributed to computer science in many ways.

Remapping, Mapping and Keeping Distributed (& Available)

One key property of a consistent hash is that it minimizes the number of keys that must be remapped. With a regular hash changes, the entire key hash must be remapped.

Consistent hashing is based around mapping each object to a point of a circle. The system maps each storage bucket to pseudo-randomly distributed points on the edge of this circle.

The system finds where to place the object based on the key on the edge of the circle. It then walks the circle falling into the first bucket it finds. This results in the buckets containing the resources between its point and the next bucket point.

When a bucket disappears for any reason, the pseudo randomly mapped objects will now get re-mapped to different buckets. When a bucket appears, such as becoming available again or being added, a similar process occurs.

The Basho Docs describe in brief that,

“Consistent hashing is a technique used to limit the reshuffling of keys when a hash-table data structure is rebalanced (when slots are added or removed). Riak uses consistent hashing to organize its data storage and replication. Specifically, the vnodes in the Riak Ring responsible for storing each object are determined using the consistent hashing technique.“

NOTES: This is not a single blog entry topic by any means. This is merely a cursory look at consistent hashing. This entry I aimed to provide a basic description and coverage of the actions around consistent hashing. For more information and to dive even deeper into consistent hashing I’ve included a few links that have extensive information on the topic:

• Consistent Hashing – Basho Docs
• Wikipedia Article
• For comparable hashing, check out Distributed Hash Tables (DHTs).
• For a more thorough description and topic coverage check out the Riak docs.

Setting up Riak to test out, prototype against, develop and use in a general way is extremely easy. Just setup a devrel on your local development machine. This is however limited to certain *nix based operating systems, so Windows as a dev platform is out – but not completely. Get a virtual machine running on Ubuntu, RHEL or some other Linux instance and you’re ready to go. What I’ve put together here is an example of getting a devrel up and running with an Ubuntu Virtual Machine.

Step 1: Get the basic reqs installed.

• Install Erlang R15B01.

  	sudo apt-get install build-essential libncurses5-dev openssl libssl-dev git
  	wget http://erlang.org/download/otp_src_R15B01.tar.gz
  	tar zxvf otp_src_R15B01.tar.gz
  	cd otp_src_R15B01
  	./configure && make && sudo make install
  

• Install Riak from source.

  curl -O http://downloads.basho.com.s3-website-us-east-1.amazonaws.com/riak/1.3/1.3.1/riak-1.3.1.tar.gz
  tar zxvf riak-1.3.1.tar.gz
  cd riak-1.3.1
  make rel
  

• Install 4 Riak nodes to #devrel.

  make all
  make devrel DEVNODES=4
  cd dev
  

Step 2: With each of the nodes, now join and build the Riak cluster.

• Start each node.

  dev1/bin/riak start
  dev2/bin/riak start
  dev3/bin/riak start
  dev4/bin/riak start
  

• Check to determine that the riak services are running.

  ps aux | grep beam
  

• Add each node to a single node.

  dev2/bin/riak-admin cluster join dev1@127.0.0.1
  dev3/bin/riak-admin cluster join dev1@127.0.0.1
  dev4/bin/riak-admin cluster join dev1@127.0.0.1
  

• Set and get the cluster plan.

  dev2/bin/riak-admin cluster plan
  

  NOTE: This plan can be run from any of the instances.

• Last, commit the cluster plan.

  dev2/bin/riak-admin cluster commit
  

  NOTE: The commit can also be run from any of the instances.

First things first – If you haven’t registered for the Polyglot Conference yet, get registered ASAP as it could sell out!

I have a workshop coming up on Friday of this week at the Vancouver BC Polyglot Conf. In addition I’ll be attending the conference itself and looking to have a lot of conversations and open spaces sessions around distributed systems, technologies, platform as a service, other *aaS and generally anything for modern, distributed, scalable architecture and systems. Hopefully I’ll get to converse with all of you about as many topics as possible  – it’s always a great time to dig into these topics at conferences like these. It’s where the real next steps in technology come from!

In the coming days before and after the workshop I’ll be blogging material I have put together for the workshop. Previously I’ve put together one blog entry on backing up distributed systems ala the Riak variety. Check it out “Backup Riak – Learning About Distributed Databases :: Issue 001“.

The Workshop, for those of you interested in surviving a Zombie Apocalypse (or just learning about Riak) can sign up still, there’s a few spaces left! Below is a the basic outline of what we’ll be touching upon. There’s a LOT more topics beyond this, but we’ll save those for open space sessions. If there’s anything specific you’d like me to cover, feel free to throw a comment on the blog and I can easily add 1-2 topics of interest if attendees would like!

• Introducing Riak, a database designed to survive the Zombie Plague. Riak Architecture & 5 Minute History of Riak & Zombies.

Architecture deep dive:

• Consistent Hashing, managing to track changes when your kill zone is littered with Zombies.
• Intelligent Replication, managing your data against each of your bunkers.
• Data Re-distribution, sometimes they overtake a bunker, how your data is re-distributed.
• Short Erlang Introduction, a language fit for managing post-civil society.
• Getting Erlang

Installing Riak on…

• Ubuntu, RHEL & the Linux Variety.
• OS-X, the only user centered computers to survive the apocolypse.
• From source, maintained and modernized for humanities survival.
• Upgrading Riak, when a bunker is retaken from the zomibes, it’s time to update your Riak.
• Setting up

Devrel – A developer’s machine w/ Riak – how to manage without zombie bunkers.

• 5 nodes, a basic cluster
• Operating Riak
• Starting, stopping, and restarting
• Scaling up & out
• Managing uptime & data integrity
• Accessing & writing data

Polyglot client libraries

• JavaScript/Node.js & Erlang for the zombie curing mad scientists.
• C#/.NET & Java for the zombie creating corporations.
• Others, for those trying to just survive the zombie apocolypse.

Cheers!

This is it, last string of introductions. Hope you’re registered.

Adam Baldwin is presenting…

Introducing NodeSecurity.io

Adam Baldwin

Adam Baldwin is a web app hacker, team lead at ^Lift Security and the CSO for &yet. Adam has presented at various security & dev conferences in the past including, DEFCON, Djangocon, Toorcamp and RealtimeConf.

The node.js community is growing at an amazing rate. At the time of writing there was 27,757 modules publised on npm. Have you ever stopped to think just what you are putting into your project when you npm install somebody else’s module? Do you trust that code? This is an insane project to find out the answer to that question.

This talk will introduce the nodesecurity.io project, it’s goals, current results in hopes of inspiring involvement and receiving feedback directly from the node community!

James Halliday is presenting…

beep boop

James

Oh hello. I write too much code. I co-founded browserling. Here are some pretty pictures.

Unix philosopher and methodological reductionist etc.

Learn how to make computer sounds in node and the browser with the same api.

Using just a single function that takes a parameter t, time in seconds, and returns an amplitude between -1 and 1, inclusive, you can create music!

You can use this basic approach to write songs and synthesizers. In javascript. Yay!

Ryan Jarvinen is presenting…

Clustering Node.js on OpenShift

Ryan

Ryan Jarvinen is an Open Platform Advocate working with RedHat’s OpenShift team. He lives in Oakland, California and is passionate about open source, open standards, open government, and digital rights. You can reach him as ‘ryanj’ on twitter, github, and IRC.

Learn how to automate builds, deployment tasks, and application scaling as we use OpenShift’s platform architecture on-demand to build your own git-based release pipeline, including: development, testing, staging, and cloud-scaling production environments for node.js.

Slides:

• http://darkslides-rjdemo.rhcloud.com
• https://github.com/ryanj/darkslides/

Posts:

• https://www.openshift.com/blogs/secret-free-source-on-paas
• https://www.openshift.com/blogs/release-management-in-the-cloud

---

An adaptation of this talk was presented recently at HTML5DevConf in SF – http://html5devconf.com/sessions.html#r_jarvinen

Intro to Sails.js

Mike McNeil is presenting…

Mike

Mike autobiogrophies himself as, “I’m Mike, a developreneur based out of Austin, Texas and connoisseur of fine code. I’m also the creator of Sails.js, the open-source BaaS framework which allows front-end developers to build robust, scalable APIs using only JavaScript.

My first startup was in social television, where I saw the need for more efficient, easy-to-use solutions for realtime social features. Because of that, I got involved in Node.js early on, and after building a few early apps, recognized the need for an MVC solution to normalize patterns. Early last year, I founded Balderdash, a UX-focused mobile and web studio, which has given me an excellent opportunity to build out and utilize Sails.js in production.”

Sails.js makes it easy to build custom, enterprise-grade Node.js apps. It is designed to resemble the MVC architecture from frameworks like Ruby on Rails, but with support for the more modern, data-oriented style of web app development. It’s especially good for building realtime features like chat.

Sails empowers UX and design teams to build hi-fi prototypes in no time without waiting for the back-end to be finished. This means focusing more resources on the user experience, which means better products. One Sails.js project at a time, companies move their legacy architecture over to a simpler, more efficient Node.js cloud. Each new client-side code base is more maintainable, since it’s built using the universal language of the internet: a RESTful JSON API.

Chris Dickinson is presenting…

Implementing Git in JavaScript & the Browser: A Case Study

Chris

Chris describes himself as “I make silly things with JavaScript: I particularly love bit-twiddling and WebGL-based projects. I live in Portland OR and work at Urban Airship as a JavaScript engineer.”

Git is one of my favorite things to hack on. It’s long been my goal to get a working (workable?) implementation of git running in pure JS, in the browser. My first attempt two years ago failed; and for a long time I’ve let the thought bounce around in the back of my head.

Spurred on by the recent interest in js-git, I recently restarted the journey towards an in-browser git, in order to help creationix deliver the best possible js-git. Newly armed with browserify and the small-module ethos, I’ve come much closer to a working git in browser and Node, and in the process have really put browserify and its shims through their paces.”

This talk will be comprised of:

A quick intro to the git object model and transport protocol How browserify and the small module ethos have enabled great successes in the project. Difficulties encountered in the process, both with Node.JS itself and with browserify, and how I’ve worked through them. How I’ve diagnosed and worked through various performance issues. Where is this project going?

Horse JS is presenting…

JavaScript, This is Confusing

….Horse JS Tweets, nuff’ said.  Horsing around…

Are you signed up?  BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

…and the fifth iteration of Node PDX Introductions!

Zach Bobb is presenting…

Building a Computer In Your Browser

Zach the TriMet Ticket Man!

Zach is a mobile engineer with GlobeSherpa working hard to bring you the app that will let you buy TriMet tickets on your phone.

Want to learn how computers work under the hood while learning Web technologies? Come learn about the Von Neumann 51, a series of tutorials that will take you through building a computer from the ground up in JavaScript, HTML5, and CSS3.

The talk will start by laying out some of the motivation behind the project and then demonstrate some of the components people will build through the tutorials: logic gates, adder circuits, displays, etc. The end of the talk will demo a prototype of the vN51.

Some of the technologies in this talk are:  D3

A JavaScript library for dynamic data-driven SVG graphics HTML5 Canvas

Pixel manipulation for display output

Paul Jungwirth is presenting…

Handling Errors with Cluster and Domains

Paul

Paul will jump into,

“Node’s callback pattern makes error handling difficult: throwing an exception kills the entire node process, terminating all current requests, and every callback initiates a new stack, so stacktraces are terse and don’t indicate how you got where you died. You can solve these problems using some newer features of Node called clusters and domains. This talk with explore using these tools for better error handling.“

Forrest Norvell is presenting…

Do as I say, not as I do: Node in the real world

Forrest

Forrest is a software engineer who has spent the better part of the last two years working with Node full-time, first working on bringing Singly’s Locker Project up to web scale, and then exploring the wilds of monitoring the performance of Node applications for New Relic. He has learned an unreasonable amount about JavaScript, V8, libuv, and making it all work well in the real world.

Forrest has spent the last year plumbing the depths of Node for New Relic, as he brings support for Node to New Relic. It’s a process that has required ingenuity, guile, and the willingness to break a lot of otherwise very sensible rules. He will take you on a tour of some of the more powerful but dangerous tools in the JavaScript toolbox: monkey patching, working with Node’s internals, and mixing synchronous and asynchronous code without setting your hair on fire. Charlie Key is presenting…

Building a Multiplayer World for Pillow Pets

Charlie

Charlie is Co-founder of Modulus, a premier Node.js hosting solution. He has spent the last six-years working in the software, where he has created over a dozen production websites and applications for many global brands.

He is also a core contributor on Pulse game engine and one of members of the team who developed Pillow Pets World.

The talk will dive into the design goals, architecture and end result of creating this massive virtual world. Come see how Pulse (a HTML5 game engine) and Node.js were combined to create a fast, expandable, mobile ready world. Targeting a game at 10,000,000 people? Then come and find out how Pillow Pets World was built. Pillow Pets World is a virtual world built for millions of kids. Scalability and performance were aspects that were included from the start.

The technologies included in this talk are:

Pulse

HTML5 Game Engine Node.js

Scalable small servers Socket.io

Real-time communication using Web sockets Redis

Small in memory storage used for pub/sub communication between servers

Are you signed up?  BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!