Tag: distributed

Orchestrate.io JavaScript Client Library

Today I’m starting a project working with Orchestrate.io’s API & open source software collaborations. More about the project in a moment, let’s get up to speed on what I’ll be including in this project. My main focus is to build a client library to access Orchestrate.io. During building this I’ll dive into the key value, graph and other storage mechanisms that the client library will provide. Beyond that, I’ll take a stroll through building an NPM library and the pertinent JavaScript the library. So buckle up, we’re going on a code slinging hash writing hacking session.

Over the course of putting together this material, I’ll be posting most of the core material on Orchestrate.io’s blog, so subscribe for updates as they come out. Feedly is a good option, connect via searching for “orchestrate.io” or navigate over to the Orchestrate.io blog itself. 😉

Orchestrate.io
JavaScript

Project Effort Context

During building the client I’ll take a dive into who, what, where, when, why and how to interact with the various data structures. I’ll aim for the client to follow the model of the existing Go Client Library that is available at Orchestrate Go Client on Github. It follows a basic model as shown below in Go language.

[sourcecode language=”cpp”]
c := client.NewClient("Your API Key")
// Get a value
value, _ := c.Get("collection", "key")
// Put a value
c.Put("collection", "key", strings.NewReader("Some JSON"))
// Search
results, _ := c.Search("collection", "A Lucene Query")
// Get Events
events, _ := c.GetEvents("collection", "key", "kind")
// Put Event
c.PutEvent("collection", "key", "kind", strings.NewReader("Some JSON"))
// Get Relations
relations, _ := c.GetRelations("collection", "key", []string{"kind", "kind"})
// Put Relation
c.PutRelation("sourceCollection", "sourceKey", "kind", "sinkCollection", "sinkKey")
[/sourcecode]

I’ll be working on this client, but don’t hold back on me, feel free to jump in with some of your own code or telling me I wrote some code wrong or whatever. I’d gladly accept any committers jumping in to help out. The more we all work together the more useful information I can provide during this project.

Once this project has produced a workable client pending interest from the community I’ll put together some material about where, how and some best uses around using the client in your Node.js Application. Even prospectively build a JavaScript client side library prospectively for use with Angular or other popular client side libraries.

References

Consistent Hashing – Learning About Distributed Databases :: Issue 002

One of the core tools in the belt of the distributed database is consistent hashing. In Riak this is especially true, as it stands at the core of a Riak Cluster. Hashing, using a hash function, is an algorithm that maps data to variable length to data that’s fixed. In other words, odd things like the name of things mapped to integers. Consistent hashing is a special kind of hashing that provides the pattern for mapping keys and all related functionality around a cluster ring in Riak.

Consistent hashing was originally devised by David Karger, a professor of computer science at MIT (Massachusetts Institute of Technology). He’s also known for Karger’s Algorithm, a Monte Carlo method that computes the minimum cut in a connected graph (graph theory related stuff). Along with these developments he’s been part of many other efforts and contributed to computer science in many ways.

Remapping, Mapping and Keeping Distributed (& Available)

One key property of a consistent hash is that it minimizes the number of keys that must be remapped. With a regular hash changes, the entire key hash must be remapped.

Consistent hashing is based around mapping each object to a point of a circle. The system maps each storage bucket to pseudo-randomly distributed points on the edge of this circle.

The system finds where to place the object based on the key on the edge of the circle. It then walks the circle falling into the first bucket it finds. This results in the buckets containing the resources between its point and the next bucket point.

When a bucket disappears for any reason, the pseudo randomly mapped objects will now get re-mapped to different buckets. When a bucket appears, such as becoming available again or being added, a similar process occurs.

The Basho Docs describe in brief that,

“Consistent hashing is a technique used to limit the reshuffling of keys when a hash-table data structure is rebalanced (when slots are added or removed). Riak uses consistent hashing to organize its data storage and replication. Specifically, the vnodes in the Riak Ring responsible for storing each object are determined using the consistent hashing technique.“

NOTES: This is not a single blog entry topic by any means. This is merely a cursory look at consistent hashing. This entry I aimed to provide a basic description and coverage of the actions around consistent hashing. For more information and to dive even deeper into consistent hashing I’ve included a few links that have extensive information on the topic:

An Ubuntu Riak #devrel

Setting up Riak to test out, prototype against, develop and use in a general way is extremely easy. Just setup a devrel on your local development machine. This is however limited to certain *nix based operating systems, so Windows as a dev platform is out – but not completely. Get a virtual machine running on Ubuntu, RHEL or some other Linux instance and you’re ready to go. What I’ve put together here is an example of getting a devrel up and running with an Ubuntu Virtual Machine.

Step 1: Get the basic reqs installed.

Step 2: With each of the nodes, now join and build the Riak cluster.

  • Start each node.[sourcecode language=”bash”]
    dev1/bin/riak start
    dev2/bin/riak start
    dev3/bin/riak start
    dev4/bin/riak start[/sourcecode]
  • Check to determine that the riak services are running.[sourcecode language=”bash”]
    ps aux | grep beam[/sourcecode]
  • Add each node to a single node.[sourcecode language=”bash”]
    dev2/bin/riak-admin cluster join dev1@127.0.0.1
    dev3/bin/riak-admin cluster join dev1@127.0.0.1
    dev4/bin/riak-admin cluster join dev1@127.0.0.1[/sourcecode]
  • Set and get the cluster plan.[sourcecode language=”bash”]
    dev2/bin/riak-admin cluster plan[/sourcecode]

    NOTE: This plan can be run from any of the instances.

  • Last, commit the cluster plan.[sourcecode language=”bash”]
    dev2/bin/riak-admin cluster commit[/sourcecode]

    NOTE: The commit can also be run from any of the instances.

Distributed Coding Prefunc: Up and Running With Erlang

Erlang LogoBefore diving into architecture, coding, descriptions and other things related to distributed computing over the coming months. It helps to become familiar with a language like Erlang. I’m going to dive immediately into getting Erlang up and running before any theory, description or otherwise, so following the most direct installation…

Installing Erlang

This is easy on OS-X. Pending of course you have the XCode and Developer Tools installed.

[sourcecode language=”bash”]
curl -O http://erlang.org/download/otp_src_R15B01.tar.gz
tar zxvf otp_src_R15B01.tar.gz
cd otp_src_R15B01
[/sourcecode]

Then compile the latest XCode and tools you can use the LLVM Compiler.

[sourcecode language=”bash”]
CFLAGS=-O0 ./configure –disable-hipe –enable-smp-support –enable-threads \
–enable-kernel-poll –enable-darwin-64bit
[/sourcecode]
[sourcecode language=”bash”]
make
sudo make install
[/sourcecode]

For more information on building Erlang you can also check out the Erlang Organization Site. It’s that simple, so now that it is up and running you should be able to check that all is right with the install by pulling a version.

[sourcecode language=”bash”]
erl -version
[/sourcecode]

In my follow up blog entry I’m going to take you through the Rebar Riak Core Templates. This will get you up and running with an Erlang Application. This application can then be used either as a stand alone Erlang App for whatever you want to build with it or as a great starting point to build against Riak.

SITREP – Stoking Iron Foundry, Thor Hammering & Joining… ?

What have I been up to? Here’s a quick recap. You may want to get involved with some of these projects!

Iron Foundry & Tier 3 Web Fabric

Iron FoundryBack when I left the kick ass team at Russell Investments in Seattle I stepped directly to bat as team lead at Tier 3. My job, get a PaaS built on Cloud Foundry and extending that with Iron Foundry. It was an ambitious effort that would provide the most extensive framework and language support available from any PaaS Provider on the market.

Well, we did it, thanks to the capabilities of Cloud Foundry Community, the great minds of Jared Wray @jaredwray, Luke Bakken, Eric Lee @saintgimp, Cale Hoopes & the rest of the Tier 3 Team! I was able to add this to my list of successes. We had some bumps, some collisions, a brick wall or two and other scheduling problems – ya know, the standard things that happen on a project. But in spite of it all, we got the Web Fabric released – and it continues to be the only PaaS available with such a wide framework and language support. It ranges from Ruby on Rails, Erlang, Node.js & Java to .NET! If you’d like to check out the open source PaaS of Cloud Foundry & Iron Foundry both projects are always looking for participation & contributions!

Thor Brings the Hamma!

After the release of the Tier 3 Web Fabric I started the search for a wicked smart and capable OS-X / Cocoa Coder – it seems their availability is pretty limited these days! Well I finally lucked out and found Benjamin van der Veen (@bvanderveen, thanks for the intro Selena @selenamarie!) to help me get started on the Thor Projects. There’s the Cocoa Thor Project & the Windows 7 WPF Metro based Thor .NET Project that we’re wrapping up with v1 releases coming really soon. To check out more on these projects that I’ve lead & coded on check out the code bases & information, all linked on ironfoundry.org. The projects are open source, so feel free to jump in and help out or fork & submit pull requests. The team will be happy to review & discuss ASAP.

While we’re wrapping these projects up right now, I’ll actually be continuing on and supporting the projects: Thor & Thor .NET. I will continue to be involved, as I was saying, in a number of ways in the PaaS space. So don’t think I’m disappearing form that realm!

Basho Sings my Song

I’ve been keeping track of Basho for a while now. Riak caught my interest many months ago as a really well built, well thought out & advanced distributed database. As you might guess, being into the whole “cloud computing” industry, I’m just ever so slightly interested in distributed systems. The other thing that I’m a huge fan of, which Basho does, is heavily support and involve itself in the open source software community and movement. The icing on the cake, was their diverse use of systems and language use around Erlang. All things that are massive wins.

Bailey's Taproom
Bailey’s Taproom

Well during a random conversation with Eric @coderoshi at Bailey’s Tap Room & then attending the RICON 2012 Conference (article here and pictures) I spoke to some of the team and found out they were looking for some particular skill sets. Well it just happened that I was keenly interested in meeting those skill set requirements! So December 1st I’ll be joining the Basho team full time as developer advocate, evangelist, messenger or such for the northwest working with a few people you may know such as Mark Phillips @pharkmillups, Andy Gross @argv0 (thanks for the intro James @wattersjames), Eric Redmond @coderoshi, Shanley Kane @shanley, Casey Rosenthal @caseyrosenthal and many others. Simply, I’m freaking stoked.

How This Helps You Help Me Help You

Alright, so it’s great but how can I help you in your day to day? What data do you work with? Do you work with a data scientist? Are you a data scientist? Do you work with huge sets of data, many objects, large objects? I want to know about your data usage and data problems, because there’s a good chance we’ll have more than few things to discuss. Here’s some ways I can help you, help me, help you. Ping me if you’re interested in…

  • talking about your data usage at the monthly Riak user group.
  • coding, pairing & otherwise learning Erlang and the monthly Erlang group.
  • interested in coding, deploying and inventing new paradigms and patterns of data storage.
  • interested in pairing up to learn how to deploy, migrate, upgrade or otherwise use NoSQL solutions – namely Riak.
  • interested in Ruby on Rails, Node.js, Map Reduce, .NET, Java, PHP and how these things can and do work against data in everything from relational databases to the new echelon of NoSQL databases.

I hope to hear from you soon and see you at an upcoming user group, cheers!