Distributed Coding Prefunc: Rebar Multinode Riak Core

Before diving into this entry, you might want to check out some of my other getting Erlang installed with appropriate testing frameworks entries. Moving on…

At Basho we’re are always trying to make it easier to do big things. A short time ago we pushed forward on Rebar, Riak Core and getting things put together to make it simpler to get kick started working on distributed systems like the Riak Database & distributed system itself. There’s way more that is possible, which I’ll get into in just a minute. Before diving into some of those things, here’s a few quick links & context of what exactly Rebar and Riak Core are.

Riak Core Github: https://github.com/basho/riak_core

Riak Core has been available for quite some time. We’ve also been hustling for a while getting together a robust array of material around Riak Core. One excellent place to get started on learning about Riak Core is the “Introducing Riak Core” blog article published on the Basho blog a while back. To describe Riak Core, or riak_core, it is the underpinnings of what Riak is built on. It provides many features to get you started building distributed systems. A few of the key features are being able to track and manage the nodes, clusters and related pieces of the distributed architecture within a system.

Rebar Github: https://github.com/basho/rebar Wiki: https://github.com/basho/rebar/wiki

Rebar is an Erlang build tool that helps you in putting together projects based on Riak Core.

Rebar Riak Core Github: https://github.com/basho/rebar_riak_core

The Rebar Riak Core project repository template helps you start writing things like the Riak Database itself. It’s based on setting up Riak Core via template scripting an N Node Cluster devrel, vnodes, etc. Once you’re up and running it can be used to help develop distributed, scalable and fault tolerant applications.

For more on the Rebar Riak Core check out the README.md in the github repository. There are some great examples of how to get a multinode devrel running in a few steps.

Rebar Riak Core Quick Start

The quickest way to get started using the Riak Core and Rebar scripts is to get the prebuilt binaries or you can just clone and install the Rebar scripts if you’d like all the things. To get the binaries and executables you can download and have them ready by wget (or use your preferred method to download).

wget http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x rebar

To get the cloned repository and ready for use.

$ git clone git://github.com/rebar/rebar.git
$ cd rebar
$ ./bootstrap

Now the easiest way possible is to use the Riak Core Templates with a quick git clone. After cloning the repo, copy them to the rebar templates directory (note that you’ll need to create this initially) and then create a working directory to put the project in and navigate into that directory.

git clone git://github.com/rzezeski/rebar_riak_core.git
mkdir -p ~/.rebar/templates
cp rebar_riak_core/* ~/.rebar/templates
mkdir projectNameHere
cd projectNameHere

Now that a template is available, run the following command to create the Erlang Project.

rebar create template=riak_core_multinode appid=rabbits nodeid=rabbits

You’re now ready to go to work using Rebar and the template you’ve created. I followed the try-try-try example repo in the example above to get started, check it out for a great walk through that dives in deeper to Riak Core, each small element of the project and files created, and a multi-node project as the sample.

So what to do now?

This is where it is time to throw around some creativity to get real solutions to real problems. Building distributed systems is becoming more and more paramount to effective usage of infrastructure and systems. Using Riak Core to get started building out your distributed system is an ideal place to start. These are a few ideas that the team was brainstorming on. Over the coming weeks we’ll be putting together material to outline ways to not only get started, but to implement systems like this.

Distributed Web Caching Tier

Caching tiers often come up in conversations, whether related to distributed systems or not, and often end up on the distributed topic. The question resounds, “how do I create a caching tier that can be distributed and provide real session, state management, cached elements, live data and other needs?” Well Riak Core is a great place to start developing a custom distributed caching tier that could even extend to use Riak KV (the Riak Database implemented on Riak Core), Redis, Rabbit MQ or many other solutions by pulling them together to provide appropriate cache at the appropriate tiers of an application architecture.

In House Cluster Monitoring & Smart Resolver

One of the things that the Riak Core would be used to great effect for is a multi-node, clustered and geographically dispersed monitoring system for any multi-data center application. This could be built out and used for almost any actual environment, with custom specifics or a completely generic situation of pizza box servers. Because fo the distributed concepts behind Riak Core it would provide an ideal basis for monitoring – and re-launching or otherwise dealing with systems that need high uptime and recovered as fast as possible if they go down.

Logging, Web, Server, and Business Analytics

In any situation where analytics are collected there are often dozens if not thousands of servers, various systems and even numerous devices that may be emitting data via services or other mediums. Riak Core is a great place to lay the groundwork for a distributed system that could maintain a massive store of managed data for fast searching of analytics. This could be the groundwork for biotech research analytics, analysis of market data or a dozen other things that need highly available systems storing vast data with map reduction or other search capabilities. Think Business Intelligence (BI) with serious technological power.

Multi-node Project

Of course, as the example I used to create the first sample above, dive into the try-try-try tutorial for some great multi-node how to. If you have any questions please jump in ping me on twitter @adron or ping @basho, join the mailing list, the IRC #riak channel on freenode.

Light up a Riak Cluster with AWS, A Few Notes…

I wanted to write up an intro to getting Riak installed on AWS, even though the steps are absurdly simple and already available on the Basho Docs site, there’s a few extra notes that can be very helpful for a few specific points during the process.

Start off by logging into AWS. At this point you can take two different paths that are almost identical. You can follow the path of using the pre-built AWS Marketplace image of Riak, or just start form scratch. The difference is a total of about 2 steps; installing & setting some security port connections. I’m going to step through without using the prebuilt image in these instructions.

Security Group

First thing you’ll need to get a security group with the correct permissions setup. For that, you’ll need to make a security group.

NOTE: No, I didn’t mean to misspell Riak, but it’s in there now.  😉

Before adding the ports, go to the security group details tab and copy the security group id. I’ve pointed it out in the image above.

Now add the following three and assign the security group to the ports; 4369, 8099 & 6000-7999. For the source set it to the security group id. Once you get all three added the list should look like this (below). For each rule click the Add Rule button and remember to click the Apply Rule Changes. I often forget this because the screen on some of the machines I use only shows to the bottom of the Add Rule button, so you’ll have to scroll down to find the Apple Rule Changes button.

Now add the standard port 22 for SSH. Next get the final two of 8087 and 8098 setup and we’re ready for moving on to creating the virtual machines.

Server Virtual Machines

For creating virtual machines I just clicked on Launch Instance and used the classic wizard. From there you get a selection of items. I’ve used the AWS image to do this, but would actually suggest using a CentOS image of your choice or Red Hat Enterprise Linux (RHEL). Another great option is to use the Ubuntu 12.04 LTS. Really though, use whatever Linux version or distro you like, there are 1-2 step instructions for installing Riak on almost every distro out.

Next just launch a single instance. We’ll be able to launch duplicates of these further along in the process. I’ve selected a “Micro” here but I’m not intending to do anything with a remotely heavy load right now. At some point, I’ll upgrade this cluster to larger instances when I start putting it under a real load. I’ll have another blog entry to describe exactly how I do this too.

Keep hitting continue until you get to the key pair selection. Pick the key pair you want, either making a new one for this cluster or use one you already have. Either way works fine.

Continue again until you can select the security group that we created above.

Now keep hitting that continue button, until you get to launch, and launch this thing. Once the instance is launched launch your preferred SSH connection tooling. The easiest way I’ve found for getting the most current private IP to connect to with the appropriate command is to right click on the instance in the AWS Console and click on Connect. There you’ll find the command to connect via SSH.

Paste that in and hit enter in your SSH App, you’ll see something akin to this.

$ cd Codez/working-content/
$ ssh -i riaktionz.pem root@ec2-54-245-201-97.us-west-2.compute.amazonaws.com
The authenticity of host 'ec2-54-245-201-97.us-west-2.compute.amazonaws.com (54.245.201.97)' can't be established.
RSA key fingerprint is 31:18:ac:1a:ac:fc:6e:6d:55:e8:8a:83:9a:8f:c7:5f.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'ec2-54-245-201-97.us-west-2.compute.amazonaws.com,54.245.201.97' (RSA) to the list of known hosts.
Please login as the user "ubuntu" rather than the user "root".

Enter yes to continue connecting. For some instance types, like Ubuntu you’ll have to do some teaks to log into as “ubuntu” vs. “root” and the same goes for the AWS image or others. I’ll leave that to you, dear reader to get connected via ole’ SSH.

One of the other things, that you may have to do some tweaking about and googling, is figuring out the firewall setups on the various virtual machine images. For the RHEL you’ll want to turn off the firewall or open up the specific connection ports and such. Since the AWS firewall does this, it isn’t particularly important for the OS to continue running its firewall service. In this case, I’ve turned off the OS firewall and just rely on the AWS firewall. To turn off the RHEL firewall, execute the following commands.

[root@ip-x-x-x-x]# service iptables save
iptables: Saving firewall rules to /etc/sysconfig/iptables:[  OK  ]
[root@ip-x-x-x-x]# service iptables stop
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Unloading modules:                               [  OK  ]
[root@ip-x-x-x-x]# chkconfig iptables off
[root@ip-x-x-x-x]#

Now is a perfect time to start those other instances. Navigate into the AWS Console again and right click on the virtual machine instance you’ve created. On that menu select Launch More Like This.

Go through and check the configuration on each of these, make sure the firewall is turned off, etc. Then move on to the next step and install Riak and cluster them. So it’s time to get to the distributed, massively complex, extensive list of steps to install & cluster Riak. Ok, so that’s sarcasm.  😉

Step 1: Install Riak

Install Riak on each of the instances.

package=basho-release-6-1.noarch.rpm && \
wget http://yum.basho.com/gpg/$package -O /tmp/$package && \
sudo rpm -ivh /tmp/$package
sudo yum install riak

NOTE: For other installation methods, such as directly downloading the RPM or other Linux OSes, check out the http://docs.basho.com/riak/latest/tutorials/installation/Installing-on-RHEL-and-CentOS/.

Step 2: Setup the Cluster

On the first instance, get the IP. You won’t need to do anything to this instance, just keep the IP handy. Then move on to the second instance and run the cluster command.

sudo riak-admin cluster join riak@<ip_of_the_first_node>

Do this on each of the instances you’ve added, using that first node. When you’ve added them all, on that last instance (or really any of them) then run the plan. This will get you a display plan of what will take place when the cluster is committed.

sudo riak-admin cluster plan

If that looks all cool. Commit the plan.

sudo riak-admin cluster commit

Get a check of the cluster.

sudo riak-admin member_status

That’s it all done. You know have a Riak Cluster. For more operations to try out your cluster, check out this list of base API Operations.

Write the Docs, Railsconf Portland, RICON East, Node PDX, Vancouver Polyglot, Open Source Bridge and OSCON…

…if you are deciding what to attend this year, here’s the top of the list.

Just a few key conferences that will kick ass in technical & academic content. The other great thing about these conferences is that they either have a “code of ethics” or are reknown for real conference diversity vs. the “hey a bunch of privileged sameness all ended up in this room purely out of meritocracy” nonsense. So you can rest assured that at these conferences you’ll have interesting conversations, be actively involved in things that will expand our personal sphere of the world and in the end leave you dramatically more enriched than those “corporate warm body vacation conferences“. This list doesn’t mean I’ll be able to make it to all of them, but each conference I’m in some way intimately involved with and fully support with the “Conference Seal of Approval“! So here we go…

Write the Docs – April 8th and 9th – @writethedocs – A new movement, to know where we are, where we’ve been and where we’re going. Get here, know how and why to bring knowledge forth through documentation. This is not your grandpa’s documentation.

Portland RailsConf 2013 – April 29th-May 2nd – @railsconf – The rails community has grown by leaps and bounds. Regardless of your love or hate for the framework, it’s revolutionized the way web applications are built over the last decade.

RICON East in New York City – May 13th & 14th – @basho – (I do work for Basho, but this is on here because RICON last year ROCKED and meets the requirements, come see for yourself!)  🙂

Node PDX – May 16th & 17th – @nodepdx – Troy Howard and I are putting this together, we’re working hard to make sure it’s more of what you want, more than it was last year and kicking to the curb the things you don’t want. So come hack some node, JavaScript and enjoy yourselves.

Polyglot 2013 – May 24th through the 26th – @polyglotConf – Open spaces, with tons of really, truly smart people with no presumptuous marketing and sales bullshit to get in the way. This is about software development, across the realms of frameworks, languages and more.

OS Bridge – June 18th through the 21st – @osbridge – open source bridges the divide here and new thinking is created. You have to attend to see…

OSCON – July 22nd through the 26th – @oscon – This is the premier open source conference in North America. It’s in Portland. Nuff’ said.

I’ll be at a few more conferences this year, but these are the key conferences, if you have to pick one to go to, it should be on this list. If you can go to one or two others, pick em’ from this list. Software + Data + Giant Phat Data + NoSQL + Future Thinking + Leaders o’ Thoughts == Top Conference List.

See ya there. Cheers!

Systems Thinking, Measuring Things and Really, Cultural Change is Free and Why Your Measurement are Likely Screwing Up Your Business

I was going to write this up. I stumbled upon this video of John Seddon presenting. But seriously, WATCH the video, especially if you work in an enterprise. If you’re in management or in executive leadership you desperately need to watch this and know it, understand it and listen to it. To benefit of yourself and those around you and those that interact with you business. Whatever I write is tertiary in relation to what John says here.

Watched the video? Well, either way here’s a quick overview of things. John breaks out why targets make organisations worse and controlling, or managing costs, actually makes them higher. He explains, in a rather entertaining way, why the public sector along with the private sector is doing horribly compared to what they should be doing. As stated in the video description, the best way to put it is “Target Obsession Disorder” laid bare!

It’s like a breath of fresh air.

How does this tie back to programming and code? Simple. To truly draw together what needs to be built, to see the big picture, the focus of the consumer and where we’re heading one must try, as hard as it is, to understand the whole of a system. Thus, systems thinking is the cerebral approach to actually write software well, for what it is really needed for. As I’ve been saying a lot as lately, “Never make a decision on a single metric, it’ll very likely be the wrong decision.”

As John states, I’ll leave this article with, destroy your “failure demand“.

Only Yahoos Work in an Office!?

Ok, so I think almost everybody has either slammed Marissa Mayer about the new Yahoo non-remote worker policy or said that it’s the medicine they have to swallow. Very few are actually pointing out however, that Yahoo was probably just really bad at managing their remote employees. In the end, I don’t care, that just means there are now going to be more people who are probably great that will be available in a tech market that is happy to hire the talent!

But I do digress, I have something else to talk about that is actually productive. I’m going to kick this off with a short story, that actually inspired me to write this post.

Remote Workers Unite!

I was in the office, a coworking office called NedSpace in downtown Portland with my fellow Bashoians Eric Redmond (@coderoshi) & Chris Tilt. Generally we all work remotely, because everyone at Basho is remote. The entire company, from CEO to HR to Marketing to Engineering, everybody is remote. Basho does a solid job of working like this, it is indeed, a modern Internet enabled software company. However today we were all in the office.

During the day I pulled up Scott Hanselman‘s Blog Entry “Being a Remote Worker Sucks“. I found this on twitter and shortly after reading it, tweeted,

Case Study Coffee

Case Study Coffee

Within a few minutes, lo and behold in the glare of my Apple Cinema Display I see Scott Hanselman walk up behind me! (Isn’t this building secure?!!? Scott’s name is now Hacker Hanselman!) Wow, that doesn’t happen everyday or does it? So Eric & Chris finished up some coding. Chris headed off to return home, where he remotely works from. However Eric, Scott & I headed out for a stroll through the city of Portland to Case Study Coffee. We had a great conversation, discussing all sorts of blog entry ideas, Node.js oddities, duplications between Rail & Node.js & Java & .NET and all sorts of other things. I mean, you probably know how it goes when you have three geeky people going on endlessly about awesome nerdy stuff. Simply put, we covered a lot of topics.

…that’s when I thought…

Offices Are Often The Worse Place to Cross Pollinate Ideas

Not a little bit, but likely the absolute worse places to cross-pollinate ideas. But then I got to thinking a little bit more. Is Yahoo on to something? Is there a need or a reason why people should be in a similar place? I mean, I know for a fact the most rapid learning I ever had was pair programming with other coders. But I’ve done this in an office and outside of the office, at coffee shops and hackathons, on the rooftop of buildings and in dark and dreary bars. I’ve paired with a lot of coders and learned a lot by doing that.

Another question came up, is it really important to be in the office or is it important to be near others you are working with? More of that thinking and remembering things and more experiences popped into my mind. Whew, this was becoming a serious thinking session. I’d also had a great experience working remotely and learning at practically the same rate with friends. We were all working on an open source project. We asked questions in general chat on IRC or other places, and got instant feedback and help when we needed it. We’d put ideas into chat and discuss them readily and sometimes excessively. We weren’t anywhere near each other, specifically we were about 1400+ miles from each other in different time zones. But it worked and it worked damn well!

My correlations in trying to figure out, should employees be in office or out of office could go on. As Eric said when we all returned form the coffee shop, “correlations do not show causation, but they sure as hell imply it“. Well, I didn’t want to just imply onsite or remote work is better I wanted to know what really works or doesn’t. More thinking ensued and then…

Freedom Punched Me In The Face

Then it hit me. Geographic location doesn’t make any difference. The difference, which also makes Yahoo’s demand for no more remote working, is again the freedom of the individuals involved. The freedom to work where we are needed with who we ought to and need to and when we are most efficient and capable of performing the work best. The key is this freedom is granted, and that driven self organizing individuals make use of this freedom. It may be that one day we might end up in the office while another day we end up in a park, at home or in some coffee shop.

So forget remote working or in office working. These things aren’t correlated to someone being productive or not, the core reasons someone is productive is the ability that they have the freedom to be productive in the way that best works for them and for the company they’re providing services for. The most important thing is being able to give freedom to those doing the work to be productive. Remote or not is pure distraction and in turn is bullshit.

Don’t get distracted by the remote or onsite worker debate, figure out the best ways to work with your employees based on individual freedom. You’ll have far more productive individuals than any existing hierarchical corporate structure nonsense.