Erlang R15B01, What You Need For Building Some Riak and Shredding Code

My Shredding Code videos are slowly but steadily coming together. You can follow the channel on Vimeo at https://vimeo.com/channels/shreddingcode. The videos I recently put together are the specific instructions for installing Erlang R15B01 on Cent OS 6.4, Ubuntu 12.04 LTS and OS-X. These are a part of an overall project that will eventually cover a complete introduction to distributed systems with Riak and more. If you’re interested in this, sign up and follow my Shredding Code Channel and you’ll have first access to them.

Installing Erlang R15B01 on OS-X from Adron Hall on Vimeo.

Installing Erlang R15B01 on Ubuntu 12.04 LTS from Adron Hall on Vimeo.

Installing Erlang R15B01 on CentOS 6.4 from Adron Hall on Vimeo.

Distributed Coding Prefunc: Chicago Boss, Rails Based Erlang Power

Troy Howard and I sat down on a Friday night to do some straight up thrashing of Erlang and Chicago Boss. It seemed like a framework that has a lot of promise. Thus, we wanted to hack at it a bit and see what we could come up with.

First things first, I running OS-X and Troy hacking with the latest Ubuntu, we dove in. I first downloaded from the main Chicago Boss Site, which I immediately decided not to use and went straight for a fork and cloning of the repo.

[sourcecode language=”bash”]
git clone git@github.com:Adron/ChicagoBoss.git
cd ChicagoBoss
make
make app PROJECT=bigonion
cd ../bigonion
./init-dev.sh
[/sourcecode]

The ‘./init-dev.sh’ will launch the server to view whatever is available on the site. Now when I went to the localhost at port 8001, it shows some general mess which is kind of nondescript. But one can identify that the server is running if this works.

Hitting http://localhost:8001
Hitting http://localhost:8001
Boardwalk Explosion!
Boardwalk Explosion!

Oh dear, then the explosion…

At this point Troy started running into a number of problems on Ubuntu. Ranch wasn’t building based on what was included. He went through fighting and attempting to manually install this dependency but no go. He had originally downloaded RB14 of Erlang which should have worked. However it did not. It caused all sorts of madness.

I spooled up my VM of Ubuntu and went through the same steps above, even did the quick intro to setup a hello world, and it worked. We then checked out Erlang versions and I was running with R15B01. Troy went with installing that version and lo’ n’ behold, magic of the druids! Mayor Daley kicked this thing and made it run. So if you are running anything besides R15B01 on Ubuntu, we can’t attest to this stuff working. Just sayin’.

So Back to the Chicago Boss

So go back to the bigonion project above that we created and throw in a controller file. Be sure to use the exact naming, because this is seriously like rails, follow the convention or you’ll spend the rest of your life in the jail of convention-less hell!

Add a file at /src/controller/bigonion_greeting_controller.erl and add the following code.

[sourcecode language=”erlang”]
-module(bigonion_greeting_controller, [Req]).
-compile(export_all).

hello(‘GET’, []) ->
{output, "<strong>Daley will tell you how to run the Big Onion!</strong>"}.
[/sourcecode]

Now execute the code, or if you have the server running with ‘./init-dev.sh’ still then just navigate to ‘http://localhost:8001/greeting/hello&#8217; and you’ll get a hello world reality check about Chicago history. 😉

On to next things, let’s get a conformance test in there. Add a file called ‘src/test/functional/bigonion_test.erl’ with the following code snippet of a test in it.

[sourcecode language=”erlang”]
-module(bigonion_test).
-compile(export_all).

start() ->
boss_web_test:get_request("/greeting/hello", [],
[ fun boss_assert:http_ok/1,
fun(Res) -> boss_assert:tag_with_text("strong",
"Daley will tell you how to run the Big Onion!", Res) end ], []).
[/sourcecode]

Now build the project to production.

[sourcecode language=”bash”]
./rebar compile
./rebar boss c=test_functional
[/sourcecode]

…which left me with

[sourcecode language=”bash”]
$ ./rebar boss c=test_functional
==> bigonion (boss)
FATAL: Config file "boss.test.config" not found.
[/sourcecode]

Joy, another completely random error. But hey, that’s software development right. So that’s coming later… for now, I’m out. Keep hacking the Erlang bits. If you know what happened with the last command to run the test above, please leave a comment and help out! 🙂

Distributed Coding Prefunc: Rebar Multinode Riak Core

Before diving into this entry, you might want to check out some of my other getting Erlang installed with appropriate testing frameworks entries. Moving on…

At Basho we’re are always trying to make it easier to do big things. A short time ago we pushed forward on Rebar, Riak Core and getting things put together to make it simpler to get kick started working on distributed systems like the Riak Database & distributed system itself. There’s way more that is possible, which I’ll get into in just a minute. Before diving into some of those things, here’s a few quick links & context of what exactly Rebar and Riak Core are.

Riak Core
Github: https://github.com/basho/riak_core

Riak Core has been available for quite some time. We’ve also been hustling for a while getting together a robust array of material around Riak Core. One excellent place to get started on learning about Riak Core is the “Introducing Riak Core” blog article published on the Basho blog a while back. To describe Riak Core, or riak_core, it is the underpinnings of what Riak is built on. It provides many features to get you started building distributed systems. A few of the key features are being able to track and manage the nodes, clusters and related pieces of the distributed architecture within a system.

Rebar
Github: https://github.com/basho/rebar
Wiki: https://github.com/basho/rebar/wiki

Rebar is an Erlang build tool that helps you in putting together projects based on Riak Core.

Rebar Riak Core
Github: https://github.com/basho/rebar_riak_core

The Rebar Riak Core project repository template helps you start writing things like the Riak Database itself. It’s based on setting up Riak Core via template scripting an N Node Cluster devrel, vnodes, etc. Once you’re up and running it can be used to help develop distributed, scalable and fault tolerant applications.

For more on the Rebar Riak Core check out the README.md in the github repository. There are some great examples of how to get a multinode devrel running in a few steps.

Rebar Riak Core Quick Start

The quickest way to get started using the Riak Core and Rebar scripts is to get the prebuilt binaries or you can just clone and install the Rebar scripts if you’d like all the things. To get the binaries and executables you can download and have them ready by wget (or use your preferred method to download).

[sourcecode language=”bash”]
wget http://cloud.github.com/downloads/basho/rebar/rebar && chmod u+x rebar
[/sourcecode]

To get the cloned repository and ready for use.

[sourcecode language=”bash”]
$ git clone git://github.com/rebar/rebar.git
$ cd rebar
$ ./bootstrap
[/sourcecode]

Now the easiest way possible is to use the Riak Core Templates with a quick git clone. After cloning the repo, copy them to the rebar templates directory (note that you’ll need to create this initially) and then create a working directory to put the project in and navigate into that directory.

[sourcecode language=”bash”]
git clone git://github.com/rzezeski/rebar_riak_core.git
mkdir -p ~/.rebar/templates
cp rebar_riak_core/* ~/.rebar/templates
mkdir projectNameHere
cd projectNameHere
[/sourcecode]

Now that a template is available, run the following command to create the Erlang Project.

[sourcecode language=”bash”]
rebar create template=riak_core_multinode appid=rabbits nodeid=rabbits
[/sourcecode]

You’re now ready to go to work using Rebar and the template you’ve created. I followed the try-try-try example repo in the example above to get started, check it out for a great walk through that dives in deeper to Riak Core, each small element of the project and files created, and a multi-node project as the sample.

So what to do now?

This is where it is time to throw around some creativity to get real solutions to real problems. Building distributed systems is becoming more and more paramount to effective usage of infrastructure and systems. Using Riak Core to get started building out your distributed system is an ideal place to start. These are a few ideas that the team was brainstorming on. Over the coming weeks we’ll be putting together material to outline ways to not only get started, but to implement systems like this.

Distributed Web Caching Tier

Caching tiers often come up in conversations, whether related to distributed systems or not, and often end up on the distributed topic. The question resounds, “how do I create a caching tier that can be distributed and provide real session, state management, cached elements, live data and other needs?” Well Riak Core is a great place to start developing a custom distributed caching tier that could even extend to use Riak KV (the Riak Database implemented on Riak Core), Redis, Rabbit MQ or many other solutions by pulling them together to provide appropriate cache at the appropriate tiers of an application architecture.

In House Cluster Monitoring & Smart Resolver

One of the things that the Riak Core would be used to great effect for is a multi-node, clustered and geographically dispersed monitoring system for any multi-data center application. This could be built out and used for almost any actual environment, with custom specifics or a completely generic situation of pizza box servers. Because fo the distributed concepts behind Riak Core it would provide an ideal basis for monitoring – and re-launching or otherwise dealing with systems that need high uptime and recovered as fast as possible if they go down.

Logging, Web, Server, and Business Analytics

In any situation where analytics are collected there are often dozens if not thousands of servers, various systems and even numerous devices that may be emitting data via services or other mediums. Riak Core is a great place to lay the groundwork for a distributed system that could maintain a massive store of managed data for fast searching of analytics. This could be the groundwork for biotech research analytics, analysis of market data or a dozen other things that need highly available systems storing vast data with map reduction or other search capabilities. Think Business Intelligence (BI) with serious technological power.

Multi-node Project

Of course, as the example I used to create the first sample above, dive into the try-try-try tutorial for some great multi-node how to. If you have any questions please jump in ping me on twitter @adron or ping @basho, join the mailing list, the IRC #riak channel on freenode.

Distributed Coding Prefunc: Some Basic Erlang & Erlang Shell Basics

Erlang LogoThe first few parts of this series include:

This continues the series, with the intent of getting to “Distributed Coding” with this all ramping up around Erlang and distributed programing in general. In this entry we’ll jump into some of the basics of Erlang and the shell that we can use to test and write some code.

Erlang Basics : The Shell & Some Integers

Before getting to deep into things, here’s some basic shell usage commands and integers to get a feel for the Erlang shell. To start the shell just type erl, which will present the following in the terminal.

[sourcecode language=”erlang”]
$ erl
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1>
[/sourcecode]

To exit the shell just type “q().” and hit enter like this.

[sourcecode language=”erlang”]
$ erl
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1> q().
ok
2> $
[/sourcecode]

It’ll execute a second line that will dump you back out to the terminal. One thing to note, is that when you execute “q().” it drops to a second line numbered two above. If we start the shell back up again and type in some actual code we’ll see that the numbers increment. The idea is similar to each line of code in a code file.

Start up the shell again and type the following.

[sourcecode language=”erlang”]
$ erl
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1> q().
ok
2> Adrons-MacBook-Air-2:erlangMagic adronhall$ clear
Adrons-MacBook-Air-2:erlangMagic adronhall$ erl
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1> -69.
-69
2> 2#1212
2> 2#1212.
* 1: syntax error before: 212
2> 2#1010.
10
3> $A.
65
4> $B.
66
5> $a.
97
6> 2*2+100.
104
7>
[/sourcecode]

What I’ve done here is enter some values, primarily all integers. These integers, on the first line is a negative integer. By entering a “.” at the end, the lines executes and returns the value. In this case, the value is “-69”.

The second line, I entered “2#1212” without entering a period to finish the line of code. Thus, nothing executes.

The third line I entered the same thing as the second, “2#1212” which is not valid, giving us a syntax error. On the next line I tweak my value just a bit, entering a valid hexadecimal number “2#1010” with the appropriate period at the end. The line executes and it displays the decimal integer “10”.

So far at this point, this shows us several things:

  • Every line must end with a period to execute.
  • We’ve seen a negative integer, a positive hexadecimal integer and an invalid integer and the error that throws “* 1: syntax error before: 212”.
  • We’ve seen that when the line in the shell successfully executes we get the next incremental line, such as “-69.” was on the first line, the first successful second line was the “2#1010.” value and as we move through each line above the code line increases.
  • A valid integer includes -69, but also includes a hexadecimal representation of an integer such as “#2#1010”.

Moving beyond these values the “$A”, “$B” and “$a” are all ASCII representations and return their numeral values.

On line six I actually perform a basic calculator type multiplication and addition of integer values, which returns the integer value 104.

Now that we have the shell start and stop figured out, that gives us a platform in which to start stepping through some of the basics of Erlang.

Erlang Floats

Above we’ve seen how regular integers are represented such as 2, 100 or -69 along with hexadecimal and ASCII numeric representations. For floats, we’re looking at similar uses. Spool up the shell again and try out these examples.

[sourcecode language=”erlang”]1> 21.21.
21.21
2> -1234.1234.
-1234.1234
3> 3042.1.
3042.1
4> 650.21
4> 650.21.
* 2: syntax error before: 650.21
4> 650.21
4> 650.21.
* 2: syntax error before: 650.21
4> 650.21.
650.21
[/sourcecode]

Lines 1-3 show some standard floats. On the first part of line four I left off the period to show that it won’t accept the line, even with a period in it, because of the numbers after that period. In that second line number four I’ve added the period, but it throws a syntax error.

Then note I did the exact same thing again, to show that entering a float and then adding the period, will still leave the float broken because of the way the Erlang compiler is waiting to execute the code. It’s almost like the compiler sees “650.21650.21” which of course doesn’t mean anything. However on the fifth line of the number four lines, I finally get a good execution and it displays the float I’ve entered “650.21”. Be sure when you’re fixing values like this, the compiler could be stuck in mid execution, because without the period at the very end, being the last character on the line, the compiler is still waiting for the period to actually start the execution.

Mathematical Operators, You Know Those Things That Help Math Explain the Universe!

Erlang has the standard operators that you’re used to, especially if you’ve done anything with a language based around C syntax such as JavaScript, Java, C#, C++, Objective-C… well, pretty much every regularly used static typed language and more. In functional languages these operators are pretty common too. If any don’t look familiar, be sure to get intimate with them. You’ll find them all over Erlang code.

The + and – characters act as addition and subtraction, but also as unary operators. In order of precedence, the unary operations of the + and – have the highest precedence and the addition and subtraction usage of the + and – have the lowest precedence.

The * and / signify multiplication and division. They’re precedence is just below that of the unary operators of + and -.

Two that are a slight bit unique, is the usage of “div” and “rem” for division and remainder, for integer numbers. This begs to bring up the point that whenever an integer is involved in multiplication, division, addition, subtraction or other operations, it is coerced into a float when being operated on against another float. Such as 2 + 3.47 would mean that 2 is turned into a float before it is added to 3.47.

Here’s some examples of operations occurring between numbers via the shell.

[sourcecode language=”erlang”]
$ erl
Erlang R15B01 (erts-5.9.1) [source] [smp:4:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.9.1 (abort with ^G)
1> +15.
15
2> +2.
2
3> -11.
-11
4> 11 div 5.
2
5> 12 div 5
5> .
2
6> 15 div 5.
3
7> 14 div 5.
2
8> (12 + 345) div 12.
29
9> (3+2)/5.
1.0
10> 2*3*4.12.
24.72
11> 1+1+1+1+13+-2.
15
12> 2/3 + 3/4 – (1/2 + 2/3) – 1.
-0.75
13> 14 rem 5.
4
14>
[/sourcecode]

These examples show a number of effects of how the operators work. Notable is how everything looks very much like simple math being done. This is one of the more readable aspects of Erlang. Note also that the rounding generally goes down.

Distributed Coding Prefunc: Ubuntu Erlang Dev & EUnit

Erlang LogoAfter installing Erlang on OS-X and then getting QuickCheck installing via Erlang, I wanted to expand the OS options I’m using to Ubuntu (i.e. Linux). So in this entry I’m going to cover the Erlang install, a quick eunit bit, and then a QuickCheck install and sample. The first step in geting Erlang installed is deciding how you want to install it. So far, it isn’t like Rails where it is pretty important which method you pick to how well it will work for you on Ubuntu. For Erlang all methods get you started with a good version that is working. The method I used was simply to install with apt-get.

[sourcecode language=”bash”]
sudo apt-get install erlang erlang-doc
[/sourcecode]

After installing, always a good idea to run things and make sure they’re all content and happy with your machine. Startup the erlang shell.

[sourcecode language=”bash”]
erl
[/sourcecode]

Then run some of the commands. Some I’ve found that will present you with useful and interesting information ist he erlang:system_info function with appropriate parameter passed. The otp_release parameter will get the version of erlang, the cpu_topology shows you the processor outlay for you machine (or in this case my virtual machine, with a single processor core allocated to it), and allocated_areas shows a bit about system memory allocations.

[sourcecode language=”erlang”]
Eshell V5.9.1 (abort with ^G)
1> erlang:system_info(otp_release).
"R15B01"
2> erlang:system_info(cpu_topology).
[{processor,{logical,0}}]
3> erlang:system_info(allocated_areas).
[{sys_misc,80748},
{static,1007616},
{atom_space,98328,73387},
{atom_table,95961},
{module_table,9084},
{export_table,50316},
{export_list,240960},
{register_table,180},
{fun_table,3266},
{module_refs,2048},
{loaded_code,3437028},
{dist_table,403},
{node_table,227},
{bits_bufs_size,0},
{bif_timer,80200},
{link_lh,0},
{process_table,262144},
{ets_misc,52504}]
[{processor,{logical,0}}]
[/sourcecode]

Now that erlang is effectively installed we can write a little sample code. To do this I created a directory called “TestingErlang” and in it placed an Erlang code file called “eunit_tests.erl”. Note: I’m using Sublime 2 on Ubuntu, so exchange that for whatever text editor you’re using for your Erlang coding.

[sourcecode language=”bash”]
adron@ubuntu:~/Codez$ mkdir TestingErlang
adron@ubuntu:~/Codez$ cd TestingErlang
adron@ubuntu:~/Codez/TestingErlang$ sublime eunit_tests.erl
[/sourcecode]

Add the header file include.

[sourcecode language=”erlang”]
-define(NOTEST, true).
-include_lib("eunit/include/eunit.hrl").
[/sourcecode]

Adding the header file include will cause all the function with _test() or _test_() to automatically be exported. An exported function of test() is created that can be used for running all of the unit tests. This also will include the preprocessor macros of EUnit for writing tests. So now throw a super simple test into the file.

You may want to, amid the automatic export of the methods ending in _test() or _test_() not name them this, and you’ll then need to add a line at the top of your code file like this.

[sourcecode language=”erlang”]
-export([reverse_test/0]).
[/sourcecode]

After this, add the function test as shown below.

[sourcecode language=”erlang”]
reverse_test() -> lists:reverse([1,2,3]).
[/sourcecode]

The complete code file should look like this.

[sourcecode language=”erlang”]
-module(eunit_test).
-define(NOTEST, true).
-include_lib("eunit/include/eunit.hrl").
-export([reverse_test/0]).

reverse_test() -> lists:reverse([1,2,3]).
[/sourcecode]

Build it and call the function.

[sourcecode language=”bash”]
Eshell V5.9.1 (abort with ^G)
1> c(eunit_test).
{ok,eunit_test}
2> eunit_test:reverse_test().
[3,2,1]
3>
[/sourcecode]

BOOM! Passing. So now we know we have a good Erlang install and eunit is setup and usable. In the following blog entries I have in the works, we’ll dive deeper into what and how Erlang works from an extremely basic level all the way to diving into some of the more complex features.