Introducing Junction

Today I’ve officially kicked off a new project from my notebook of projects based around building a Riak admin, data manipulation, reporting and news tool for Windows 8. If you want to jump right to the project, here’s the Github Pages Site, the Github Junction Repo and eventually I’ll have it listed in the Windows 8 Store for download. Yes, it’ll be free as in beer, it’ll all be Apache 2.0 Licensed and the project is open to contributors and others that want to jump into things. There’s also a quick intro for how I setup the “Windows 8 Logos, Badges & Splash Screens of Riak“.

So now that I’ve provided the links, here’s a quick intro to each of the application sections, what this application is for, where the workflow for contributions will be and what the next steps are. Trust me, I roll easy, I’ll be working as hard as I can to make pull requests easy peasy, keep the issues down to workable contributions and the whole “this is a good OSS project”.

Riak Junction Application rocking on the Windows 8 desktop with a full tile!

Riak Junction Application rocking on the Windows 8 desktop with a full tile!

Juncture Divisions

The juncture application should be split into several key components, or application divisions of functionality. I’ve broken each out with a basic description. If you just want to watch a video where I outline each division, play the video below for a quick 5 minute intro to the application and the idea behind it all.

A quick run through of the first sample UI.

Call the Doctor! (Administration & Maintenance)

This part of the application would provide an interface for all the general administration and maintenance needs around individual nodes and around the overall cluster of nodes. The ability to add, remove and generally administer everything that is available via the riak-admin command line interface.

Time Travel That Data (Performance Benchmarking)

This section of the application will provide the ability to benchmark the timing of data in and out of a cluster. In addition it should show standard benchmarking similar to that which is offered with the basho_bench project.

Love of the Data (Reporting)

This division of the application would be focused on reporting. I’m not sure what exactly that would entail, but something with charts, graphs and pulling together trending points of some sort. If you have ideas and want to work on this part of the application, weigh in!

Golfing With Your Data (Query, Put, Deletes, Etc. Handling the CRUD)

The application will have an interface to provide access to add and remove data, as well as viewing the data that is available within a cluster. The primary means for implementing this part of the application will be with the CorrugatedIron Project. It’s a library available via Nuget that @peschkaj and @TheColonial have put together.

News! News! News! (News…  RSS Feed Reader)

The idea is that this will provide a quick and easy way to get familiar with Windows 8 dev and the project overall. I’m aiming to eat the Basho blog feed and provide it as key highlights for the application with future abilities around mining other RSS feeds or such and having those fed into a ??  Riak Cluster? Again, everything is open to change, addition or removal! So jump into the project and let me know your thoughts.

Cheers & Happy Hacking!

Farewell Basho, It’s Been Swell Yo!

Whew, it’s been a total blast working at Basho. I’ve accomplished a ton of things. Riak is a solid distributed database system and I’m glad to have worked with the team on advocating its use, teaching distributed systems ideas and concepts and generally spreading the knowledge. I’ve seen some truly great things that people are hacking together, setting up for projects and redesigning old systems to utilize newer, better, faster and more capable distributed systems concepts and ideas. Some of the things I’m happy to have contributed to in my time at Basho.

…and there has been a whole lot more. Suffice it to say, Basho has provided me with some sweet opportunities to work on some extremely interesting data projects from a very data sciency point of view (yeah I know sciency aint a word). There may be more Riak work and Riak meetups and Riak hacks and Riak who knows what coming from me, but the meetups & such are now at the hands of the core Riak crew and…

Where Am I Headed?

Right now, I’m moving 20 blocks away from where I currently live, setting up a couch to hack on and grabbing a beer. I’ve got a few personal projects I’ve been wanting to work on. Then I’m taking a few weeks to do some side projects that have been on the burner. Keep an eye out, I’ll be kicking off one, maybe two of these open source projects in the next few days. As @tsantero twitted…

…I’m going to attack my own notebook of ideas. Maybe I’ll even work on that Riak CS Video object store that Tom and I spoke about 10 months ago? Either way, whatever the projects are, I’ll have them posted right here. Until then…

Cheers & Happy Hacking!

Windows 8 Logos, Badges & Splash Screens of Riak

As I’ve started working on a Windows 8 project here at Basho, there are a few pieces of collateral that help out to bring some branding and appeal to the application appearance. The first two things to grab if you want build a good looking Riak + Windows 8 application are the assets for design.

  • Basho Design Assets : This includes several transparent images for the Basho & Riak, Riak CS and Riak Enterprise logos. Toward the bottom of the page there are also a number of Bashomen that you can download.
  • Windows 8 Store Design Assets : This page includes a lot of downloadable Photoshop & other design assets for putting together a Windows 8 user interface and experience.
  • Windows 8 Design Guidelines : This page shows how the interactions on Windows 8 are supposed to be used, developed around and what their best use is.
  • Windows 8 UX Patterns : A sampling of UX patterns for Windows 8.

Design Assets

To check out the design assets I’ve put together I’ve created a github repository junction_design_assets. In this repository are all of the Adobe Photoshop *.psd files for each logo, wide logo, small logo, store logo, badge logo and splash screen image for the upcoming application. I’ve also attached an Apache 2.0 license to this for use by others. All of the images created had a transparent background, and are set to display against a black or other dark background.

Here’s a quick intro where all of these design assets go, via Visual Studio 2012. The first step was to create a good splash screen, as defined by the Windows 8 guidelines for images at sizes of 1116x540px, 868x420px and 620x300px.

Setting up the Splash Screen (Click for full size image)

Setting up the Splash Screen (Click for full size image)

A: Open the Package.appmanifest file and the designed that is displayed above will show a screen used for editing the file.

B: In this case I changed the background color to black since I created all of the images to display on a dark background. One can’t get any darker than the darkest black, and this is that black!

C: When you click on the elipsis to upload each of the images it will upload the image to the Assets folder within the project. With that in mind DO NOT manually put the images into the project. Use this screen or you’ll end up with all sorts of headaches.

Also important is the default SplashScreen.png that is put into the Spash screen: text box. Don’t fill this in first, instead click each ellipsis under each splash screen image size and select each image for each size. The odd thing is (one of those completely non-intuitive things) is that the Assets\SplashScreen.png file doesn’t actually exist in the Assets directory, but is instead created somehow through pixie dust magic and added to the project. This will be the same for other sections of the Package.appmanifest file settings and logo images that are to be added. So remember to NOT change this but to instead upload all of the other images required first.

The second step then was to get a good logo as defined by the guidelines at 270x270px, 210x210px, 150x150px and 120x120px. For this I went with the Riak Graphic Logo with images as shown at the different sizes.

Adding the Logo (Click for full size image)

Adding the Logo (Click for full size image)

Next up is the Wide Logo…

Wide Logo (Click for full size image)

Wide Logo (Click for full size image)

The Small Logo…

Small Logo (Click for full size image)

Small Logo (Click for full size image)

…and the Store Logo.

Store Logo (Click for full size image)

Store Logo (Click for full size image)

This is the first step, which is where I’m at so far with this application. The application is also available with an Apache 2.0 license with the code on github. If you’d like to jump into the code and help me build this application please feel free to reach out to me twitter @adron or hit me on Github @adron.

Philadelphia Riak Training & Presentations

R Graphs

R Graphs

This week I’ve traveled to Philadelphia to meet with a number of the Basho team to work together and receive training with the trainers or the best ways to approach content on Riak and more generally the best ways we can all brainstorm up to approach specific topics. Some of those topics include things like:

  • Access Patterns around Log Storage & Analysis
  • Bloom Filters
  • CRDTs
  • Consensus Protocols
  • Erasure Coding
  • LevelDB and Bitcask Backends
  • MDC Repl

Out of the options we discussed in training today I ran with benchmarking. It is always near and dear to many of the customers’, clients’ and curious’ that I talk to. I dove in to see what exactly we offer with basho_bench (docs info, github repo) in detail and functionality, but also dove into other benchmarks are out there that others may have run in the past.

basho_bench

What exactly is basho_bench? The basho_bench project is a code repo on Github that offers a set of benchmarking tests that are run against a Riak cluster. There are a few prerequisites to the quick steps below, the prerequisites are:

  1. Make sure you have a cluster or a devrel (Basho docs on devrel) setup that you can point basho_bench at.
  2. Erlang R15B03 should be installed (OS-X Compiler battles and fixes).
  3. Make sure R is installed.

Now to get basho_bench setup.

git clone git://github.com/basho/basho_bench.git
cd basho_bench
make all

Once that is done building, review the directory structure that is in the basho_bench directory. The following should be available in the directory.

$ ls
FAQ		deps		rebar
LICENSE		ebin		rebar.config
Makefile	examples	src
README.org	include		tests
basho_bench	priv

The examples directory has several default config files available to run with basho_bench for testing. If there is a devrel setup with the default 127.0.0.1 IP usage, just run the following command to begin generating stats. If the cluster being tested is not a devrel with 127.0.0.1 then give the configuration section of the docs a read for information on how to point basho_bench at an alternative cluster.

./basho_bench examples/http.config

You’ll see something akin to this spit out onto the screen.

$ ./basho_bench examples/http.config
10:34:41.736 [debug] Lager installed handler lager_console_backend into lager_event
10:34:41.752 [debug] Lager installed handler {lager_file_backend,"/Users/adronhall/Codez/basho_bench/tests/20130808_103441/error.log"} into lager_event
10:34:41.752 [debug] Lager installed handler {lager_file_backend,"/Users/adronhall/Codez/basho_bench/tests/20130808_103441/console.log"} into lager_event
10:34:41.757 [debug] Lager installed handler error_logger_lager_h into error_logger
10:34:41.758 [info] Application lager started on node nonode@nohost
10:34:41.767 [info] Est. data size: 488.28 KB
10:34:41.807 [debug] Supervisor sasl_safe_sup started alarm_handler:start_link() at pid
10:34:41.808 [debug] Supervisor sasl_safe_sup started overload:start_link() at pid
10:34:41.809 [debug] Supervisor sasl_sup started supervisor:start_link({local,sasl_safe_sup}, sasl, safe) at pid
10:34:41.810 [debug] Supervisor sasl_sup started release_handler:start_link() at pid
10:34:41.811 [info] Application sasl started on node nonode@nohost

....AND A WHOLE LOT MORE HERE....

10:35:42.954 [info]   {{{put_re,{"localhost",4567,"/","{\"this\":\"is_json_%%V\"}"},[{'Content-Type','application/json'}]},{put_re,{"localhost",4567,"/","{\"this\":\"is_json_%%V\"}"},[{'Content-Type','application/json'}]}},{put,{conn_failed,{error,econnrefused}}}}: 1701
10:35:42.955 [info] Application basho_bench exited with reason: stopped
10:35:42.955 [info] Test completed after 1 mins.
$

After letting the test run for it’s designated minute, run the following command to get some pretty graphs with R.

make results

…or maybe run…

priv/summary.r -i tests/current

The reason I post both is that ‘make results‘ doesn’t seem to always work to build the results and the manual execution will actually get the results built. With the results built, check the tests directory in the basho_bench directory for the summary.png file. If you open the file it should look something like this.

Default empty http.config results from basho_bench. (Click for full size image)

Default empty http.config results from basho_bench. (Click for full size image)

From here you can now run basho_bench and get the results that are specific to basho_bench. However, this now leads me to a higher abstract topic of why do benchmarking in the first place.

Why Benchmark? How to Benchmark!

The definition for benchmark,

bench·mark

[bench-mahrk]  noun 1.a standard of excellence, achievement, etc., against which similar things must be measured or judged: The new hotel is a benchmark in opulence and comfort. 2.any standard or reference by which others can be measured or judged: The current price for crude oil may become the benchmark. 3.Computers. an established point of reference against which computers or programs can be measured in tests comparing their performance, reliability, etc. 4.Surveying. Usually, bench mark. a marked point of known or assumed elevation from which other elevations may be established. Abbreviation:  BM adjective 5.of, pertaining to, or resulting in a benchmark: benchmark test,benchmark study.

While basho_bench provides an interesting baseline test that shows various pieces of data to work with, it shows nothing by default that is specific to YOUR use case. The basho_bench is not ideal for your production environment, it is not your dev or user acceptance testing or test criteria, it is an example. To truly get effective numbers that really encompass your needs for your project you will need to provide custom configuration for basho_bench or write your own specific benchmark.

The reason behind this is, with Riak as with other NoSQL solutions, is that you’re working toward a goal that is very data specific and unknown. It has specific domain logic and criteria that is specific to the use case, a custom benchmark can provide real data related to that domain logic and criteria.

In the end, even though basho_bench is a great tool to get started, do basic tests, and a great project to get ideas from it is not the panacea benchmark. You’ll need to create the specific benchmark for your use case yourself.

Happy hacking (and benchmarking).

OSCON : Conversations, Deployments, Architecture, Docker and the Future?

I wrote about my first day of OSCON “OSCON : Day 1, Windows Just Doesn’t Do Cloud Foundry… but, there’s a fix for that…“. The rest of the week was most excellent. I caught up with friends and past coworkers. I heard about people working on some amazing new projects. Some things I will try to write up in the coming days, as I’m sure some of it will be making the tech news (if not the regular people news too).

Conversations

Had some great conversations about the direction of enterprise and paas uptake. It’s great to hear that there is some movement in that space finally. As one would expect however, there is still a lot of distance for the enterprise to catch up on, but they’ll get there – or fall apart in the meantime.

There were also tons of conversation about the Indiegogo Ubuntu Edge mobile device. This device is a great looking and sounds like a solid idea. The questions arise in the fact that they’re working to make this a purely crowd funded project. This wouldn’t be a concern if they were trying to just get a few million in capital, but they’re aiming for $32 million! Overall though, with 128 GB, Dual LTE Antennas for Europe and the US, a top tier screen in quality and design, a metal body and also multiple other features that put this phone ahead of anything out there. I hope it’s successful, but I must admit my own hesitance. What’s your take on the device?

Deployments

Over the course of the conference I talked to and worked with a number of other individuals playing around with Cloud Foundry and also OpenShift. The primary aspect that we worked on was strategies around deployment of these PaaS Technologies.

We also worked with Iron Foundry to extend Cloud Foundry to support .NET. If you love .NET or hate .NET, wherever in that spectrum, it has an absolutely huge user base still. Primarily because .NET spent the last decade and a few years going head to head against Java in the Enterprise, and we all know the enterprise is slow to shift anything. So for now and the foreseeable future .NET is an extremely large part of the development world. Having it work in your PaaS is fundamental to gaining significant enterprise share. Cloud Foundry is the only open source, internally usable PaaS on the market today. There are closed source options available, but that obviously doens’t come up at OSCON.

While at OSCON, I also got to discuss architecture and deployment of Riak with a number of people. The usage of Riak continues to grow and the environments, use cases and tooling that people are using Riak with and for is always an interesting space for me. I also got to discuss deployment of Cassandra and even some Neo4j, Redis and Riak side by side deployments. People have used an interesting mix of NoSQL solutions out there to pull their respective data together for their needs.

Among all these deployments, conversations regularly returned to a known topic of mine. Cloud computing and who is capable of what, where and when. AWS is still an easy leader in cloud computing, not just in customers but in technology. This also brought up the concerns and apathy that some have around OpenStack (hat tip to Ben Kepes for the write up) working more homogeneously with AWS. Whatever the case might be, the path for OpenStack needs to be clarified regularly. I imagine the next movement is going to be away from being too concerned with infrastructure and increased concern with portability of applications and development of applications.

Another growing topic of discussion was around building applications for, on and with Windows Azure. Microsoft has actually become dramatically more involved in open source in an honest and more integrity based way. I’m honestly amazed at how far they’ve come from the declaration years ago that “open source is a cancer” and the all too famous, “linux is communism“. Whatever that was supposed to mean, they didn’t seem to get it back then. Now however, they regularly contribute to open source projects on codeplex but also github and other places. Microsoft has even contributed to the Linux kernal a few months ago.

That leads me to the next topic that came up a number of times…

Architecture

There’s been a lot of discussion about architecture around PaaS, containers (more on that in a moment), distributed systems in general and distributed databases. As I wrote about recently, “Architectural PaaS Cracks or Crack PaaS” the world of distributed systems and distributed databases has more than a few issues when working together in a PaaS environment. This brought up the discussion about what solutions exist today, solutions I look forward to writing and building in the coming months.

The most immediate solution to scalable data sources is still to run your operational data sources such as Neo4j, Redis, Riak or other database autonomously but residing close to your PaaS System. The current public PaaS Providers do exactly this and in some cases extend that to offer the databases and data sources as services through add-ons. These are currently great solutions, but require time, effort and custom development work when setting up internally.

This leads me to the last topic…

The Story of a Container – Docker

Well, not just Docker, but containers in general and Docker specifically. First some context about what a container is.

Container – In this particular context I’m writing about a container, or more specifically a runtime-container, that isolates resources for applications or services. Containers are common in PaaS technologies to help isolate the specific services or applications when they’re on a single physical machine or instance. For each of the respective PaaS systems that came up at OSCON we have dotCloud from the same team that created Docker, Cloud Foundry has Warden and OpenShift has gears and Red Hat Enterprise Linux OS specific containers.

I’ve studied Warden a little in the past while I was working with AppFog and Tier 3 around Cloud Foundry. Warden is a great piece of technology. However the star at OSCON was clearly Docker. I jumped into a number of conversations around Docker. This conversation would then take the direction to containers becoming the key to PaaS tooling and systems growth and increasing capabilities. That leads me back to my previous blog entry “Architectural PaaS Cracks or Crack PaaS” and one of the key solutions to the data tier issue.

Containers, A Solution for Scaling the Data Tier

One of the issues that comes up when trying to scale any distributed database in a PaaS Environment is how to provide multi-tenancy without spooling up new instances for each and every single installation of a node within that distributed database. Here’s an example diagram of the requirements behind a scalable distributed database.

Masterless, Distributed Cluster of Nodes

Masterless, Distributed Cluster of Nodes

In a default configuration you’d want each node to be running on a physical machine or dedicated virtual instance. This is for performance reasons as well as reasons for load balancing, security, data integrity and a host of others. This is the natural beginning state of a highly available distributed database or distributed system.

Trying to deploy something like this into a PaaS environment is tricky. Take into account that there is no such thing in application or service speak as an instance, and especially not anything such as a physical server. The real division between process and resources are containers. These containers are what actually needs to run the distributed system node. This becomes possible, if a distributed system node can be deployed to and executed from within a container.

Enter Docker

After reviewing Docker, the capabilities around it and the requirements of a distributed database, it looks like an ideal marriage of the two technologies. Already Docker has Redis and other database technologies running on it. The Container technology around Docker looks like an ideal fit to extend distributed systems to run autonomously of a single physical machine or single instance per node. This would enable nodes to be deployed as resources are available to provide a more seamless and PaaS style deployment for systems like Cassandra, Riak and related distributed systems. Could this be the next evolution of affordable distributed systems, containers to the rescue?

I’ll be reporting back on my progress, this could be cool!

Stay tuned for a write up on Docker in the near future. For more information now check out http://www.docker.io.