I’m in the process (with the team I work with) of trying to figure out what would be most useful to you, the community and its members in which we all work. Whether you’re a coder, working toward being a coder, programmer, engineer, or whatever it is that you aim for we want to know what would help you out? I myself produce a ton of material that I personally find entertaining and fun to produce myself, and hope it’s useful for people. So – if you would, take a moment and answer these few questions. Thanks and cheers!
Tag: oss
Riak is… A Whole Big List of Things
What is Riak? Who builds it? Who maintains it? Can I download it? How does it work? What are the features?
Here’s the start of answers to these questions and more.
First, the basic high level description:
“Riak is an open source, highly scalable, fault-tolerant distributed database.”
That’s the first line you’ll read when checking out the product via the Basho product link. It provides good information, but here I’m going to add more to the definition without the need to dig around yourself. Maybe I can save you some time & provide some links directly to solid information in the docs. Kind of a “Cliff Notes” of Riak. Let’s take this feature by feature which will in turn get us to a definitive definition of what exactly Riak is.
Riak is Open Source.
Riak is built and contributed to by the community, with Basho being the steward and an active member that extends, builds and provides support for additional products. The avenues to reach the Riak Open Source Community members is pretty straight forward, following known avenues of communication. Hit us up on the email list, especially feel free to contribute & ask questions via the Github Basho organization, there is the Basho Riak Blog, the weekly recap and jump into the IRC chat room #riak on freenode. Oh, and there’s a twitter feed @basho.
So what exactly does this get you, when you become a user or contributor of Riak? The entire community is behind you, will help you get started using Riak and provide help whenever you run into problems. If you want SLAs or 24 hour support Basho can provide this for you. But for bugs, issues, queries, searching and all sorts of other related development questions there is the community. An open source community like this is passionate, which means you’ll have support like no closed source company will ever provide you, and absolutely no closed source product’s community will provide you. We’re talking about a different level of interest, passion and levels of personal involvement.
Riak is a key value based database store.
Riak is a key value store. What exactly is a key value store? It’s pretty simple and you’re probably already familiar with what a key value store is. A key value is made up of two pieces of data, the first is the identifier for the second element within the data structure. This gives a system or developer using key value storage a schema-less way of working with data.
Riak is designed for highly distributed environments.
This type of distributed isn’t the “we put one database over here and one database over here and you gotta figure out how they work together” type of distribution. So this isn’t some of that oddball pretend stuff Oracle keeps hoisting on people. This is the honest to goodness distribution of the sort, when one node goes down you don’t blink, you don’t stop eating dinner, you don’t sweat it. You just continue onward with life knowing full well that you’ll just spool up another node when you need to.
Riak is master-less, with no single point of failure.
This is one of self explanatory features. But what does a master-less system provide us? One thing is no single point of failure. Being that all nodes can act autonomously to work around the loss of one or more nodes it also helps add to the high availability of the system.
Riak is fault tolerant, like a disk drive you wish was real.
Ever have a backup disk drive? What? You don’t have one of those? Ugh. Ok, so imagine you had a backup disk drive that had an unfortunately high failure rate. Well, why, because you know, they have an oddly high failure rate. If you do backups like good practices dictate, eventually you’ll end up with some dead drives.
RAID, both software and hardware, are built specifically to deal with this type of failure. With a distributed system like Riak, it bumps the level of abstraction above software or hardware RAID, enabling another level of even greater fault tolerance. Not to remove the relevance of RAID capabilities, but with a multi-node system like Riak, you can easily remove nodes and swap them out as needed, keeping costs down by using simple drives in simple machines. If you want to, you could indeed get higher I/O machines and faster drives, but it isn’t necessary to insure fault tolerance in a Riak Database System.
Riak scales, with hot swappable nodes enabling zero downtime.
The ability to commit hot swappable changes while in the midst of operating starts at a very low level for Riak. The language used to build Riak, Erlang has the ability to change pieces of an application system in realtime built into the precepts of the language. This provides, at the core, the inherent capability to change out systems, and by proxy of architectural design, the ability for nodes in Riak to be changed out simply by removing them from a cluster ring. Once that is done it is just as simple to add another node or nodes back into the cluster ring, enabling a number of additional practices around upgrades, hot swaps for failures, or even version changes.
Riak can be used as a building block for distributed (aka cloud) infrastructure.
The concepts and contractual components that Riak Database is built on are available for use via the Riak Core Project. If you’re looking into starting a project around distributed systems this is a great place to get start. Also be sure to do a general web engine (re: google) search for “riak core” and you’ll find lots of material around the project, and projects people have started with the project as a base. I’m currently in the process of putting together one of these projects myself.
Riak is eventually consistent.
The term eventually consistent is becoming more and more common place. Riak is one of the many systems, that inherently often apply to distributed systems, that use the concepts of eventual consistency. The idea, is that even though all nodes may not immediately receive a new piece of data, or updated piece of data, they eventually will receive that update and by synchronized with the cluster ring of nodes. This goes back to the equality of nodes and removal of the master-less concepts, providing the availability and other capabilities, with some trade off in the synchronization of data through eventual consistency.
In Summary
That’s round one for the many features of Riak. I’ll be adding more in the future, but for now this is a good starting point in knowing about and knowing what Riak is, what it can be used for, and how it might help you extend, maintain or invent the next great piece of technology.
What really is Open Source Software and what’s this community nonsense they ask…
Open Source Software (OSS), Why Some Fail At It
OSS has won the war. It has been over for years now. Microsoft has ceded, Oracle, VMware and many others have stepped up and attempted to embrace the open source community. Sometimes they’ve been successful, sometimes they haven’t. They’re slowly changing their models to play well with that of the open source software model. Sure, some software is kept closed, but that software in large part is becoming more and more irrelevant while open source efforts are becoming the forefront of technological progress.
What exactly is open source, besides just the opening up of code for others to download? Open source covers a vastly larger ideal than merely providing code for download. A case in point, has been the learning phase Microsoft has gone through. Microsoft, as a company, used to attempt to dictate to its consumer & developer base standards and practices that the company had deemed necessary or in some cases merely a good idea.
Microsoft failed at this miserably over the last decade. Time and time again an open source project would start and Microsoft would create a duplicate library – sometimes directly taking the OSS Project exactly functionality, sometimes they’d merely duplicate it with a basic understanding. Everything from Entity Framework duplicating the functionality of the dozens of ORMs before it. Basically stabbing those efforts in the back instead of being part of the community, Microsoft would remove itself and attempt to subjugate the community efforts.
It back fired over and over and over…
Oracle did something different, yet still blindly stupid. They purchased entire OSS Projects from mySQL to Java. In each case they’ve tainted the efforts significantly by attempting to make these products encourage an unspoken lock in to their proprietary tooling all while litigating (re suing). Their attempt to patent (another issue we can discuss later) the most absurd features and functionality, akin to patenting the breathing process in people! In addition they’ve tried to set legal precedent for things as simple as a URI end point and other notions. Again, something that hasn’t gone over well in the software development world. With the current result being a growing backlash against Oracle. To top all that off, their patent cases have been far worse than even Apple’s Samsung debacle. Oracle, has taken the crown for stabbing the development community, and especially the OSS Community in the back. Multiple wounds too, not just once. They’re doing it over and over as I write this.
Others have continued to make this mistake. They’re starting to suffer for it, and well justified that they do. To stab the community is more than just merely disrespecting one or two developers. It is desecrating the entire community’s efforts, the individuals and their thoughts, ideas, creations and more. It is a slap in the face in so many ways.
Some are starting to do it right, albeit slowly…
Some companies have started to get their act together. One company that is learning right now, slowly but steadily and confidently (they have a good team working on this) is VMware. With their introduction of Cloud Foundry, some could argue this, but they have generally and are trying diligently to open up and be inclusive in the community around their Cloud Foundry Product. Yes, I might have a slightly biased view since I build products for Cloud Foundry with the Iron Foundry Organization at Tier 3, I talk to the teams & individuals at VMware and they, with all their might, intend and do the best they can. As in almost all cases, as long as management keeps everything in their heads clear, they’ll maintain a great project and the future is bright.
Microsoft is another company, with tons of closed things, many attempts at opening products, and is finally starting to get it. They’re starting to be part of the community instead of trying to dictate to it. The windows azure team directly involves itself with jQuery, Node.js and other projects these days. They actively put forth a good foot and have opened up Web API and other web application frameworks and pieces, allowing for pull requests and openly having conversations int he public for full view and inclusion with the community.
So What Really Is Open Source?
Open source software itself, just the code, is simple. But it is assumed and written into the all legal licenses that are included to protect the software from theft and closing by errant companies & individuals. Open source software is code, used as examples or as production web sites is software that is available freely to others to learn from, fix, change, or otherwise modify. It often excludes sharing and using the code with closed source environments or redistributing with closed source products – because OSS efforts do not want to encourage the bad behavior and errant ideals in closed source software by contributing to it.
Building OSS includes a very specific idealism. One doesn’t just throw something into the code, one encourages and builds a culture of openness and being free in thought when working with and contributing to the code bases. The OSS Community is about sharing ideals between individuals to accelerate learning, expand the capabilities of the community as a whole, and push forward progress and development. To summarize with a standard quote, “open source software is about freedom”, and it truly is.
So far, it’s working in a huge way. Some simple successes that have been massive…
OSS Victories
Linux & FreeBSD pretty much hosts the Internet. From Facebook to Google to Amazon they all use some type of Linux variants. Estimates range from 60% to 93% of the Internet & Super Computing is hosted on UNIX machines of the Linux or FreeBSD variety. The notorious Microsoft Windows Server only claims 0.4% of the super computer space and about 30-37% of the Internet Server space. Summarized, the Internet runs on UNIX and specifically on variances of Linux.
When it comes to the web, not only is the majority of the web hosted on systems built by the open source community, but the web applications hosted and run on those systems are open source. The most widely used framework in the world is PHP. One of the biggest up and comers for serving websites and providing interactive web applications is Node.js, with Ruby on Rails being a stalwart for speedy prototyping and production application for thousands of businesses.
Mysql & postgresql round out two of the most heavily utilized databases in existence. Postgresql has grown from zero code to a massively capable database, regularly one-upping the stalwarts at Oracle, Microsoft or other database makers. Mysql has become the go to database for those starting a website or collecting 60 billion rows of data a day, such as New Relic. Many successful businesses have turned these databases into absolutely powerhouses that truly eclipse the need to expend the revenue on databases like SQL Server or Oracles Databases.
NoSQL has come into existence and exists today because of the community. Not everyone has a big data problem and a need for a NoSQL database, or what is sometimes called Not Only SQL these days. The open source community stepped up to build out solutions where relational databases and their history of vertically scaling falls down for modern web applications that run at larger than normal scale. Companies like Facebook, Twitter and others have helped to bring people into this fold and bring more great minds developing the open source that powers these things. This entire movement has been a huge win for the Internet and increased functionality – and the mere ability for many of the large sites to continue to exist at scale (think LinkedIn, Netflix, Facebook, etc)
O’Reilly Books is a company that does a number of things, two of which are run conferences and publish books. They’re a well respected company that encourages open source software and learning through its books, conferences and is heavily involved in supporting the individuals and community around open source software. Even though they don’t create open source software, they provide a massive boost to the ability of developers to write and create open source software by mere involvement. This is a perfect example of example from outside of software development that gains value and adds value back to the community as a whole.
That’s just a few. So really, there’s no reason to resist the freedom of open source. Ignoring or turning away from open source is like turning your back on your family, your community and those around you. There’s no reason to believe the nonsense about TCO (Total Cost of Ownership) that’s been wielded as a weapon against open source solutions. There’s no reason to listen to the FUD about “will it survive” or “what happens when X happens to the project”. There are many companies out there that will help you understand these solutions further, so the TCO argument is dead on arrival. The only path here is toward more software freedoms, preventing large companies from limiting our development options (re: Oracle suing to control API end point pathing, etc). When you do development these days, your first option and thought should be to open source “all the things”. It will help you as a developer, it will be better for you company and those that are leading the way, and it will be better for the community as a whole.
Stay relevant. Write code, individually contribute and be part of your community.
I’m Adron (@adron twitter or @adron app.net) and I write open source software.
Thor Project Opens Up, Building the Cloud Foundry Ecosystem with the Community
The Iron Foundry Team are big advocates of open source software. We write code across all sorts of languages, just like many of the development shops out there do. Sometimes we’re heavy on the .NET, other times we’re all up in some Java, Ruby on Rails, spooling up a Node.js Application or something else. So keeping with our love of open source and our polyglot nature we’ve created the Thor Project with three distinct apps.
Before jumping into the applications though, a little context for what and where Thor is in the grand scheme of things. We need to roll back to the Cloud Foundry Project to get into that. The Cloud Foundry Project is an open source project built around software for PaaS (Platform as a Service) which can be used to build your own PaaS internally or externally, in a cloud provider or directly on hardware. It’s your choice how, when and where you want to use it. For more context on PaaS check out my previous entry “The Confusions of IaaS, PaaS and SaaS“.
Thor Project
Cocoa for OS-X
You know who Thor is right? He’s this mythic Norse God, also known as the God of Thunder. Since we’re all about bringing the hamma we welcomed Thor into our team’s stable of applications. So starting immediately we’ve released Thor into the realms for contributions and fighting the good open source software battle! If you’d like to join the effort, check out the github project and feel free to join us!
Technically, what is the Thor Application? This is a Cocoa Application built for OS-X that is used for managing, deploying and publishing applications to Cloud Foundry enabled and or Iron Foundry extended PaaS Environments.
.NET for Windows 7
The .NET Metro version of the Thor Application is also released via github with a provided installer. We’ve almost taken the same path, except of course for the very different UX and UI queues with Windows 7 and the Metro UX design guidelines.
WinRT for Windows 8
I wasn’t really sure what to call this version. Is it Metro or WinRT or Windows 8 or something else? Anyway, there is a project, it is albeit empty at this point, but it is the project where the Windows 8 version of Thor will go! For now get the Windows 7 version and install it on Windows 8, it won’t have touch interface support and things, but should work just like a regular application on Windows 8.
The Code
- Thor OS-X (Cocoa)
- Thor .NET
- Thor WinRT
- Thor Design – This is all the design collateral created by Nathan Young (@nathanyoung) for each of the versions.
To get started with these, generally you’d just clone the repo and do a build, then get started checking out the code. There is one catch, for the OS-X version you’ll want to pull down the sub-modules with the following command.
[sourcecode language=”bash”]
git clone git@github.com:YourForkHere/Thor.git
git submodule update –init –recursive
[/sourcecode]
Once you do that in XCode just make sure to then select the right project as the starting build project.
…then when the application is launched…
I’ll have more in the coming days and weeks about Thor & Iron Foundry. For now, check out the blog entry on the Iron Foundry Blog and subscribe there for more information.
OSCON Day #3, #4, and Friday => Bailey’s Taproom, Cloud Camp, Cloud Foundry, Open Shift, PaaS, vert.x, and so much more…
Tuesday night, as usual ended with great technical conversation at Bailey’s Taproom. Bailey’s is basically the epicenter of the Portland tech scene. Almost every programmer, devops, or technical person either goes about once a month or has this establishment as a regular watering hole! It’s great, the atmosphere is chill, the beer is SUPERB, the beer menu kicks ass (see: Beer Dashboard Kick’s Ass) and the list of fun cool things just continues on and on.
This week of course OSCON adds a little spice to the regular roll call at Bailey’s. There were a number of conversations that broke out, which I’ve broken out the key topics below:
vert.x => To summarize as is written on the site itself, “Write your application components in JavaScript, Ruby, Groovy or Java. Or mix and match several programming languages in a single application. Create real, scalable applications in just a few lines of code. No sprawling xml config. Scale using messaging passing and immutable shared data to efficiently utilise your server cores. Super-simple concurrency model frees you from the hassles of traditional multi-threaded programming.”
Here’s an example from the site in a few of the languages:
Java
[sourcecode language=”Java”]
import org.vertx.java.core.Handler;
import org.vertx.java.core.http.HttpServerRequest;
import org.vertx.java.deploy.Verticle;
public class Server extends Verticle {
public void start() {
vertx.createHttpServer().requestHandler(new Handler() {
public void handle(HttpServerRequest req) {
String file = req.path.equals("/") ? "index.html" : req.path;
req.response.sendFile("webroot/" + file);
}
}).listen(8080);
}
}
[/sourcecode]
JavaScript
[sourcecode language=”JavaScript”]
load(‘vertx.js’)
vertx.createHttpServer().requestHandler(function(req) {
var file = req.path === ‘/’ ? ‘index.html’ : req.path;
req.response.sendFile(‘webroot/’ + file);
}).listen(8080)
[/sourcecode]
Ruby
[sourcecode language=”Ruby”]
require "vertx"
Vertx::HttpServer.new.request_handler do |req|
file = req.uri == "/" ? "index.html" : req.uri
req.response.send_file "webroot/#{file}"
end.listen(8080)
[/sourcecode]
Wednesday Roughness
I felt beat up a bit start Wednesday, but rolled into it after a short while. Needless to say, the intensity of conversations (and maybe a few of those rounds of beer) and number of ideas, new things to check out and fitting it all in can wear one out.
The morning sessions were solid, I attended most of “Comparing Open Source Private Cloud Platforms“. Lance did a solid job of laying out the tooling, virtualization software and where these things come together to form a number of OSS options for cloud computing. Check out more from Lance on his @ramereth, his blog Lance Albertson, or check out his band he’s in “The Infallible Collective“.
Wednesday, Thursday and Friday Meets
I met a ton of people. All of whom I must say, I hope to get to talk to again, work with on projects, or just sling some code sometime. Absolutely great people, friendly, intelligent and highly motivated. Some of these people I met included:
Andy Piper (@andypiper) – Part of Great Britain’s contingent of VMware Cloud Foundry advocates and such. We got to hang out and talk about a zillion different topics at a number of events. Andy was kind enough to show me a few tips and tricks he’s been using with Cloud Foundry, the VMC, and in general working with the platform.
Josh Long (@starbuxman) – I met Josh once before on the Cloud Foundry open tour, where he brought COBOL programming… oh no wait, he brought some great Sprint Java samples and such to demo on the Cloud Foundry Platform. I fulfilled Josh’s dreams by telling him that COBOL, could indeed run on Cloud Foundry thanks to the .NET capabilities of Iron Foundry! (ya know, if anybody is into that type of thing)
Erica Brescia (@ericabrescia) – I finally got to meet Erica in person, after chit chatting on Twitter about all the great applications her company Bitnami helps to deploy in the cloud. There are some really great deployment hosting solutions from them, check them out if you’re looking for some streamlined deployment practices. She also mentioned I need to meet…
Jono Bacon (@jonobacon) – I managed to meet Jono by randomness. He’s, well, let’s say he does some absolutely great work in the tech industry for Canonical and in the open source universe. In addition Jono has some superb tastes in music. \m/ \m/ Check out some of his work: Blog, personal site, and you can probably google him too. Do it, he’s got a lot of great material out there.
As I was saying, these aren’t the only people that I met. To all those people I didn’t mention, it was awesome hanging out, catching up and hearing about what everyone is working on and creating.
PaaS, IaaS and The Driving Open Source Coders
On the topic of PaaS, it continues to expand into new realms of publicly (or privately) run services. PaaS is quickly expanding past mere framework services around .NET, PHP, Rails, Sinatra and such and moving into the realm of databases, services buses, and other capabilities as a service. As laid out with the SOA mindset. Even though enterprises failed to bring SOAP to an effective worldwide use, RESTful services are expanding rapidly. *aaS is pushing those even further, to do what the enterprise had wanted but failed to do. Creating a universal acceptance of scalable, powerful, expandable and extensible services through APIs.
As more services are extended we’ll start seeing a lot of offerings around truly scalable databases with various feature sets around those databases offered as a key service. Examples would include “atomic database as a service”, “transactional data store as a service”, or “document store as a service”. In the end it will include the amount of usefulness for the services while eliminating a need to know each in intimate detail. Knowing the core capabilities of an option and just using the service will grossly outpace the attempt to implement these services internally.
So keep watching PaaS to grow in many various ways. Consuming the service being the driver over attempting to build the service. Of course, if the service doesn’t exist, get on that it’s business opportunity!
Random OSCON Diversions
I had a great time visiting with family while at OSCON also. To whom they all send a hello and horns up, thrash on salute to the coders of the world!
My Brother Runs an IT Shop of One…
…thanks to cloud computing capabilities.
This kind of blew my mind. I sort of of knew what he did, but it didn’t hit me how close our professional lives are until this trip. He’s just recently moved several hundred miles away from the main office, but still manages the entire company.
One of the unique happenstances is, my brother (the guy next to the bald guy that is me, he’s wearing a Tesla t-shirt) is the top IT guy for a little billion dollar a year company. Which, in this case, he’s proven the power of cloud computing. Why do I say this? Because traditionally this organization would have needed an army of PC techs, network knob fiddlers, and such. But with the advantages of cloud computing, both on premise and off premise, and have a DevOps Guy that knows what he’s doing they are able to efficiently run their entire company with one single guy.
Needless to say, with the synergy of OSCON we had more than a few conversations around tech. Some of those included the replacement of PCs with mobile devices, such as iPads or smart phones. Another was the mix of on-premise data that couldn’t easily be transferred or utilized form cloud services. These are just a few fo the things that have helped him to run the show, the entire show.
Summary
OSCON was awesome. Next time I will be taking off a day or two before and a day or two afterwards so that I can do an even more elaborate write up of the event. My aim is to have interviews, video and otherwise, and really step it up in relation to providing an eye into the event from a developer’s point of view.
You must be logged in to post a comment.