Node PDX – Introducing Scott Hanselman, Tracy Abrahms, Matthew Lyons & J Chris Anderson

Welcome to iteration 3 of introductions.

Scott Hanselman is presenting…

Azure for the non-Microsoft Person – How and Why?

Scott Hanselman

Scott Hanselman

Scott is a web developer who has been blogging at http://hanselman.com for over a decade. He works on Azure and ASP.NET for Microsoft out of his home office in Portland. Scott has three podcasts, http://hanselminutes.com for tech talk, http://thisdeveloperslife.com on developers’ lives and loves, and http://ratchetandthegeek.com for pop culture and tech media. He’s written a number of books and spoken in person to almost a half million developers worldwide.

For a dose of Scott, check out how to scale to infinity by doing NOTHING!

Join Hanselman as he digs into the open source SDKs of Windows Azure. Let’s access Azure from the command line and deploy and redeploy with Git. We’ll fire up Linux VMs, setup Mongo and run node.js apps in the cloud. We’ll look at things like SendGrid and New Relic. The future of the cloud is open and it’s a hybrid. This very technical session will cover Windows and Mac, .NET as well as pinching pennies in the cloud.

Tracy Abrahms is presenting…

Punching Rocks: my intimate adventure with rock climbing and becoming a programmer

Tracy Abrahms

Tracy Abrahms

Tracy tells a story, “A funny thing happened at the rock gym… I kept running into programmers. Rock climbing is a constant challenge. Physical? Hardly. Tired muscles is a concern once you’re two pitches up and can’t figure out the next move. I’m afraid of heights! The psychological and mental tenacity required to complete a wall feels eerily similar to the daily challenges of the Programmer. You will commit yourself to situations that you pretty much HAVE to find a way out of. Sound familiar?

Malcom Gladwell’s 10,000 hours–I’m not that far in. How are people learning to program nowadays? MOOCs, tutorials, workshops, communities(meetup.com), books, standard CS degrees(what of non-tech degrees?), internships, apprenticeships, code schools. What am I doing? What have I done? How many callouses have I build in the process? Experiences shared from my own perspective and others I have met on my journey have shown me a number of great ways to help move forward those willing to take up the challenge.

Finally, what can I do, along with knowledgeable and helpful Node.js programmers, to build the knowledge base and accessibility into the community? How do I get programmers hooked? And how can I convince all of these brainiacs to get out and punch a few rocks?”

Matthew Lyon is presenting…

Let’s Make Music

Matthew Lyon

Matthew Lyon

I got to work with Matthew @mattly for a short period while I was helping out at AppFog a while ago. I also have had the great fortune of catching Matthew talk about JavaScript magic and more!

Also a homebrewer, hacker, artist, photographer Mattew Lyon is coming to give us a lesson on making music. He describes his course as thus, “Music happens over time and so does asynchronous code. Therefore, let’s write music in Javascript! We’ll build a network sound sequence server that will talk to freely-available softsynths and possibly an HTML5 Audio API, covering how to model things such as drum patterns, melodies, oscillators and pattern banks.”

J Chris Anderson presenting…

Hands On Realtime Text Analytics

J Chris Anderson

J Chris Anderson

J Chris just recently returned to Portland because of many different reasons after hacking in the San Francisco area. If you know what livable streets are (re: Bike Portland and if you intend to stay and live in Portland, you probably should get in on this, livable streets are too good to merely pass up), you’ll want to keep tabs on J Chris’ efforts ongoing, I definitely will be! J Chris also refers to himself as a hacker dad, couchbase cofounder & mobile architect.

With “Hands on Realtime Text Analytics” J Chris will present to us  as described:

“A common problem in large scale computing, is coordinating workers when they can be scattered across compute nodes. For workloads like this, atomic operators like increment and decrement reduce contention between distributed processes.

In this talk I’ll show a full text analysis tool which ranks words in the Twitter firehose. By storing each token in a key based on its characteristics, we can provide word rankings both globally, as well as over time and space.

We’ll show the running application, and take a tour through the code, as well as discuss cluster sizing and how it is impacted by variations in the input data stream.

For instance a tweet in English from San Francisco might say “Go Giants” so counters for 2012:go and usa-sf:2012-07:giants (among a few dozen others) are incremented. Even using memory like this, the counts from a full corpus of English text would only take a few gigabytes to hold, making this architecture more efficient than a traditional index-and-rollup approach.”

Are you signed up?  BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

Backup Riak – Learning About Distributed Databases :: Issue 001

I’ve got more than a few series in the queue, so why not another one eh! The intent is, I’ll grab a specific topic to break down and add details to related to distributed systems, primarily around Riak. I will however diverge into other distributed databases too, but I’ll primarily be sticking to Riak. Without more introduction, the first topic is…

Backing Up and Recovery of Riak (Nodes)

I’ve been asked approximately 423,983,321.7 zillion times how this is done. So here’s a quick summary and respective links to the best ways to backup Riak, how to recover nodes.

When backing up Riak there are two key things that need copied to the backup storage; the ring and data directories. Each of these things are specific based on the backend used with Riak. In addition to the core backup containing the ring and data, another good thing to backup is the configuration directory. When recovering this comes in useful.

For the locations of data, it depends slightly based on the operating system being used. The two big variances are OS-X and Linux Distros. On OS-X the data path, ring data and configuration are located at the locations listed below:

  • Bitcask data: ./data/bitcask
  • LevelDB data: ./data/leveldb
  • Ring data: ./data/riak/ring
  • Configuration: ./etc

For each specific distro, there are slight variations on where the locations are, for a full list check out the Basho Riak docs on backups. But on Linux distros the paths are as follows:

Debian and Ubuntu

  • Bitcask data: /var/lib/riak/bitcask
  • LevelDB data: /var/lib/riak/leveldb
  • Ring data: /var/lib/riak/ring
  • Configuration: /etc/riak

Fedora and RHEL

  • Bitcask data: /var/lib/riak/bitcask
  • LevelDB data: /var/lib/riak/leveldb
  • Ring data: /var/lib/riak/ring
  • Configuration: /etc/riak

Other Operating System Paths

Freebsd

  • Bitcask data: /var/db/riak/bitcask
  • LevelDB data: /var/db/riak/leveldb
  • Ring data: /var/db/riak/ring
  • Configuration: /usr/local/etc/riak

SmartOS

  • Bitcask data: /var/db/riak/bitcask
  • LevelDB data: /var/db/riak/leveldb
  • Ring data: /var/db/riak/ring
  • Configuration: /opt/local/etc/riak

Solaris

  • Bitcask data: /opt/riak/data/bitcask
  • LevelDB data: /opt/riak/data/leveldb
  • Ring data: /opt/riak/ring
  • Configuration: /opt/riak/etc

When backing things up, it’s important to note that each node could have slightly inconsistent data. The data however is rebuilt by the Riak read-repair system once it is recovered and brought into use.

Backup Jobs

One of the easiest ways to backup Riak is to setup a cron job with your choice of cp, rsync or tar. Then just get those files onto whatever your choice of backup medium. An example tar cron job to backup a Bitcask backend is shown below (snagged from the documentation) just to give you an idea of where to start.

tar -czf /mnt/riak_backups/riak_data_`date +%Y%m%d_%H%M`.tar.gz /var/lib/riak/bitcask /var/lib/riak/ring /etc/riak

For a leveldb back end the most important thing to note is that the node must be stopped. The basic workflow of backing up a node in this manner is to stop the node, backup the data, ring and configuration and then start the node back up.

Backup Recovery / Restoring

When recovering data on a node that is replacing an existing node that has the same name (fully qualified or IP) then follow the steps below:

  1. Install Riak
  2. Restore the old node’s configuration, data & ring.
  3. Start the node

Once you’ve got the node started back up it’s a good idea to do a ping or status against the node to verify it is in a good state.

If node names have been changed there are additional steps.

  1. Mark the original instance down

    riak-admin down
    
  2. Join the restored cluster

    riak-admin join
    
  3. Replace the original with

    riak-admin cluster force-replcae
    
  4. Get the cluster plan built

    riak-admin cluster plan
    
  5. Commit the changes

    riak-admin cluster commit
    
  6. Change the -name setting in the vm.args configuration file to match the new name.
  7. Change & verify that the IP reflects the instances IP in the app.config for http and protocol buffer interfaces.

Cluster Backups via Riak Enterprise Multi-Data Center (MDC)

In the above sections I wrote about the traditional backup approaches. This is very similar to the way RDBMS are backed up. However, with a distributed system like Riak there is another great alternative if you’re utilizing multiple datacenters and Enterprise Riak. In this version of Riak, which is basically Riak with additional features and capabilities, one of the possible backup scenarios is to use the Multi-Data Center, or MDC, to replicate a duplicate cluster and use it as an active, real-time and always ready backup.

One workflow that is an exceptionally effective way to provide backups is to setup the “backup” cluster beside the current operative cluster. As an example, if your cluster is operational in AWS and it is running in X region and Y zone then you’d want to put the backup cluster in that same region and zone. Once you’ve setup Riak Enterprise and MDC, then just setup a full sync. Once the full sync is done you can then remove the backup cluster and it provides a point in time backup of the data.

riak-repl start-fullsync

It’s easy to schedule full sync operations to low usage periods and it is also possible to pause and resume full sync operations.

riak-repl resume-fullsync<br />riak-repl pause-fullsync

The variations on backing up data with Riak Enterprise and MDC are pretty expansive. Doing a point in time, maintaining a secondary live copy of the data, using the replication as a data dump to another cluster or even just using the MDC replication to dump all of the data to a single instance.

File System Snapshots

One other technique that is extremely efficient, fast and thorough is snapshotting the file system. The backup workflow for snapshots is extremely easy. First stop Riak, then snapshot, then start Riak again. Of all the methods, snapshotting is one of the easiest of the options. Just like setting up a cron job, automating snapshots based on some pre-defined schedule and meshing that with automated start and stop of Riak provides a very thorough backup.

With these options, have fun strategizing your stratagems into strategies for backups.

Diskettes

One of the oldest, tried and true backups is the old diskette. The bestest way to backup with diskettes is to backup each node on three diskettes each. The send one of each diskettes to a geographically dispersed to a bank lock box or other secure facility. Do this for each node, and if need be use as many diskettes for each node as needed. A particularly useful method is to use the sharded zip strategy to stripe a backup across many diskettes. Once each lock box has a copy of the node for each node in the cluster, you’ll have one of the most secure backups in existence. Nothing compares to the diskette backup!

References:

  1. Basho Docs – Backups
  2. Basho Docs – MDC Full Sync

Node PDX – Introducing Jason Denizac, Thorsten Lorenz and Peter Braden

Here’s the second of a series of introductions for the upcoming Node PDX Conference while I train ride up to Seattle today.

Jason Denizac presenting…

Programming With a Purpose

Jason Denizac

Jason Denizac

I met Jason over a year ago while he was in Portland at a coding conference. Since then we’ve kept in touch, and everytime I’ve enjoyed seeing him speak. This guy is from a place far over the hills and valleys somewhere in that central part of the United States, but I’m stoked he’s out here on the west coast now bringing more brainpower to the region.

Jason described his talk, “As programmers we are wizards. Our job is to manufacture super powers. Like with the Manhattan Project, wielding such great technological power entails moral implications which we ignore at our peril. But we can recognize this power, embrace it, and use it for great good. Node gives us a tool kit to confront great problems and share in solving them. Specifically: radical decomposition, horizontal reuse, and positive community norms around testing and documentation.

Audience participation requested: be prepared to share your expertise in a real world problem space like education, civics, social equity, environmental conservation, healthcare, or your own favorite “intractable” problem. If time and scheduling permits, I’d love to have a series of lightning talks in which people could introduce other developers to the problem domain.”

This should be a good one.

Thorsten Lorenz presenting…

Module Driven Programming

Thorsten Lorenz

Thorsten Lorenz

Thorsten Lorenz has been creating nodejs projects for over two years mostly in his spare time. The main reason why he can’t (and won’t) stop is, that ideas can be transformed into something that works in a very short amount of time when implemented in JavaScript. Aside from these projects, he also contributes to projects important for the community, one of the recent additions is source map support to browserify. These source maps even allow debugging transpiled languages (i.e. CoffeeScript) right in the browser.

Thorsten threw a bullet point list of the key points for his presentation, which follows:

  • what constitutes a module
  • why it is desirable to build smaller modules
  • challenges and patterns for separating the application into independent modules
  • process of pulling out a module from an application
  • replpad case study
  • how to become module driven phase1 and phase2
  • browserify and how it enables to even manage your client side modules with npm
  • quick primer on tools like npm init, pkginit, travisify and npm link that help with module driven development

Peter Braden presenting…

Teaching Robots to See With Javascript

Peter Braden

Peter Braden

Peter Braden is the author of the node-opencv bindings that allow node scripts to interface with this powerful computer vision library. Peter related to us that he’s both excited and terrified for a future of seeing drones and robots. Currently he works as a web consultant-of-all-trades at frozenridge.co.

In his talk he’ll be diving into, “As we start to use javascript to control quadrocopters and robots, it becomes important that they can see with images from cameras onboard. In this talk we’ll take a step back and look at the field of computer vision; look at some of the exciting practical problems that can be solved with CV techniques, and look at how we can do this with javascript. Whether it’s detecting and recognising faces, building a picture of surroundings from a video stream, or tracking shapes, we’ll look at how we can make our javascript programs see. And all going well, we’ll have a live demonstration of a drone navigating based on onboard camera data.”

…

…

…ready to get your tickets?

Are you signed up?  BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

Node PDX – Introducing Ward Cunningham, Nexxy, Jerry Seivert and Hannah Fousanon!

Here’s the first of a series of introductions for the upcoming Node PDX Conference here in bright and sunny Portland, Oregon!

…and no, that isn’t sarcasm, it’s the summer time now so we’re allowed to have sun and warm weather! With that, the speakers for Node PDX!

Ward Cunningham presenting…

My Sensors Love Node.js

Ward Cunningham

Ward Cunningham

Who’s Ward? In his own findings, “The Oregonian describes me as the Old Growth of the Silicon Forest. I appreciate the thought but move way too fast to be compared to trees. Think of me as your intellectual immune system separating good technology infections from bad. Ditch that fever. Go with node.”

As Ward describes this presentation, “I’ve tested a half-dozen home sensor integration technologies over as many years and learned something important about architecture with each generation. I’ve replaced Arduino hardware with Teensy which offers much better USB support. I’ve replace C++ with Perl then with Ruby/Sinatra and now Node/Wiki each time feeling the fresh air of a more friendly and dynamic environment. I’ve plotted results with ascii-art, java-2d, flot and now d3.js which can be a career in itself. I’ll share the good parts of each of these and suggest how you will know when it is time for you to move on.”

Nexxy presenting…

Realtime Hardware with Node.js

Nexxy

Nexxy

Key facts about Nexxy:

  • Nexxy has been hacking on things in one way or another SINCE THE BEGINNING OF TIME.
  • Nexxy is also known for her occasional use of hyperbole as a literary device.
  • Nexxy first began hacking arduinos with node while she was working with a vegan strip club named Casa Diablo.
  • Nexxy is now enjoying her work with Ninja Blocks as they take over the world with the internet of things!
  • Nexxy inexplicably decided at the last second to write this section of the proposal in 3rd person, bullet form.

…and I have recreated this proposal section here for your information!

Join Nexxy & all for another installment of “Realtime Hardware with Node.js” as we take a look at just how exactly one should go about making a fool of themselves on stage with a bunch of electronics. We will cover the basics of getting started with hardware, demonstrate some cool tech, and conclude with a super rad (slightly hazy) and interactive dance party of epic (modest) proportions — if everyone cooperates. Questions are welcome throughout the presentation and audience members are encouraged* to participate.

* bribed with stickers

Jerry Seivert presenting…

Know Your Environment

Jerry Sievert

Jerry Sievert

Jerry is a conousier of fine beer, drinks and other assorted things. The relevant list of coarse includes Lego, trains and JavaScript. His own words of these things, being a drink or beautifully coded up application, “I love to build things!” summarizes up Jerry’s love of building perfectly!

Jerry will pose the question, “What if your software knew about its environment and could react?” With very basic electronics skills, and the ability to read a datasheet, you can be well on your way to a smarter and more responsive application. We will discuss some common hardware protocols and how to interface your code with them to build something all “teh aWeSOME“!

Hannah Fousanon presenting…

Optimizing Single-Page Javascript Websites for SEO

Hannah

Hannah

Hannah Fouasnon is a cofounder and lead engineer at DJZ based in San Francisco. While a full time hacker, she’s currently focusing on creating DJZ’s next single-page javascript website optimized around playing media and games. Former projects include Luckysort, a big-data, text-analytics company based in Portland, OR, where she implemented a large portion of the node.js architecture.

Hannah will cover how to create single-page javascript websites and enable developers to more easily support advanced web client features. For example, the persistent playing experience on soundcloud is supported by a single-page backbone.js architecture. One of the downsides when deciding on this architecture is that web crawlers have trouble indexing content, which can hurt seo and facebook sharing.

This presentation covers how we solve this problem at DJZ. The example will be a simplified angular.js application optimized for seo using primarily phantom.js. Viewers can follow along with code on github (link to come).

Are you signed up?  BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

Vagrant for VMware Fusion with plugin issues… Part 2

After the previous blog entry I wrote up, working through getting vagrant to spool up a vmware image I got a few other suggestions via Twitter.

With that quick delete of the hidden vagrant directory I gave it a shot again with the provider flagh.

rm -rf .vagrant/
$ vagrant up --provider=vmware_fusionBringing machine 'default' up with 'vmware_fusion' provider...
[default] VMware requires root privileges to make many of the changes
necessary for Vagrant to control it. In a moment, Vagrant will ask for
your administrator password in order to install a helper that will have
permissions to make these changes. Note that Vagrant itself continues
to run without administrative privileges.
Password:
[default] Box 'bosh-solo-0.6.4' was not found. Fetching box from specified URL for
the provider 'vmware_fusion'. Note that if the URL does not have
a box for this provider, you should interrupt Vagrant now and add
the box yourself. Otherwise Vagrant will attempt to download the
full box prior to discovering this error.
Downloading or copying the box...

Which this seemed to work. Downloading the helpers and such started and I waited patiently.

A Few Thoughts…

Needing to delete a hidden file struck me as one of those completely arbitrary and random solutions. It worked, which is awesome, but it working is a completely counter intuitive solution. I did a ‘destroy’ previously along with a number of things that were somewhat not intuitive. At this point the steps were fine, I had to ask for help, and I got help really fast. That’s awesome, but needing to go through those steps was unfortunate and ties back around to @jeffsussna‘s tweet earlier.

Anyway, as soon as I did this I decided Virtual Box it is. As it went through a 40 minute download of an image (??) it finished and displayed…

$ vagrant up --provider=vmware_fusionBringing machine 'default' up with 'vmware_fusion' provider...
[default] VMware requires root privileges to make many of the changes
necessary for Vagrant to control it. In a moment, Vagrant will ask for
your administrator password in order to install a helper that will have
permissions to make these changes. Note that Vagrant itself continues
to run without administrative privileges.
Password:
[default] Box 'bosh-solo-0.6.4' was not found. Fetching box from specified URL for
the provider 'vmware_fusion'. Note that if the URL does not have
a box for this provider, you should interrupt Vagrant now and add
the box yourself. Otherwise Vagrant will attempt to download the
full box prior to discovering this error.
Downloading or copying the box...
Extracting box...te: 119k/s, Estimated time remaining: --:--:--)
The box you attempted to add doesn't match the provider you specified.

Provider expected: vmware_fusion
Provider of box: virtualbox

… because the image isn’t available for vmware according to Vagrant, so for now, with some solutions and more questions I’m just going to go with the Virtual Box Solution and get back on track with the larger picture blog entry I’m writing. Thanks to @brianmmclain, @mitchelh, @jeffsussna and @thoward37.