History of Symphonize.js – JavaScript Client Pivot to Data Generation Library

…the history of symphonize.js So Far!

NOTE: If you just want to check out the code bits, scroll down to the sub-title #symphonize #hacking. Also important to note I’m putting the library through a fairly big refactor at the moment so that everything aligns with the documentation that I’ve recently created. So many things may not be implemented, but we’re moving toward v0.1.0, which will be a functional implementation of the library available via npm based entirely on the documentation and specs that I outline after the history.

A Short History

I started the symphonize.js project back on the 1st of November. Originally I started the project as a client driver library for Orchestrate.io, but within a day Chris Molozian commented and pointed out that there was already a client driver library for Orchestrate.io available that Steve Kaliski (Github @sjkaliski and Twitter @stevekaliski and http://stevekaliski.com/) had coded called logically orchestrate.js. Since this was available I did a pivot to symphonize.js being a data generation project instead.

The comment that enabled symphonize.js to pivot from client driver to data generation library.
The comment that made me realize symphonize.js should pivot from client driver to data generation library.

The Official Start of Symphonize.js

After that start and quick pivot I posted a blog with Orchestrate.io titled “Test Data Builder Symphonize.js With Chance.js (1/3)” to officially start the project. In that post I covered key value and graph basics, with a dive into using chance.js and orchestrate.js with examples. Near the same time I also posted a related blog on publishing an NPM module, which is the deployment focus of Symphonize.js.

Reasons Reasoning

There are two main reasons why I chose Orchestrate.io and a data generation library as the two things I wanted to combine. The first, is I knew the orchestrate.io team and really dug what they were building. I wanted to work with it and check out how well it would work for my use cases in the future. The ability to go sit down, discuss with them what they were building was great (which I interviewed Matt Heitzenroder @roder that you can watch Orchestrate.io, Stop Dealing With the Database Infrastructure!) The second reason is that my own startup that I’m co-founding with Aaron Gray (@agray) needed to use key value and graph data storage of some type, somewhere. Orchestrate.io looked like a perfect fit. After some research, giving it a go, it fit very well into what we are building.

CRUD, cURL Hacking & Next Steps

Early December I knocked out two support articles about testing APIs with cURL in Some JavaScript API Coding With Restify & Express & Hacking it With cURL …Segment #1 (with some Webstorm to boot) and Some JavaScript API Coding With Restify & Express & Hacking it With cURL …Segment #2 and an article on the Orchestrate.io Blog for part 2 of that series titled Symphonize Some Create, Read, Update & Delete [CRUD] via Orchestrate.js (2/3).

December then rolled into the standard holiday doldrums and slowdowns. So fast forward to January post a few rounds of beer and good tidings and I got the 3rd in the series published titled Getting Serious With Symphony.js – JavaScript TDD/BDD Coding Practices (3/3). The post doesn’t speak too much to symphony.js usage but instead my efforts to use TDD or BDD practices in trying to write the library.

Slowly I made progress in building the library and finally it’s in a mostly releasable state now. I use this library daily in working with the code base for Deconstructed and imagine I’ll use it ongoing for many other projects. I hope others might be able to find uses for it too and maybe even add capabilities or ideas. Just ping me via Twitter @adron or Github @adron, add an issue on Github and I’ll be happy to accept pull requests for new features, code refactoring, add you to the project or whatever else you’re interested in.

#symphonize #hacking

Now for the nitty gritty. If you’re up for using or contributing to the project check out the symphonize.js github pages site first. It’s got all the information to help get you kick started. However, you can keep reading as I’ve included much of the information there along with the examples from the README.md below.

NOTE: As I mentioned at the top of this blog entry, the funcitonal implementation of code isn’t available via npm just yet, myself and some others are ripping through a good refactor to align the implementation fo the library with the rewritten and newly available documentation – included blow and at the github pages.

How to use this project in one of your projects.

[sourcecode language=”bash”]
npm install symphonize
[/sourcecode]

How to setup this project for development.

First fork the repository located at https://github.com/Adron/symphonize.

[sourcecode language=”javascript”]
git clone git@github.com:YourUserName/symphonize.git
cd symphonize
npm install
[/sourcecode]

Using The Library

The intended usage is to invocate the JavaScript object and then call generate. That’s it, a super simple process. The code would look like this:

[sourcecode language=”javascript”]var Symphonize = require(‘../bin/symphonize’);
var symphonize = new Symphonize();
[/sourcecode]

The basic constructor invocation like this utilizes the generate.json file to generate data from. To inject the json configuration programmatically just inject the json configuration information via the constructor.

[sourcecode language=”javascript”]
var configJson = {"schema":"keyvalue"};

var Symphonize = require(‘../bin/symphonize’);
var symphonize = new Symphonize();
[/sourcecode]

Once the Symphonize data generator has been created call the generate() method as shown.

[sourcecode language=”javascript”]
symphonize.generate();
[/sourcecode]

That’s basically it. But you say, it’s supposed to do X, Y or Z. Well that’s where the json configuration data comes into play. In the configuration data you can set the data fields and what they’ll generate, what type of data will be generated, the specific schema, how many records to create and more.

generate.json

The library comes with the generate.json file already setup with a working example. Currently the generation file looks like this:

[sourcecode language=”javascript”]
{
"schema": "keyvalue", /* keyvalue, graph, event, geo */
"count": 20, /* X values to generate. */
"write_source": "console", /* console, orchestrateio and whatever other data sources that might come up. */
"fields": {
/* generates a random name. */
"fieldName": "name",
/* generates a random dice roll of a d20. */
"fieldTwo": "d20",
/* A single lorum ipsum random statement is genereated. */
"fieldSentence": "sentence",
/* A random guid is generated. */
"fieldGuid": "guid" }
}
[/sourcecode]

Configuration File Definitions

Each of the configuration options that are available have a default in the configuration file. The default is listed in italics with each definition of the configuration option listed below.

  • “schema” : This is used to select what type of data structure type is going to be generated. The default iskeyvalue for this option.
  • “count” : This provides the total records that are to be generated by the library. The default is 1 for this option.
  • “write_source” : This provides the location to output the generated data to. The default is console for this option.
  • “fields” : This is a JSON field within the JSON configuration file that provides configuration options around the fields, number of fields and their respective data to generate. The default is one field, with a default data type of guid. Each of the respective entries in this JSON option is a self contained JSON name and value pair. This then looks simply like this (which is also shown above in part):[sourcecode language=”javascript”]{
    "someBoolean": "boolean",
    "someChar": "character",
    "aFloat": "float",
    "GetAnInt": "integer",
    "fieldTwo": "d20",
    "diceRollD10": "d10",
    "_string": {
    "fieldName": "NameOfFieldForString",
    "length": 5,
    "pool": "abcdefgh"
    },
    "_sentence": {
    "fieldName": "NameOfFiledOfSentences",
    "sentence": "5"
    },
    "fieldGuid": "guid"
    }
    [/sourcecode]
  • Fields Configuration: For each of the fields you can either set the field to a particular data type or leave it empty. If the field name and value pair is left empty then the field defaults to guid. The types of data to generate for fields are listed below. These listed are all simple field and data generation types. More complex nested generation types are listed below under Complex Field Configuration below the simple section.
    • “boolean“: This generates a boolean value of true or false.
    • “character“: This generates a single character, such as ‘1’, ‘g’ or ‘N’.
    • “float“: This generates a float value, similar to something like -211920142886.5024.
    • “integer“: This generates an integer value, similar to something like 1, 14 or 24032.
    • “d4“: This generates a random integer value based on a dice roll of one four sided dice. The integer range being 1-10.
    • “d6“: This generates a random integer value based on a dice roll of one six sided dice. The integer range being 1-10.
    • “d8“: This generates a random integer value based on a dice roll of one eight sided dice. The integer range being 1-10.
    • “d10“: This generates a random integer value based on a dice roll of one ten sided dice. The integer range being 1-10.
    • “d12“: This generates a random integer value based on a dice roll of one twelve sided dice. The integer range being 1-10.
    • “d20“: This generates a random integer value based on a dice roll of one twenty sided dice. The integer range being 1-20.
    • “d30“: This generates a random integer value based on a dice roll of one thirty sided dice. The integer range being 1-10.
    • “d100“: This generates a random integer value based on a dice roll of one hundred sided dice. The integer range being 1-10.
    • “guid“: This generates a random globally unique identifier. This value would be similar to ‘F0D8368D-85E2-54FB-73C4-2D60374295E3’, ‘e0aa6c0d-0af3-485d-b31a-21db00922517’ or ‘1627f683-efeb-4db8-8174-a5f2e3378c87’.
  • Complex Field Configuration: Some fields require more complex configuration for data generation, simply because the data needs some baseline of what the range or length of the values need to be. The following list details each of these. It is also important to note that these complex field configurations do not have defaults, each value must be set in the JSON configuration or an error will be thrown detailing that a complex field type wasn’t designated. Each of these complex field types is a JSON name and value parameter. The name is the passed in data type with a preceding underscore ‘_’ to generate with the value having the configuration parameters for that particular data type.
    • “_string“: This generates string data based on a length and pool parameters. Required fields for this include fieldName, length and pool. The JSON would look like this:[sourcecode language=”javascript”]"_string": {
      "fieldName": "NameOfFieldForString",
      "length": 5,
      "pool": "abcdefgh"
      }
      [/sourcecode]

      Samples of the result would look like this for the field; ‘abdef’, ‘hgcde’ or ‘ahdfg’.

    • “_hash“: This generates a hash based on the length and upper parameters. Required fields for this included fieldName, length and upper. The JSON would look like this:[sourcecode language=”javascript”]"_hash": {
      "fieldName": "HashFieldName",
      "length": 25,
      "casing": ‘upper’
      }
      [/sourcecode]

      Samples of the result would look like this for the field: ‘e5162f27da96ed8e1ae51def1ba643b91d2581d8’ or ‘3F2EB3FB85D88984C1EC4F46A3DBE740B5E0E56E’.

    • “_name”: This generates a name based on the middle, *middleinitial* and prefix parameters. Required fields for this included fieldName, middle, middle_initial and prefix. The JSON would look like this:[sourcecode language=”javascript”]"_name": {
      "fieldName": "nameFieldName",
      "middle": true,
      "middle_initial": true,
      "prefix": true
      }
      [/sourcecode]

      Samples of the result would look like this for the field: ‘Dafi Vatemi’, ‘Nelgatwu Powuku Heup’, ‘Ezme I Iza’, ‘Doctor Suosat Am’, ‘Mrs. Suosat Am’ or ‘Mr. Suosat Am’.

So that covers the kick start of how eventually you’ll be able to setup, use and generate data. Until then, jump into the project and give us a hand.

After this, more examples on the way, cheers!

Mapping Domain Names with name.com, Elastic Beanstalk, Elastic Load Balancer and AWS Route 53

I finally wrapped up my name server and DNS mapping needs with Name.com, Route 53 and Elastic Beanstalk. Since this was a little confusing I thought a short write up was in order. Thanks to Evan @evandbrown for helping out!

The first thing needed is a delegation set of name servers for your DNS and name server provider. These can be found by creating a hosted zone. The way to do this is open up the AWS Management Console and navigate into the Route 53 management area. The Route 53 icon is under the Compute & Networking section on the management console.

Beanstalk, Route 53 - Click for full size image
Beanstalk, Route 53 – Click for full size image

Upon navigating to the Route 53 console area click on the Create Hosted Zones button.

Create Hosted Zone
Create Hosted Zone – Click for full size image

When the zone is created then the delegation set can be found under the Hosted Zone Details. This delegation set now needs setup as the name servers for whoever, in this case name.com, is the domain provider.

Delegation Set - Click for full size image.
Delegation Set – Click for full size image.

Open up the management console for the name server administration.

…

Upon adding them the list should look something like this.

Name servers list built from the delegation set of the hosted zone. Click for full size image.
Name servers list built from the delegation set of the hosted zone. Click for full size image.

Once the name servers are setup, those will need time to propagate. Likely this could take a good solid chunk of time, somewhere in the hours range likely, and don’t be surprised if it takes a little bit more than a day.

While the propagation starts navigate back to the AWS Management Console and open up the EC2 section of the console. On the right hand side of the Resources list there is a Load Balancers section. Click it.

Load Balancers - Click for full size image.
Load Balancers – Click for full size image.

In this section there is a listing of all load balancers that have been created manually or by Elastic Beanstalk.

Load Balancers - Click for full size image.
Load Balancers – Click for full size image.

Make note of the Load Balancer Name for selection in Route 53. This is what Route 53 needs in order to point an alias at for incoming traffic to that particular Elastic Beanstalk application. In this particular image above there are 4 load balancers listed, the easiest way to prevent confusion is to take note of the load balancer name at the time of creation, but this is the easiest way to find them otherwise.

Record Set - Click for full size image
Record Set – Click for full size image

Now when going back to the hosted zone to set it up with the appropriate information, create a new record with the appropriate name, in this case I was setting up the admin.deconstructed.io (no it isn’t live yet, I just set it up to test it out) to point to an alias target. Just leave the Type set to A – IPv4 address and click the radio control so that Alias is set to Yes. In the alias target select the appropriate load balancer for the Elastic Beanstalk (or whatever it points to) application.

That’s it, give it a few hours (or a day) and eventually the domain or subdomain will be pointed appropriately at the Elastic Beanstalk load balanced application.

9 Ways To Survive The Shit Storm of Developer Evangelism

I started to write a blog entry a few months ago about my time doing developer evangelism. First in practice, along with product management and team leadership and then as a full time developer evangelist with Basho. Then I felt many different things, nothing which translated into a very useful blog entry. Well past any motivation to write up where and what I was doing at the time and why I decided it wasn’t something I wanted to keep pursuing, I ran into this blog entry titled “Developer Evangelism The Whole Story“. At that point I thought, “alright, I’m going to add my two cents after all”.

For one of the same reasons Keith Casey wrote his entry. People have asked me numerous times about becoming an evangelist or advocate. Be sure to read Casey’s write up, and here’s mine to throw more into that fire.

Positives:

  • You’ll be able to go to all sorts of cities and meet a whole bunch of different people.
  • You’ll be on display and actually able to do something to improve the industry. Not just technologically but to help resolve sexism, discrimination and other issues and treat people well.
  • Do right by people as an evangelist and you’re set for a plethora of possibilities when you finally get burned out.
  • You get to play with all sorts of tech.
  • You get to travel a lot, which makes you really start to respect your home base, wherever that may be.

Negatives:

  • You’re barely ever home, usually you’re on the road with familiarity often becoming the stink of a plane or the confused expression as the TSA security circus actually recognizes you and just starts ignoring you.
  • Even though you can help improve the industry, you’re ability to make a home, make a difference where you live is dramatically reduced to basically nothing. For most people, considering civic involvement in the United States, this probably doesn’t even matter. For some, it’s destructively depressing.
  • If you get mis-portrayed, say something dumb out of jest, or the media mis-quotes you it can be anywhere from annoying to career limiting or ending. If you make the mistake of pissing of someone that has a lot of pull then it could also be super destructive – even if that person is a total jack ass and everyone routinely knows it and admits to it.
  • You get to play with all sorts of tech, but you lose a lot of credibility because you don’t actually build anything real anymore. This is a huge problem, and I’d even suggest most evangelists go work on an actual dev team every other year or so. It doesn’t matter who you are, you will start to be perceived as a shill of some sort by a reasonable amount of people, even though they could be extremely wrong in that perception.
  • Your home base, you often don’t get to have a real home base. You are a vagabond. For a musical definition, listen to Metallica’s “Wherever I May Roam”.

Now if you still think this is a great gig for you. Thicken up that skin, get some callouses and get ready for a bad ass trip that’ll teach you about all sorts of human interactions and more. But be prepared and keep a solid look out for burn out and the degradation of any of the situations mentioned above. If you do you’ll likely do well. If you’re still interested, here’s a few things to get your kick start in developer evangelism:

  1. Get a social media presence, get it fast, and get a nick that you can use in almost all contexts. Don’t even pretend you can skip this step. The most successful evangelists have a huge social media presence and manage it. They manage it hard core, work it into a system, and learn efficient and positive ways to interact with that social media presence. Shut up, don’t even try to skip it, just go out there and manage it.
  2. Make sure to spend at least an hour a day doing something technical. Hacking on Docker, writing some scripts or heaven forbid writing some actual code. This is massively important because you’ll find yourself losing direction all the time from the task switching and not getting to do these little technical things that will help you keep your edge.
  3. Learn to speak. I don’t mean read a little book and think you know how to speak in front of a crowd. Likely, you really suck at it. I’m talking about practicing in the mirror, talk to yourself, record yourself and watch it and do all of these things without becoming nihilistic or pompous. Most of us tech speakers are so bad we’re lucky that the people in the industry are actually focused on the tech and not our stuttering horror of speaking abilities.
  4. Drop all fear to speak with people in positions of power. Remember, everybody is human, don’t get intimidated and don’t intimidate.
  5. Not that anybody in the software industry or tech industry or any industry needs told this but I’ll say it. Don’t overdo the drink. We’re all dangerously close all the time to being worthless drunkards. Some of us stay pretty functional on a drink or two, but that only lasts for a short time before you do indeed go downhill. Don’t deny yourself, you are NOT part of the one percent that can stay sharp and rot your brain. So keep the drink in check.
  6. Find a way, anyway, to stay physically healthy. If you don’t the travel can very likely kill you. I don’t mean like “I’m tired and want to go to bed” killed but more like “hmmm, Tory Joe McQuerty here sure did see like they were fine, too bad we’re putting them six feet under” killed. Oh, the “I’m tired and want to go to bed” will happen all the time too, just make sure you keep that as the only killed you get.
  7. Attain a huge amount of apathy for the extra overdose of everyone’s opinions about how everything sucks in the world. Many programmers are notoriously negative, especially if they work in the enterprise. It’s part of the daily war story if you get sucked in. Remember to stay focused on what’s important, your health and your loved ones, the job comes second. Anybody that tells you different, put them in that apathy category.
  8. Never feel like you have to explain yourself when you need to take some family time or personal time. Just say you need to and do it. Even if you’re pretty close to people on your team, they need to respect that and let you get some time in. This is extremely important.
  9. Don’t give to many fucks. Learn that at some point you gotta call it a day and turn in. Just drop it all and get a good night of sleep.

Summary: Think really hard about what you want when signing up for a dev evangelist or advocacy gig. It will wreck hell on your life, but it could be immensely rewarding too. But please, if you go into evangelism, practice at it and be prepared. I hate the idea of seeing more people burn themselves right out of the industry.

If you have anymore survival suggestions, please do comment!

Docker Course, Ubuntu, WordPress, Angular.js, Notes, Rich Hickey, Datomic…

Updates, updates, updates…

Docker Course @ Pluralsight

I added a new course on Docker to my Pluralsight list of courses today. This joins my one other course on Riak, which I’m aiming to have more added to that list in the future! Check those out and let me know what you think, how I could improve, what I did right and what you learned (or already knew). I’d greatly appreciate it!

Rich Hickey, Datomic, Clojure, Angular.js and Notes

I started a section on the blog here for notes on topics I’m studying. The first two I’ve hit on are Angular.js and Rich Hickey, Clojure and Hammock Driven Development. I’ll be adding to these over time and will likely report whenever I add good chunks of info or helpful tutorials, how-to docs or just whatever I deem worth mentioning. Simply put I won’t broadcast it much, unless I add some real goodies that are worth it.  😉

Ubuntu & WordPress

I needed a kind of WordPress Workstation to hack around testing some WordPress so I put together quick notes on the fastest and cleanest way to setup a WordPress VM from scratch.

Until later, happy coding, have a metal \m/ \m/ Friday!

Sexism, Racism and t-shirts… Why Stop There?

I attend a lot of technical conferences, hackathons, workshops and all sorts of events where I get loaded up with t-shirts. Ridiculous amounts of t-shirts. Generally I love this! I have had a basic standard about all of these t-shirts. If it is a company and product that I know has done well by its users, by the community and the people involved have tried to do good by what they’re advocating, I’ll wear the t-shirt. Thus, I have a lot of tech t-shirts to wear.

However…

American Apparel is one of the top suppliers of t-shirts for tech companies to print their stuff on and bring to trade shows, hackathons or wherever they’re going. American Apparel definitely has some positive attributes, here’s a list:

  • the products for the USA are made in the USA.
  • the workers make almost 2x as much as other factory workers in the surrounding areas (Los Angeles)
  • paid time off
  • sick leave
  • health care
  • company-subsidized lunches
  • bus passes
  • free English as an additional language classes
  • on-site massage therapists
  • free bicycles
  • on-site bike mechanics
  • free parking
  • proper lighting and ventilation

Now some of these things, many, to the tech industry seem like a joke. But these are very serious benefits for labor. In general, American Apparel does good by it’s employees. At least, in the United States.

The Problems

In China they do not hold these same standards, they hold the standards of labor practices in China for Chinese Factories. These are, dramatically lower than standards in factories in the United States. The conditions there are harsh, but there is no shortage of labor and it is considered an honor to get work. It’s a perverse irony, but it exists.

Then there is the problem that American Apparel has with sexism. There ads have obvious double standards in so many places. In England they’ve actually managed to get some ads banned. That’s an accomplishment since the English and Europeans are often far more liberal about things than US Citizens are. Namely, we as a population in the US are prudes by comparison, so if it got banned in England, holy moly I’m sure it would have raised absolute hell in the United States.

So on Twitter Red Hat evangelist @TheSteve0 posted this tweet.

Pointing to an article on the double standards of American Apparel. I’m all about stepping back and buying product, that has all the above positives in the company and has higher integrity in marketing than American Apparel, but what are the choices? That’s where another tweet a few minute later rallying several other fellow technical evangelists to stop using American Apparel as the top t-shirt.

More Thoughts

This leads me to another thought too, why hand out throwaway t-shirts in the first place? Why hand out anything that’s throw away? Why not hold ourselves to even higher standards? Why not only give away things that people will keep and really use. I can’t imagine how many t-shirts people take and then throw away. We as an industry should absolutely do better.

Some of the things that @thoward37 and I often do is haul excess cloths down to the local Goodwill or Homeless Shelter. Notice how cold it is right now in parts of the country? People are dying (nope, not exaggerating) right now from the cold. Because they don’t have blankets, shelter space and other simple amenities. The least we as an industry could do is provide our excess to those most in need so they can at least stay alive!

Solutions (These are NSFW)

  • Organize efforts at conferences to call-out companies that are being sexist, racist or otherwise discriminatory. Do it was a conference, as a group, as a whole and make it count. Our industry actually gives a shit and we can make things happen. Don’t pretend we can’t.
  • Pull your head out of your ass if you think that buying t-shirts from some sweat shop from X company is a good idea. Do better, if your company doesn’t have the money to buy good product, the DON’T BUY PRODUCT.
  • When you provide swag or whatever to conference goers, treat the conference attendees with respect and actually get them something they can use, that they’ll be happy to say, “yeah, I got this at X conference, it so rocked in so many ways!”
  • Think outside the box, do we really need another thousand t-shirts floating around? If we do have them floating around are you really going to be able to give them away? To people that will wear them? Just think about these things a little. Every single technical evangelist I know is smart enough this takes about 2 seconds. So just spend that 2 seconds and help out.
  • Also, there are more and more women getting involved in tech, in spite of the trolls that waste so much of all our time. Think about this when you’re ordering products, don’t just get a bunch of giant t-shirts. Sure, get some, but realize the community is quickly diversifying and many if not most of the community that is stepping up to take leadership of the tech industry is not going to be one size fits all anymore. (thank goodness)
  • Last one is simple. Just stop and think about what would matter to people. What do you want to give to others that would actually matter. I mean, it really boils down to basic GOOD, high integrity marketing. Don’t treat people like shit, treat them like they’re important. They’re prospective customers, they pay money and put food on your table. So just simply, easily, give a shit.  🙂

On this same topic, I’ll have another post soon about what you’d like to see at Node PDX for swag. What in general would you find to be excellent swag to receive at a conference that you’d be proud to lug home and wear, use or otherwise not throw it away? Until then, please comment and we’ll get some round table conversation going.

Cheers & happy hacking!