A Story of an Ivory Tower Architect Clusterfuck

Astronaut Architect or Ivory Tower Architect – Both of these are generally terms of dirision toward software architects who have become disconnected from anything related to actual implementation, coding, or related knowledge and work that has to do with the work of building software applications and services.

Astronaut Architecturalism or Ivory Tower Architecturalism – Topics that an architect might bring up which could be a sign they have reached the Ivory Tower or Astronaut level of architecture work with software. This ism is something to be very concerned about, as sometimes these individuals can completely derail and wreck a project with lofty – yet extremely disconnected ideas – about how or in what way the software should be built for a project.

Chief Emotive Product Technology Architect

Circa 2004 – In the city of Portland, Oregon.

The team, a reasonably sized startup team, of ~12 people sat around the table. It was a oval shaped table, specifically in this conference room so that all members could have involvement while seated at the table, in a conference room called “King Arthur’s Round Table”. Oh, the “LOLz”. The 12 individuals had been working on a project that was a B-round funded venture that had to do with musical things. These 12 individuals had so far, self-organized and been working at a reasonable clip, to get this software built and realize the ideas behind the venture.

The 12 had been gathered here by their manager so that a new member of the team could be introduced. Nobody knew who this person was or what the role of this person would be, except of course the hiring manager. Because I suppose, that was how things were going to be done in this year of 2004 with this startup with this manager. Yes, all sorts of indicators were already being fired off for the experienced of the 12. As these indicators fired off in their mind, concerns were growing, but the experienced had kept their mouths shut so far and offered the manage the benefit of the doubt.

The manager, who I’ll call Thomasen stood before everyone, in conflict with the ideals of Arthur’s Round Table. Thomasen mentioned the ideals of Arthur’s Round Table, which seemed ironic since he stood there before – removed from equal voice by the mere act of standing and spoke to everyone. But also it seemed fitting, for he then announced that he had hired a new member for the team. This unilateral decision of Thomasen’s seemed as if done in spite of the ideals of the Round Table, in spite of his just stating the ideals of the Round Table. What kind of fresh hell was being birthed at this moment? Well, that’s were things get lit, if not outright explosive!

Ericson, one of the experienced developer’s asked, “Ok, you just made a unilateral decision to hire someone, but what even is this person’s role? What are they gonna do here?” Thomasen replied, “I’ve hired Jacobson Tirelorn to join us as the Chief Emotive Product Technology Architect!” To which, the metal head of the team whipped to attention in his seat and inquired, a bit forcefully as a metal head might be imagined to do, “What the Holy Hell fuck is a Chief Emotive Product Technology Architect?”

Thomasen replied, “Kirk, jeez man you don’t have to be so aggresive and vulgar, could you tone it back about 10 notches?” Kirk, not one to take bullshit at even volume level 1, let alone at 11, responded kindly with a simple statement, “Nope.” Thomasen breathed heavily, the sigh clearly evident to all in the room. The newer folks of the 12 watched this with apprehension, while those that were familiar with Kirk’s lack of accepting bullshit, smirked as he laid out the guantlet for Thomasen. Thomasen responded finally after this long sigh, “Ok Kirk.”

Thomasen continued, “I’ve hired Jacobson Tirelorn in this role to help use build our product and really get connected on an emotional level to our users. Kirk, again sniped in, “We don’t have users yet we’re a startup in stealth mode.” Thomasen growing frustrated, “I know but we will.” Kirk, “Sure. But what exactly is the emotional level of our users?” Grunting almost, “I’m getting to that point Kirk.”

Thomasen elaborated, “Jacobson, the architect will also help us significantly figure out what technology we’ll use for the product and how all the parts should fit appropriately to make a highly scalable solution for our users!”

Lasinia raised her hand, “Can I ask a question?” to which Thomasen responded calmly, “Sure Lasinia, what’s your question?” Lasinia started off, “Per what you said, I decided to raise my hand since I’m confused about the Round Table ideals, if we all have an equal seat why are you standing and why do I have to raise my hand to speak?” Another of the 12 responded, “you don’t have to raise your hand…” “yeah but I felt like it because we’re not really following the ideals very well if Thomasen is standing up, Thomasen, could you sit down and continue telling us about this architect you hired?”

Thomasen pulled up a chair and sat down. Muttering somewhat under his breath, but still clear to the 12, “God dammit everybody.” He sat and continued finally.

Eventually he ended. The era of theh Chief Emotive Product Technology Architect was upon the team, regardless of what that meant!

The Chief Emotive Product Technology Architect Era Begins

Two weeks after the first Round Table, Jaconson arrived and called a meeting for Wednesday. It being Monday everybody easily accepted the meeting for 8am Wednesday. On Tuesday morning Jacobson sent out a document detailing all sorts of great and lofty goals for his architect role. He was, after all exuberently ready to get things started! Maximum emotional user support and all!

Around mid-day Tuesday, after the document with the details had arrived that morning Jacobson sent out another calendar invite for 9am Wednesday. Again, everybody went along with this and now had an 8am and 9am meeting scheduled for Wednesday! However, the foreshadowing got amped up another level when Jacobson is his zest and zeal sent out another meeting for 1pm that Wednesday. This meeting invite, however, was sent out at 4:51pm on Tuesday, which would have placed the meetings at 8am, 9am, and 1pm. The scheduling, and titles, went something like this.

8am – Software Release Schedule
9am – Emotional User Stories
1pm – Design Studies

This was going to be one helluva kick off of the Chief Emotive Product Technology Architect Era!

The day rolled around. Having sent out that 1pm invite at 4:51pm multiple people had not received the invite in time to know about the meeting before the 8am meeting kicked off. Thus, this is when the horror of the era truly began.

8am everybody stood at the door to the conference room, again King Arthur’s Round Table conference room. The time ticked past 8am. This seemed normal, it was after all Portland and rarely are people precisely on time on the west coast. 8:03am ticked by. Still, none of the 12 were concerned yet as Thomasen arrived to open the door. Again, this whole irony here, being the door to the Round Table conference room was locked, as if someone was going to sneak in and steal the massive multi-hundred pound table?

Thomasen and the 12 all sat down at the table. Being 2004, Thomasen was the only one with a laptop, another important piece of context here. In 2004 most people still programmed at desktop machines, thus, barely any of the 12 knew of the 1pm meeting, but were still ready to get things started and give this thing a shot!

8:31am arrived and Jacobson walks in as if no tardy one moment. He then pulls out a chair and sits down at the table and announces, “Hello everybody, it’s great to be here I’m Jacobson Tirelorn and I’m going to help you get this software solution, your processes, and your emotional well being figured out!” Kirk, yes Kirk again, spearheads the next immediate assertion, “Confirming, you’re going to help with the software solution architecture, the processes for the project, and our emotional well being? Why would I want you to help me with my emotional well being? Is there an assumption my emotional well being is fucked up?”

Now I need to paint some context here. Jacobson had entered the room casually, as if not late, but also wearing the garb of a 1970s era hippy. Multicolored attire and jeans, which wasn’t to crazy for a startup in Portland, but was still slightly dated and odd for this era. To add to this off-date situation, Jacobson looked like he was approximately 18 or so years old.

Thomasen chimed in, “Kirk, we’ll talk about your emotional well being later, let’s go ahead and let Jacobson get into the architecture and planning first.” Kirk “alright.” shrugs.

Jacobson starts in on some wording, “Alright, what we’re building is going to need to scale, massively, at a moments notice and currently we’ve only got the capacity for about 20% of this capability. So we need to really amp up our systems and our architecture to handle 150% of our expected capacity! With that in mind I’ve drawn this architecture diagram to help us get there.”

He then commences to put this on display, after messing with a projetor for approximately 5 minutes. What he then shows looks like a soup bowl was spilled out onto paper, which bright colors and odd shapes representing the Sun Servers, which were drawn not with the icons of the era that represented a Sun Server but with actual suns showing bright yellow. The network connections and backplane were shown with fuzzy ropes and other tangled bushes. It almost looked more like a wildlife sketch about the magic of the birds and bees than anything to do with software.

Jacobson then made a statement that immediately shaped the project, “What I’ve done here is use a little artistic license to draw up the server and network diagram that will get us to the needed scale!”

All of the 12, in their minds, and with their coursing eyes looking amongst each other thought in horror and disbelief. The all imagined that an oil train had just derailed and fell from a cliff into the ocean. As Jacobson continued the idea spread and so did the oil spill idea, as if it were lit on fire to burn uncontrollably.

Jacobson continued. He talked for the raminder of the meeting and then stated, “everybody take a bio break now and grab a snack for the next meeting, we’ll all meet back here at 9:05am to get started on the emotional user studies.”

At 9:05 everyone except Jacobson and Kirk entered the room and sat down. Nobody spoke, but just sat and waited patiently, some snacking on food and others just pondering what was going on. At 9:11 Jacobson walks back in and shows another chart that resembled Maslav’s Heirarchy. He then asks, “Where is Kirk?”

Kirk walks back in at 9:12, just a mere minute later and Jacobson says, “Good you’re here, a little late but we can get started now.” Kirk states, “you walked back in here a minute ago, it’s 9:12 now. You were late, I’m just following your lead.” Jacobson seems to not even acknowledge that Kirk has spoken and being a spiel about emotional well being.

Just a few minutes into this Sarah, one of the 12, new to the team asks, “What are the key aspects of everybody’s well being should we primarily be focused on?” “We should be focused 100% on the zen of people’s well being.”

Kirk stands up. Walks toward the door and leaves. Thomasen responds, “go ahead and continue Jacobson I’ll see what the problem is.” Jacobson continues.

Meanwhile Thomasen and Kirk have a chat, Kirk states simply that this guy is bullshit and you can keep me and I’m just going to work on the project or you can toss him, but I’m not going to work with this guy while he spouts this nonsense. Kirk having been key so far the lead of the development teams and practically a founder, leaves Thomasen to agree to the terms. Kirk informs him he’ll give Jacobson a chance but isn’t going to listen to the emotional zen nonsense, so he’ll be in other meetings, and Thomasen seems relieved, and life continues for the project.

Thomasen rejoins the meeting, and the meeting runs on and on and on and on. At 12:07 Jacobson says, “everybody take lunch now and we’ll get back to things later! thanks all!” To which everybody, releived, heads off to lunch. But do note, not a single person of the 12, except for Kirk, has been back to their desks to work or read emails on their desktops. Not a soul, except Kirk, knows now that there is a 1pm meeting. Not even Thomasen.

1pm rolls around and Kirk walks into the conference room ready for design details. Jacobson enters and immediately states, “well I guess since you’re the lead nobody else really needs to join us.” But Kirk knowing the anti-pattern in that states, “Well, others should join us as everybody needs to know the design of the system that is going to be working on the design of the system.” “Well, you Kirk can tell them later right?” Kirk smiles wryly and laughing says, “Alright, this is turning into a train wreck already so yeah, let’s go with that idea.”

Jacobson informs Kirk of his ideas.

Kirk finishs the 1pm meeting with the conclusion that Jacobson is an Ivory Tower Architect of no use to the team that needs to implement the product.

Three months pass. A status meeting is called by the CEO of the company. It starts off plainly.

“So how are things going team? I’d like to get a preview look at the product before our coming release in two weeks.”

Kirk looks up and asks, “In two weeks?”

Jacobson responds, “Yeah, we’re releasing in two weeks.”

Kirk and Sarah both look toward each other and simultaneously ask, “We’re releasing what exactly?”

Jacobson starts to respond and says, “It’ll be version one of the prod…”

Thomasen cuts in, “Wait a second, just to clarify we’re releasing a beta of the version 1 of the product.”

Sarah, “So like, an alpha product?”

Jacobson, “You could say that, but it’ll be more like a beta, using the design patterns we’ve implemented and architecture I’ve designed.”

Larry, one of the 12, softspoke and rarely speaks unless specifically asked a question, “What architecture and design patterns?”

Kirk says, “The ones Jacobson dreamed up, but don’t worry, it’s those that I’ve shown you but we don’t really call them design pattersn. We’ve just been calling it our architecture.”

Larry, almost assured “Oh, that, ok. But, per Sarah’s question, what are we releasing exactly?”

Jacobson says, “The v1 product.”

Thomasen corrects, “The beta v1 product.”

The CEO asks, “What about the alpha product?”

Thomasen “There is no alpha product.”

Jacobson emotes “Well, we could just rename this the alpha v1 product.”

CEO stands, causing consternation among everybody, “What do you mean rename, we don’t have an alpha product? How can we not have an alpha product and we’re about to release a beta product? Why do you keep skiping the beta part and saying v1 and Thomasen keeps interjecting beta? Who the fuck is in charge of this?”

“I am, and Jacobson is building the architecture design.”

“No he’s not” interupts Kirk, “I’m building the design based on Jacobson’s architecture.”

As you can imagine, none of this is getting resolved quickly, so let’s fast forward to the results of this CEO scheduled meeting.

Post Trash Fire Meeting SITREP

The SITREP, or situational report, went something like this. The meeting went on for another twenty plus minutes of the CEO, Jacobson, and Thomasen being confused about alpha, beta, and v1 variations and Kirk eventually sat back and just let it happen unabated. Sarah got up about 15 minutes into the meeting and nobody noticed she left. In addition to leaving the meeting though she went to her desk, got her personal things and left never to return again.

The others among the 12 listened and eventually faded out and started ignoring the clusterfuck that unfolded before them that day. Two others besides Sarah left within the next three days. A week after that meeting, Kirk decided he was done too and resigned. Jacobson, being the root of many of these problems was fired by Thomasen, which then the CEO in a fit of rage, merely 2 weeks after this meeting as the startup fell apart, fired Thomasen.

4 Weeks after that meeting the CEO was then fired, even though a founder, and the board having all the power to do this started the process of rehiring everybody to fill the roles of those that had left. First they managed to get Kirk to come in for a conversation about what exactly went wrong.

Board inquired “So Kirk, thanks for coming to speak with us. Could you tell us about what exactly happened? We’re not really sure ourselves.”

Kirk happily, ya know happily for a metal head, smiled and simple said, “Not really, I know, but it’s not worth going into.” and then turned just a bit and opened up a laptop he’d bought. “However, here’s some design and ideas about what should be built to acheive what you want and how much it’ll cost to have me come in and get it done.” The board started looking at the architecture and then asked, “Could you elaborate on what all this architecture shows us?”

Kirk went on to detail, in depth the technical challenges and what the design would require to meet or exceed the needs of the prospective userbase. Then, the board flipped the page after they had felt they understood enough. On this next page of details Kirk had layed out a SOW, or Statement of Work, to detail what he’d cost and what he would do to make this happen. The board was agast at Kirk’s hefty 2004 hourly price tag of $100 per hour. They stated they would have to think about it.

A week later they decided, after talking to Jacobson, to hire Jacobson at $80 bucks an hour to lead the effort around the architecture that Kirk had shown them. At least, to the best of their memory.

The project kicked back off again, now many weeks behind. At this point you might know what happened next, Jacobson cratered. He left a blast radius of unhired roles, unfinished design that didn’t make sense, and a massively unfinished project. This took about 4 months before the board wised up to their poor hiring decision, and having hired this astronaut architect stuck in his lofty ivory tower, they then opted to fold and reposition the company, eating the entirity of the losses.

TLDR Summary

2004 was a whopper of a year, considering the economy hadn’t even improved much after the 2000 apacolyptic tech crash! It was a time when many startups had survived with just enough of their hubris and naivety intact that they were trying some pretty crazy things trying to get products and services delivered. In spite of that, many continued failing.

At this point I want to add an important caveat, that the names are changed but this is the recollection of events from multiple people involved in this particular startup. This wildly happened more than a few times during this period when many startups went under. However, many others started to rise from the ashes during this time too.

Moral of the story, beware the astronaut or ivory tower architects!

Speaking Trips, Tech, and Treks

Original Content Post: http://blog.adron.me/articles/speaking-trips-tech-and-treks/

I’ll be off on some coding adventures in the coming months. I also hope to catch up with a lot of people and their respective projects, learn a few things, and if it’s possible maybe teach a few people a thing or three about immutable infrastructure, lessons I’ve learned, and how to avoid infrastructure catastrophes while building the next bad ass awesome application. This blog entry is about the details of my logistics, and I’ll follow up with more details of the coding adventure along the way. For now: details, details, and details.


takes place in London October 19–21, 2016 and San Francisco November 14–16, 2016

Moving Enterprise Practices and Development to Open Source

in London

In this talk I’ll tell the story of our efforts at Home Depot, and provide bullet points and suggestions for helping you to acheive the move to open source practices in your enterprise. The benefits are huge, and overall it’s a lot more fun to boot!

Key Points:

  • Steps we took to get into the open.
  • Practices to take up and start using.
  • Things to avoid when moving to open source models.
  • How to make the most of things for the community and the enterprise.
  • Tooling, interoperability, services, and how to get it done.

Building Immutably to Continuous Delivery with Minimal Inputs

in London and in San Francisco

Official Talk URI:

My Talk URI:

Prerequisite knowledge

Basic understanding of web applications, architecture & design, and basic knowledge of windows & linux server systems.

Materials or downloads needed in advance

Github Account, AWS (or GCE, Azure, etc) Account, and CodeShip Account. Also a computer with OS-X, Windows, or Linux loaded with Node.js is also required.

This workshop focuses on building a continuously delivered pipeline using Node.js (however easily transferable to Ruby/Rails/Java/Scala/.NET etc.). The workshop will trace the steps from inception to deployed application (with a domain pointed appropriately and all) that can then be developed against to continue whatever effort and intent of the developer(s)!

Key Points:

  • The parts: Application, Domain Name, Infrastructure, Integration, & Delivery
  • Build an Application: Steps for building & actually building a simple Node.js Application to deploy.
  • Getting a domain name, determining name servers & DNS servers, setting it up and getting it pointed at our application.
  • Setting up and determining the deployment scenario on AWS & discussion of other infrastructure choices.
  • Deploying the application through the complete process of code, integrate, test, build, deploy.


Organizing Infrastructure Config & Workflow

September 7–8, 2016 @ Napa Valley, California

I’m super stoked about this talk, as I’m getting to co-present with Evan Brown. Evan’s a friend of mine since I met him a whiel back while he worked at AWS, however he’s now a Senior Software Engineer at Google. We’re going to tag team style this talk to bring you as much information as we can about organzing your infrastructure configuration and workflow.

My abstract for this talk goes something like this, “When starting with the various products Terraform, Packer, Vagrant, and others, it isn’t always apparent where and in what way one should organize the actual project. In this talk I’d like to delve into what I’ve done to organize solutions for development, production, and related pipelines. I’ll talk from my point of view and what I’ve seen others do to keep their workloads organized and their infrastructure and application pipelines clean and well organized.

Original Content Post: http://blog.adron.me/articles/speaking-trips-tech-and-treks/

Riak Developer Guidance

The “Client Round Robin Anti-Pattern”

One of the features that is often available in Riak Client software (including the CorrguatedIron .NET Client, the riak-js client and others) is the ability to send requests to the Riak Cluster through a round robin style approach. What this means is each IP, of each node within the Riak Cluster is entered into a config file for the client. The client then goes through that list to send off requests to read, write or delete data in the database.

The client being responsible and knowledgeable about the data tier of the application in an architecture is an immediate red flag! The concept around SoC (Separation of Concerns) dictates that

“SoC is a principle for separating a computer program into distinct sections, such that each section addresses a separate concern.

Having the client provide a network tier layer to round robin communication with the database leaves us in a scenario that should be separated into individual concerns. Below is some basic guidance on eliminating this SoC issue.

  • Client ONLY sends and receives communication: The client, especially in the situation with a distributed system like Riak should only be dealing with sending and receiving information from the cluster or a facade that provides an interface for that cluster.
  • Another layer should deal with the network communication and division of nodes and node communication. Ideally, in the case or Riak, and most distributed systems this should be dealt with at the network device layer (router).
  • The network device (router) layer would ideally be able to have (through software likely) a way to automate the failure, inclusion or exclusion of nodes with the cluster system. If a node goes down, the network device should handle the immediate cessation of communication with that node from all clients, routing the communication accordingly to an active node.
  • The node itself needs to maintain a continual information state available to the network. Ideally the network state would identify any addition or removal of a node and if possible the immediate failure of a node. Of course it isn’t always possible to be informed of a failure, but the first line of defense should start within the cluster itself among the nodes.

The Anti-Pattern

Having the client handle all of these parts of the functional architecture leads to a number of problems, not merely that the guidance of the SoC concept is broken. With the client attempting to track and be aware of the individual nodes in the cluster, it sets the client with a huge responsibility.

Take for instance the riak-js client. If a node goes down the client will need to be aware of which node has gone down. For a few seconds (yes, you have to wait entire seconds at this level) the node will be gone and the client won’t know it is down. The client would just have to reasonably wait. When the communication times out, the client would then have to have the responsibility of marking that particular node as down. At this point the client must track which node it is in some type of data repository local to the client. The client must also set a time or some way to identify when the node comes back up. Several questions start to come up such as;

  • Does the client do an arbitrary test to determine when the node comes back up?
  • When the node comes back up is it considered alive or damaged?
  • How would the client manage the IP (or identifier) of the node that has gone down?
  • How long would the client store that the node is down?

The list of questions can get long pretty quick, thus the bad karma of not following a good practice around separating your concerns appropriately! One has to be careful, a god class might be right around the corner otherwise! That’s it for this quick journey into some distributed database usage guidelines. Until next, happy data sciencing.  😉

Cloud Software Architect Necessities

Cloud Architecture is becoming more and more relevant in the software industry today.  A lot of efforts are becoming less about software and more about cloud software.  The whole gamut of cloud technology; platform, service, infrastructure, or platforms on platforms are growing rapidly in number.  The days you didn’t need to know what the cloud was are rapidly coming to an end.

As I move further along in my efforts with cloud development, both at application level and services levels, I’ve come to a few conclusions about what I do and do not need.  The following points are what I have recently drawn conclusions about, that cause frustration or are drawn from the beauty of cloud architecture.

REST is King, Period

When I use REST in this context, I don’t just mean a nice clean URI or “not SOAP”.  I’m talking about the whole entire enchilada of REST Architecture Principles.  A good place to start learning about REST is the Wikipedia Article.  Other key resources to check out include;  Roy T. Fielding’s Dissertation, specifically chapter 5, Principled Design of the Modern Web Architecture, and the O’Reilly Book RESTful Web Services.

REST Architecture is fundamental to the web and the fact that the web is continuous in uptime.  The web, or Internet, doesn’t go down, doesn’t crash, and is always available in some way.  REST Architecture and the principles around it are what enables the web to be this way.  The cloud seeks to have the same abilities, functionality, and basically be always on, thus REST Architecture is key to that underpinning.

Core Development

Currently I do most of my development with C# using the .NET Framework.  This is great for developing in cloud environments in Windows Azure and Amazon Web Services.  The .NET Framework has a lot of libraries that work around, with, and attempt to provide good RESTful Architecture.  However, there are also a lot of issues, such as the fact that WCF and ASP.NET at their core aren’t built with good intent against RESTful Architecture.  ASP.NET MVC and some of the latest WCF Releases (the out of band stuff) are extensively cleaning this up and I hear that this may be sooner than later.  I look forward to have cleaner, bare bones, fast implementations to use for RESTful development in Azure or AWS.

The current standing ASP.NET (Web Forms) Architecture is built well above the level it should be to utilize REST Architecture well.  Thus it creates a lot of overhead and unnecessary hardware utilization, especially for big sites that want to follow good RESTful Practice.

WCF on the other hand had some great ideas behind it, but as REST increased in popularity and demanded better use of the core principles the web is founded on, the WCF model has had to bend and give way some of its functional context in order to meet basic REST Architecture.

Anyway, that’s enough for my ramblings right now.  Just wanted to get some clear initial thoughts written down around Cloud Architecture needs for the software developer.