Tag: devops

Terrazura – A Build Out of an Azure based, Hasura GraphQL API on Postgres

I created this repo https://github.com/Adron/terrazuraā€‹ during a live stream on my Twitch Thrashing Code Channel šŸ¤˜ at 10am on the 30th of December, 2020. The VOD is now available on my YouTube Thrashing Code Channel https://youtube.com/thrashingcodeā€‹. A rough as hell year, but wanted to wrap it up with some solid content. In this stream I tackled a ton of specifics, in detail about getting Hasura deployed in Azure, Postgres backed, a database schema designed and created, using database schema migrations, and all sorts of tips n’ tricks along the way. 3 hours of solid how to get shit done material!

For live streams, check out and follow at https://www.twitch.tv/thrashingcodeā€‹ šŸ‘ŠšŸ» or for VOD viewing check out https://youtube.com/thrashingcode

A point in coding during the video!

02:49ā€‹ – Shout out to the stream sponsor, Azure, and links to some collateral material.
14:50ā€‹ – In this first segment, I start but run into some troubleshooting needs around the provider versions for Terraform in regards to Azure. You can skip this part unless you want to see what issue I ran into.
18:24ā€‹ – Since I ran into issues with the current version of Terraform I had installed, at this time I show a quick upgrade to the latest version.
27:22ā€‹ – After upgrading and fighting through trial and error execution of Terraform until I finally get the right combination of provider and Terraform versions.
27:53ā€‹ – Adding the first Terraform resource, the Azure resource group.
29:47ā€‹ – Azure Portal oddness, just to take note off if/when you’re working through this. Workaround later in the stream.
32:00ā€‹ – Adding the Postgres server resource.
44:43ā€‹ – In this segment I switched over to Jetbrain’s Intellij to do the rest of the work. I also tweak the IDE to re-add the plugin for the material design themes and icons. If you use this IDE, it’s very much IMHO worth getting this to switch between themes.
59:32ā€‹ – After getting leveled-up with the IDE, I wrap up the #Postgresā€‹ server resource and #terraformā€‹ apply it the overall set of resources. At this point I also move forward with the infrastructure as code, with emphasis on additive changes to the immutable infrastructure by emphasizing use of terraform apply and minimizing any terraform destroy use.
1:02:07ā€‹ – At this time, I try figuring out the portal issue by az logout and logging back in az login to Azure Still no resources shown but…
1:08:47ā€‹ – eventually I realize I have to use the hack solution of pasting the subscription ID into the
@Azure portal to get resources for the particular subscription account which seems highly counter intuitive since its the ONLY account. šŸ§
1:22:54ā€‹ – The next thing I setup, now that I have variables that need passed in on every terraform execution, I add a script to do this for me.
1:29:35ā€‹ – Next up is adding the database to the database server and firewall rule. Also we get to see Jetbrains #Intellijā€‹ HCL plugin introspection at work adding required properties to the firewall resource! A really useful feature.
1:38:24ā€‹ – Next up, creating the Azure container to deploy our Hasura GraphQL API for #Postgresā€‹ to!
1:51:42ā€‹ – BAM! API Server is done and launched! I’ve got a live #GraphQLā€‹ API up and running in Azure and we’re ready to start building a data model!
1:56:22ā€‹ – In this segment I show how to turn off the public facing console and shift one’s development workflow to the local Hasura console working against – local OR your live dev environment.
1:58:29ā€‹ – Next segment I get into schema migrations, initializing a directory structure for Hasura CLI use, and metadata, migrations, and related data. Including an update to the latest CLI so you can see how to do that, after a run into a slight glitch. šŸ˜¬
2:23:02ā€‹ – I also shift over to dbdiagram to graphically build out some of the schema via their markdown, then use the SQL export option for #postgresā€‹ combined with Hasura’s option to execute plain ole SQL via migrations…
2:31:48ā€‹ – Getting a bit more in depth in this segment, I delve through – via the Hasura console – to build out relationships between the tables and data so the graphql queries can introspect accordingly.
2:40:30ā€‹ – Next segment, graphql time! I show some of the options of what is available immediately for queries and mutations via the console.
2:50:36ā€‹ – Then some more details about metadata. I’m going to do a stream with further details, since I was a little fuzzy on some of those details myself, in the very very near future. However a good introduction to what the metadata does for the #graphqlā€‹ API.
2:59:07ā€‹ – Then as a wrap up to all of this… I nuke EVERYTHING and deploy it all out to Azure again inclusive of schema migrations, metadata, etc. šŸ¤˜šŸ»
3:16:30ā€‹ – Final segment, I add some data to the database and get into a few basic queries and mutations in #graphqlā€‹ via the #graphiqlā€‹ console interface in #Hasuraā€‹.

TRIP REPORT: O’Reilly Velocity & Software Architecture Conf 2019

This past week the O’Reilly Velocity and Software Architecture Conferences took place. I’ve attended both before, the 2nd time for Velocity in San Jose and the 2nd time for Software Architecture, however this time in San Jose and the last I attended was in London. The locations for these conferences dictate much about what is presented and how conversations, meeting and interacting, learning, and explorations take place during the conference, but more on those specifics in a moment.

The overarching theme from keynotes and many of the conversations I had met on a few key topics:

  1. When you’re building software, and you want to do it well you first and foremost must, absolutely must, invest in the people building your software.
  2. Focus on simplicity, remove complexity at every opportunity.
  3. Organizational structure can have direct impact in the complexity or simplicity of software, structure your organization efficiently and make every effort to keep it simple.

TLDR; Keep your people happy, focus on simplicity, minimize organizational noise from bureaucracy.

Topically Elaborating on Edge and Serverless

Ok, so a number of conversations came up around edge computing and serverless. Both interesting, but it also seems like there isn’t a strong play for the Enterprise is either space just yet. At this point however, a lot of enterprises are struggling with their Kubernetes, Cloud, and Hybrid solutions enough as it is that they haven’t even broached the edge compute and serverless realm. But that hasn’t stopped a lot of forward thinking individuals to start tackling how to cut out a useful spectrum of application with both edge and serverless.

Serverless Oh My!

Serverless, like so many other names, is kind of a garbage name at first. It effectively tells us nothing useful. It’s a word that requires more words just to give it meaning. It’s kind of like if I said “I like food!” What does this even mean? So, everybody eats food and most people like it, so what does “I like food!” actually mean? Same with serverless, because the first thing it doesn’t mean is a system that has no servers. What it generally means is something about your applications and you not doing anything with servers. It’s the mythic NoOps by simply removing the computers somehow.

Serverless, or simply code that executes via compute, and you get a result, has actually been around for some time. There were a number of startups that were well ahead of AWS Lambda and the other respective implementations that Azure and GCP have. These startups had been attempting to usher in what it took AWS’s massiveness and clout to actually get people to pay attention to. Serverless however has gone far beyond merely Lambda at AWS and now we’ve got to contend not only with the option in the existing cloud providers but where, how, and when can we get it into our data centers! The TLDR is that enterprise wants serverless and they’re interested in throwing it onto Kubernetes or whatever they’ve got. But often the infrastructure and systems to really make use of this simply isn’t there.

Most of the conversations I had evolved around the who, what, where, when, and how do we make use of these options for what we do? This is where most companies, at least enterprise and large companies, currently seem to be in the market. Then there are the companies that have already made the leap and are doing all sorts of stuff with Lambda and related serverless offerings. The gulf, that middle ground, doesn’t seem to have been broached by many others. Everybody, anecdotally of course, seems to be either trying to figure out how to start or already made the leap!

Edge Compute

This kept coming up, regardless of how or what people defined it as, it came up as something a number of people were very interested in. This notion, loosely based around using edge devices; smart phones, IoT devices, your car, or your washer for example, it could be almost anything. These devices do compute on the edge and thus the term. However it’s interesting because it isn’t like, for example, cloud computing that has core features like compute, storage, and related elements. Edge computing can run the gamut of any device doing any kind of work and the related capabilities of that device. It kind of leaves the space wide open. However, there were a few focal points that kept coming up.

The most common topic that came up around edge computing was doing tasks at point of presence. Such as having a phone do facial recognition, computing path finding (i.e. traffic directions), and related compute on the device versus round tripping it back to the cloud. It almost seems like after all these years of pushing things to the server we’re really starting in earnest to bring smart processes and tasks back to the devices we have in hand – no pun intended. It’s an interesting space, interesting paradigms, and I’m still not ready to call a specific thing within the world of edge compute and say, “that’s the next billion dollar idea”. Largely because, there are a lot of billion dollar ideas out there these days.

Speaking of edge compute and serverless, my fellow DataStaxian also had a few of these conversations on said topics. Patrick wrote up a post on a few observations over on the DataStax blog “Velocity Conference Shows Whatā€™s Gaining Velocity in Data Management“.

Geographic Location

As I mentioned, this set of conferences is in San Jose, the home of Silicon Valley, but the southern segment of the area. It’s a walk-able area with a number of places to break out from the conference and really dig into the hallway tracks (i.e. impromptu conversations!) that come up. For those willing to jump on the light rail, or scooter around, San Jose opens up even more to the local area providing a wide variety of coffee, food, and other operations to share conversations over.

All in all, the geographic location for the event is solid, being in the center of the city where it is. However one issue did arise, the Marriott lost power as an electrical fire in the control room of the multi-story hotel blew out the power. At last I checked upon leaving, it still didn’t have power! With the temperature at 105f going on multiple days at this point, the hotel because extremely hot inside, and being a kind of sealed airspace the air calculators also weren’t refreshing the air. That left a number of guests in less than stellar condition to attend, let alone attain value, from the conference events. Myself I ended up checking out in short order, getting sick the last day of the conference anyway, and being unable to provide the presentation that I had paired up with Lena (@lenadroid) for! I’ve been thinking, that maybe she and I can provide an online version of it for those that had wanted to hear us present on “Flexible Cloud Architectures: Decision Making Best Practices“.

Next year’s Velocity looks like it’ll be in Santa Clara, which doesn’t really excite me as it’s kind of a nebula of sprawling suburbia of boredom. This is were location becomes fundamental to what will or what can be the potential of secondary and tertiary conversations at a conference like this. Don’t get me wrong, the hallway track is excellent, but having options to step out and walk across the street from the event to converse further adds a tremendous value.

Santa Clara simply doesn’t do that unfortunately.

The fortunate thing between now and then, albeit the conference is moving to Santa Clara, they’re having subsequent conferences in the Velocity series in Berlin, and Software Architecture Conf series in the amazing cities of New York and Berlin. Those locations are worth traveling to for far more than a conference, increasing my interest in attending both of those future events. I’m looking forward to these!

Twitter Talk @VelocityConf

FromĀ @DataStaxDevsĀ a thread! Click through for all ~17 parts.

Some Build Engineer Work – Click through for the whole construction thread.

Some of the Keynote Threads

Alena Hall – @lenadroid

Jessica Kerr – @jessitron

…and there were a bunch of others too, solid, check out the hash tag of #velocityconf to read up on more.

The Lagniappe

After the conference I finally managed to pick up a pocket Constitution.

If you’re ever in search of good coffee in San Jose, one place I found that’s tops is Academic Coffee, both the coffee and service are great. Good jovial crew and lots of cyclists in and out.

Making progress on the CaSMa, tweeted a bit on the topic while en route to the conference. If you’d like to get involved, please do let me know!

Other arbitrary statistics:

  • Stickers collected: 11 unique, ~7 of each. Total: 77 stickers.
  • T-shirt Swag: 2.
  • Conversations @ DataStax Booth: 11
  • Hallway Track Conversations: 7
  • Coffee Consumed: 9 over 3 days.
  • Twitter Filters Discussed: 123.
  • Fuel burned to compensate for electrical fire damage for the time of the conference: Approximately 5k gallons of fuel for the Marriott Hotel and no idea how much more fuel was or is still being burned to power the hotel.
  • Times the power still went off even with the diesel engine power trailer attached: 4.

DataStax Developer Days

Over the last week I had the privilege and adventure of coming out to Chicago and Dallas to teach about operations and security capabilities of DataStax Enterprise. More about that later in this post, first I’ll elaborate on and answer the following:

  • What is DataStax Developer Day? Why would you want to attend?
  • Where are the current DataStax Developer Day events that have been held, and were future events are going to be held?
  • Possibilities for future events near a city you live in.

What is DataStax Developer Day?

The way we’ve organized this developer day event at DataStax, is focused around the DataStax Enterprise built on Apache Cassandra product, however I have to add the very important note that this isn’t merely just a product pitch type of thing, you can and will learn about distributed databases and systems in a general sense too. We talk about a number of the core principles behind distributed systems such as the pivotally importantĀ consistent hash ring, datacenter and racks, gossip, replication, snitches, and more. We feel it’s important that there’s enough theory that comes along with the configuration and features covered to understand who, what, where, why, and how behind the configuration and features too.

The starting point of the day’s course material is based on the idea that one has not worked with or played with a Apache Cassandra or DataStax Enterprise. However we have a number of courses throughout the day that delve into more specific details and advanced topics. There are three specific tracks:

  1. Cassandra Track – this track consists of three workshops: Core Cassandra, Cassandra Data Modeling, and Cassandra Application Development.Ā [more details]
  2. DSE Track – this track consists of three workshops: DataStax Enterprise Search, DataStax Enterprise Analytics, and DataStax Enterprise Graph.Ā [more details]
  3. Bonus Content – This track has two workshops: DataStax Enterprise Overview and DataStax Enterprise Operations and Security.Ā  [more details]

Why would you want to attend?

  • One huge rad awesome reason is that the developer day events are FREE. But really, nothing is ever free right? You’d want to take a day away from the office to join us, so there’s that.
  • You also might want to even stay a little later after the event as we always have a solidly enjoyable happy hour so we can all extend conversations into the evening and talk shop. After all, working with distributed databases, managing data, and all that jazz is honestly pretty enjoyable when you’ve got awesome systems like this to work with, so an extended conversation into the evening is more than worth it!
  • You’ll get a firm basis of knowledge and skillset around the use, management, and more than a few ideas about how Apache Cassandra and DataStax Enterprise can extend your system’s systemic capabilities.
  • You’ll get a chance to go beyond merely the distributed database system of Apache Cassandra itself and delve into graph, what it is and how it works, analytics, and search too. All workshops take a look at the architecture, uses, and what these capabilities will provide your systems.
  • You’ll also have one on one time with DataStax engineers, and other technical members of the team to ask questions, talk about architecture and solutions that you may be working on, or generally discuss any number of graph, analytics, search, or distributed systems related questions.

Where are the current DataStax Developer Day events that have been held, and were future events are going to be held? So far we’ve held events in New York City, Washington DC, Chicago, and Dallas. We’ve got two more events scheduled with one in London, England and one in Paris, France.

Future events? With a number of events completed and a few on the calendar, we’re interested in hearing about future possible locations for events. Where are you located and where might an event of this sort be useful for the community? I can think of a number of cities, but organizing them into order to know where to get something scheduled next is difficult, which is why the team is looking for input. So ping me via @Adron, email, or just send me a quick message from here.