The following are some lagniappe, a little extra, about the behind the scenes adventures I’m off on when I travel or am in between coding. Ya know, coding being life and all. 😉
The first episode in this series I posted a while back on my gear I use to record the Twitch sessions and pretty much everything else. These are the story of the first half of the trip to Santa Clara and San Francisco. The rest, are still in post-production, and will be out real soon. Along with videos on a host of other adventures that will offer you good information on where the good food is, the best coding places, best meetups, and all that stuff. So subscribe on my Youtube Channel and on Twitch – the shows are coming to Youtube, and now and again I’ll pre-watch one with my Twitch audience. Cheers!
Here’s the run down of what I covered in the latest Thrashing Code Session (go subscribe here to the channel for future sessions or on Twitch). The core focus of this session was getting some further progress on my Terraform Project around getting a basic Cassandra and DataStax Enterprise Apache Cassandra Cluster up and running in Google Cloud Platform.
The code and configuration from the work is available on Github at terraform-fields and a summary of code changes and other actions taken during the session are further along in this blog entry.
Streaming Session Video
In this session I worked toward completing a few key tasks for the Terraform project around creating a Cassandra Cluster. Here’s a run down of the time points where I tackle specific topics.
3:03 – Welcome & objectives list: Working toward DataStax Enterprise Apache Cassandra Cluster and standard Apache Cassandra Cluster.
3:40 – Review of what infrastructure exists from where we left off in the previous episode.
5:00 – Found music to play that is copyright safe! \m/
8:52 – Adding a google_compute_address for use with the instances. Leads into determining static public and private google_compute_address resources. The idea being to know the IP for our cluster to make joining them together easier.
11:44 – Working to get the access_config and related properties set on the instance to assign the google_compute_address resources that I’ve created. I run into a few issues but work through those.
37:05 – I setup some files, following a kind of “bad process” as I note. Which I’ll refactor and clean up in a subsequent episode. But the bad process also limits the amount of resources I have in one file, so it’s a little easier to follow along.
54:27 – Starting to look at provisioners to execute script files and commands before or after the instance creation. Super helpful, with the aim to use this feature to download and install the DataStax Enterprise Apache Cassandra or standard Apache Cassandra software.
1:16:18 – Ah, a need for firewall rule for ssh & port 22. I work through adding those and then end up with an issue that we’ll be resolving next episode!
Session Content
Starting Point: I started this episode from where I left off last session.
Work Done: In this session I added a number of resources to the project and worked through a number of troubleshooting scenarios, as one does.
Added firewall resources to open up port 22 and icmp (ping, etc).
resource “google_compute_firewall” “bastion-icmp” {
name = “gimme-bastion-icmp”
network = “${google_compute_network.dev-network.name}”
allow {
protocol = “icmp”
}
}
[/sourcecode]
I also broke out the files so that each instances has its own IP addresses with it in the file specific to that instance. Later I’ll add context for why I gave the project file bloat, by refactoring to use modules.
Added each node resource as follows. I just increased each specific node count by one for each subsequent node, such as making this node1_internal IP google_compute_address increment to node2_internal. Everything also statically defined, adding to my file and configuration bloat.
Plans for next session include getting the nodes setup so that the bastion server can work with and deploy or execute commands against them without the nodes being exposed publicly to the internet. We’ll talk more about that then. For now, happy thrashing code!
I’ve got everything queued back up with some extra Thrashing Code Sessions and will have some on the rails travel streams. Here’s what the schedule looks like so far.
I’m going to get back into the roll of things this session after the travels last week. In this session I’m aiming to do several things:
Complete next steps toward getting a DataStax Enterprise Apache Cassandra cluster up and running via Terraform in Google Cloud Platform. My estimate is I’ll get to the point that I’ll have three instances that launch and will automate the installation of Cassandra on the three instances. Later I’ll aim to expand this, but for now I’m just going to deploy 3 nodes and then take it from there. Another future option is to bake the installation into a Packer deployed image and use it for the Terraform execution. Tune in to find out the steps and what I decide to go with.
I’m going to pull up the InteroperabilityBlackBox and start to flesh out some objects for our model. The idea, is based around something I stumbled into last week during travels, the thread on that is here.
Just got this idea. I buy plane tickets and often fly first class at economy prices because I fly and I know things. It’s not really a knee idea but @DeniseKGosnell’s keynote just pushed me further toward a prospective solution. It occurred to me… #gdfs18#graph
This Friday I’m aiming to cover some Go basics before moving further into the Colligere CLI app. Here are the highlights of the plan.
I’m going to cover some of the topics around program structure including: type declarations, tuple assignment, variable lifetime, pointers, and other variables.
I’m going to cover some basics on packages, initialization of packages, imports, and scope. This is an important aspect of ongoing development with Colligere since we’ll be pulling in a number of packages for generation of the data.
Setting up configuration and schema for the Colligere application using Viper and related tooling.
This session I’m aiming to get some more Terraform work done around the spin up and shutdown of the cluster. I’ll dig into some more specific points depending on where I progress to in sessions previous to this one. But it’s on the schedule, so I’ll update this one in the coming days.
The combined Seattle Scalability + Cassandra meetup, which I dubbed Operations & Developers of Distributed Systems, was a success. There was some awesome Twitter interactions too including Ashley Ratieri’s notes. I also posted a few pictures of the group. Fellow DataStaxian @nkmoha came into town to record the meetup, and I’ll be sure to get that posted out to the meetups and here once the videos are posted.
The next two meetups are scheduled for September and October already.
Started regularly streaming on Twitch. Follow, subscribe, comment, throw code ideas my way and any ideas you’d like me to code against. I’m open to all sorts of things to add value to the streams.
Finished up a bunch of video streams on a range of content:
Put together a Github Pages site for adding detailed information about what I’ve learned about live streaming coding and related content via Twitch and YouTube. The repo is here and the site is here.
Recently my friend swung by to record a Twitch stream on setting up the network, hardware, OS loads, PXE boot, and related items for the cluster in the office. Here’s a rundown of notes, details, and related links and such from the video, and as icing on the cake, here’s the video after some edits. I’ve managed to cut the Twitch stream down to a clean 150 minutes and 40 seconds from over 240 minutes! A little bit more digestible in this shorter format.
Some of the additional changes, that you’ll distinctly notice if you watched the original stream, is that I cleaned up the audio in some places, attempted to raise the audio in others, and for the mono audio I bumped it out to stereo so it isn’t so strange to listen to. I’ve also added callout text for all the configuration files edited, and some other commands here and there were Jeremy uses his fast typing to open a file so fast you might not see which it was. Overall, it should be easier to listen to, more bite size in segments, and more useful to reference.
In addition, below I’ve broken out key parts of the process at their respective time points.
0:00:56 Determining what’s going to be setup and installed; Starting with a net install, setting up bastion server first, then figuring out the cassandra install once we have that.
0:02:50 Whoops, wrong initial iso image. Cut out the cycling through BIOS and mess as that needs to be setup per whatever machines are involved. But now, back to the correct Debian image.
0:04:46 Plan is: Initial server will be a DHCP server setup so we can setup TFTP options to clients. – TFTP Trivial File Transfer Protocol – TFTP will then be used to grab the kernal and ramdisk for Debian which will contain the installer. This will include a preseed file for Debian.
0:07:36 Initial setup of Debian.
0:08:05 Setup of the NIC with two ports begins. One port for internet, one port for inward facing network for the cluster.
0:11:04 Setup of the drives, since there’s a number of disks in the bastion server. Includes setup of home, swap, etc and Jeremy’s thoughts on that. Then some booting conflicts with the ssd vs. flash vs. other drives.
0:12:40 Jeremy and I discuss swap, failure approaches and practices.
0:14:10 Discussing various ways to automate the installation. Preseed, kickstart, various automated installs, etc.
0:30:38 A request for a bigger font, so Jeremy tweaks that while the webcam camera goes all auto-focus nightmare.
0:31:26 Jeremy installs his dot files. For more on dot files check out here, here, here, and all the dot files on Github here are useful.
0:55:00 Adding iptables rules for additional changes made since initial iptables setup.
0:55:40 Setup/download inetd, tftpd, and other tools on bastion to setup remaining network and servers. Jeremy also provides an explanation of the daemons and how and what is getting setup now and respective DDOS concerns.
0:57:40 Starts download of netboot installer and everything else required to netboot a machine.
0:58:48 First pxelinux steps. Setup of configuration file.
1:00:52 First pxeboot attempt of a node server. KVM Switch confusion, and a flicker of life on the first node!
1:01:50 The pxe boot got the installer started, but no further steps were taken. Jeremy delves into the log files to determine why the pxe boot didn’t launch further into the installer.
1:04:20 Looks up pxelinux to setup some of the defaults and determine solutions. Good reference point for further research is available via syslinux.org.
1:05:40 After a couple minutes of pxelinux information, back to the configuration.
1:07:23 Jeremy gets into preseed configuration but before diving in too deep covers some ground on what to do in a production environment versus what is getting setup in the current environment for this video.
1:08:47 Takes a preseed example to work from. Works through the file to setup the specifics of the installation for each of the nodes.
1:21:46 Finished reviewing the installation for the nodes. Started at 1:08:47, so it took a while to cover all those bases!
1:22:28 Moves new preseed file into the correct path in the installer inetrd.
1:27:44 After some troubleshooting, the pxeboot loading gets to the business of successfully loading a node, success! We follow up a short celebration of getting the first pxeboot with a little summary, restating the purpose of pxeboot, and some form and function of how pxeboot will work, i.e. pxeboot works when all other boot methods on a node fail. So when a drive is totally formatted and no master boot record to load from, boot, kicks off via pxeboot and we get a new image. Thus, it’s as simple as plugging in a new server and turning it on and I’ll have a brand new node in the cluster.
1:29:30 Cycling through each of the servers, which we’ve powered on to start loading from pxeboot, just to watch them start loading and to try an initial load.
1:29:51 Jeremy discusses the reasoning behind setting up some things specifically to the way they’ve been setup specific to having a database loaded on the servers versus just standard servers or another configuration of use.
1:32:13 With Jeremy declaring, “it’s just YAML right?!” we opt to use Ansible for the next steps of configuration for the nodes and their respective Cassandra database installations and setup.
1:32:22 Jeremy and I have a small debate about Python being trash for CLI’s or not. I’m wrong but it’s still garbage for a CLI, why does it keep getting used for CLI’s, use Go already. Grumble grumble, whatever.
1:35:46 Jeremy and I now have a discussion on the configuration related to IP’s, what the range would give us, how to assign them specifically, service discoverability of the nodes, and how all of this complexity can be mitigated by something simple for us to setup right now, instead of later.
1:38:25 After discussing, we opt to go with a hard coded DHCP configuration for now so we can just use the static (in function, but now literally of course since they’re designated in the DHCP) IP’s.
1:41:56 Setting up the actual Ansible Playbook for Cassandra starts here.
1:43:27 Executing Ansible and troubleshooting connectivity issues between nodes.
1:43:48 Need sshpass, installed with[sourcecode language=”bash”]sudo apt-get install sshpass[/sourcecode]
1:44:18 We realize there’s a problem with the machines actually re-installing via pxe boot and change boot sequence to give that a temp fix.
1:45:25 Back into the Ansible Playbook. Further troubleshooting of the Ansible playbook.
1:45:59 Checking out the ansible hosts file.
1:46:19 Checking out the cassandra.yml for Ansible further. Which leads to…
1:46:53 …the realization we’re also installing Spark in addition to Cassandra. Hmmm, Ok, let’s do that too.
1:47:xx For the next several minutes Jeremy steps through and makes additions and fixes to the Ansible file(s).
1:59:52 Took a break at this point.
2:01:11 Had names for IP’s swapped. Fixed.
2:03:09 3 of the 5 nodes are now reachable after some troubleshooting.
2:03:27 After some more quick checks and troubleshooting executing the playbook again.
2:05:15 Oh dear it appears to be install Cassandra finally! Even with 3 nodes and 2 failing, that’s enough for quorum so moving forward!
2:08:44 Further edits and tweaks to the playbook.
2:09:20 Executing playbook again.
2:09:36 Removing further cruft from the playbook we copied.
2:09:53 Troubleshooting the Cassandra loads.
2:15:04 At this point there’s three executing Cassandra installs pulled from FTP for installation and on nodes.
2:30:40 End
As a shout out and props to Jeremy, when you need or want a new hard drive, check our https://diskprices.com/. It’s pretty solid for finding excellent disk prices and Jeremy gets a few pennies per order. Cheers, thanks for watching, and thanks Jeremy for swinging by and educating us all!
UPDATED: Video available 
This Friday I’m aiming to cover some Go basics before moving further into the 
This session I’m aiming to get some more Terraform work done around the spin up and shutdown of the cluster. I’ll dig into some more specific points depending on where I progress to in sessions previous to this one. But it’s on the schedule, so I’ll update this one in the coming days.
You must be logged in to post a comment.