Terraform “Invalid JWT Signature.”

I ran into this issue recently. The “Invalid JWT Signature.” error while running some Terraform. It appeared to occur whenever I was setting up a bucket in Google Cloud Platform to use for a back-end to store Terraform’s state. In console, here’s the exact error.


My first quick searches uncovered some Github issues that looked curiously familiar. Invalid JWT Token when using Service Account JSON #3100 which was closed without any particular resolution. Upon further searching it didn’t help to much but I’d be curious as to what the resolution was. The second is Creating GCP project in terraform #13109 which sounded much more on point compared to my issue. This appeared closer to my issue but it looked like I should probably just start from scratch, since this did work on one machine already but just didn’t work on this machine I shifted to. (Grumble grumble, what’d I miss).

The Solution(s)?

In the end this is a message, if you work on multiple machines with multiple cloud accounts you might get the keys mixed up. In this particular case I reset my NIC (i.e. you can just reboot too, especially if on Windows it’s just easier to do that). Then everything just started working. In some cases however, the JSON with the gcloud/gcp keys needs to be regenerated as the old key was rolled or otherwise invalidated.


Today’s post is brought to you by Bedlam Coffee in Seattle.

Today’s post is brought to you by Bedlam Coffee in Seattle. It’s an old school 2nd gen coffee shop, kitschy blue collar style, very cool and very endearing. It’s a great place to chill, read a book, hack a little, or generally talk to people in the Belltown Neighborhood. It’s a good part of Seattle and in the back of my mind I have that fear it’ll be replaced by a towering condo complex and a sterile modern steel, wood, and glass something another any minute now. But for the moment it’s a prime spot to get wired into one’s mind.

Post Topics: I’m gonna hit on a number of key topics. Curious your thoughts, and as always, tweet at me via twitter, comment here, or ping me however. Always glad to chat on things.

  • Climate Leave — The second I saw this post, by Anil Dash I immediately had a few thoughts pop into my mind. The first is all companies should start recording this climate leave data point so we can have hard data on what the climate catastrophe is costing companies in hard dollar terms. It’s a way to wake up some people’s thinking to the reality of these extremities and make climate change measurable in ways they’re currently unwilling, unable, or outright refusing to accept or address. On the other side, this is a hard reality of work and life today. For instance, the regularity in which northwest cities shut down over a little snow is frequent now. But it’s only because for many decades west coast cities have existed and not prepared for such, because such just didn’t happen with regularity. So many initial thoughts, and the post itself is really good and inspires additional thinking, as one often may assume from writing Anil tends to do.
  • The difference in JVM vs CLR worlds is quite interesting states Bart Sokol. He inquired with a 280 character tweet, which I’d love to see others weigh in on how this is perceived and played out for you.

  • An overheard Oracle conv thread from yours truly happened today too. I just can’t even start with Oracle, everytime I see their tooling in play or discussed, it’s almost always followed by a giant trash fire. The company owns so much and has so much potential to do good but instead all they do is spread horrid reputation, trash systems that could help people, and fall down on their job (i.e. go check out the Oregon health care trash fire they supposedly *built*. Not to release Oregon Gov from some responsibility, but they brought in Oracle with the idea that they knew what they were doing and they ruined the project and basically sucked the project dry of funds, leaving Oregonians without in the end).

  • The Former CEO of Brightcove took the rather broken tax plan of the current GOP (or whatever oddball ghost of the GOP it is) to task. It’s a fairly solid write up of the whole matter. It leads to lots of other things one might want to look into if you’re curious about the matter.
  • In a post that I can truly relate with, M.G. Siegler heads north on the Coast Starlight and wrote a nice piece about traveling by train. Why it’s as wonderful as it is, not particularly because of the train itself but what the train provides. To note, Siegler is formerly of Google Ventures, wrote for TechCrunch and VentureBeat among others and has a pretty well connected line with the tech industry. He’s a good person to follow and read.
  • In other news, David Mytton (who got a follow on Medium and Twitter out of this post!) and team at Server Density has finished a 9 month project to migrate from Softlayer to Google Cloud. Similar to efforts I helped kick off at Home Depot and am helping others do. If you’re interested in help around migration of systems; onsite or offsite, co-location or otherwise, to Google Cloud, AWS, or Azure reach out and let me know. I’d love to talk shop about your progress and if Pelotech or I could provide a helping hand. We’ve got a lot of experience in those efforts!

Build a Kubernetes Cluster on Google Cloud Platform with Terraform

terraformIn this blog entry I’m going to detail the exact configuration and cover in some additional details the collateral resources you can expect to find once the configuration is executed against with Terraform. For the repository to this write up, I create our_new_world available on Github.

First things first, locally you’ll want to have the respective CLI tools installed for Google Cloud Platform, Terraform, and Kubernetes.

Now that all the prerequisites are covered, let’s dive into the specifics of setup.

gcpIf you take a look at the Google Cloud Platform Console, it’s easy to get a before and after view of what is and will be built in the environment. The specific areas where infrastructure will be built out for Kubernetes are in the following areas, which I’ve taken a few screenshots of just to show what the empty console looks like. Again, it’s helpful to see a before and after view, it helps to understand all the pieces that are being put into place.

The first view is of the Google Compute Engine page, which currently on this account in this organization I have no instances running.


This view shows the container engines running. Basically this screen will show any Kubernetes clusters running, Google just opted for the super generic Google Container Engine as a title with Kubernetes nowhere to be seen. Yet.


Here I have one ephemeral IP address, which honestly will disappear in a moment once I delete that forwarding rule.


These four firewall rules are the default. The account starts out with these, and there isn’t any specific reason to change them at this point. We’ll see a number of additional firewall settings in a moment.


Load balancers, again, currently empty but we’ll see resources here shortly.


Alright, that’s basically an audit of the screens where we’ll see the meat of resources built. It’s time to get the configurations built now.

Time to Terraform Our New World

Using Terraform to build a Kubernetes cluster is pretty minimalistic. First, as I always do I add a few files the way I like to organize my Terraform configuration project. These files include:

  • .gitignore – for the requisite things I won’t want to go into the repository.
  • connections.tf – for the connection to GCP.
  • kubernetes.tf – for the configuration defining the characteristics of the Kubernetes cluster I’m working toward getting built.
  • README.md – cuz docs first. No seriously, I don’t jest, write the damned docs!
  • terraform.tfvars – for assigning variables created in variables.tf.
  • variables.tf – for declaring and adding doc/descriptions for the variables I use.

In the .gitignore I add just a few items. Some are specific to my setup that I have and IntelliJ. The contents of the file looks like this. I’ve included comments in my .gitignore so that one can easily make sense of what I’m ignoring.

# A silly MacOS/OS-X hidden file that is the bane of all repos.

# .idea is the user setting configuration directory for IntelliJ, or more generally Jetbrains IDE Products.

The next file I write up is the connections.tf file.

provider "google" {
  credentials = "${file("../secrets/account.json")}"
  project     = "thrashingcorecode"
  region      = "us-west1"

The path ../secrets/account.json is where I place my account.json file with keys and such, to keep it out of the repository.

The project in GCP is called thrashingcorecode, which whatever you’ve named yours you can always find right up toward the top of the GCP Console.


Then the region is set to us-west1 which is the data centers that are located, most reasonably to my current geographic area, in The Dalles, Oregon. These data centers also tend to have a lot of the latest and greatest hardware, so they provide a little bit more oompf!

The next file I setup is the README.md, which you can just check out in the repository here.

Now I setup the variables.tf and the terraform.tfvars files. The variables.tf includes the following input and output variables declared.

// General Variables

variable "linux_admin_username" {
  type        = "string"
  description = "User name for authentication to the Kubernetes linux agent virtual machines in the cluster."

variable "linux_admin_password" {
  type ="string"
  description = "The password for the Linux admin account."

// GCP Variables
variable "gcp_cluster_count" {
  type = "string"
  description = "Count of cluster instances to start."

variable "cluster_name" {
  type = "string"
  description = "Cluster name for the GCP Cluster."

// GCP Outputs
output "gcp_cluster_endpoint" {
  value = "${google_container_cluster.gcp_kubernetes.endpoint}"

output "gcp_ssh_command" {
  value = "ssh ${var.linux_admin_username}@${google_container_cluster.gcp_kubernetes.endpoint}"

output "gcp_cluster_name" {
  value = "${google_container_cluster.gcp_kubernetes.name}"

In the terraform.tfvars file I have the following assigned. Obviously you wouldn’t want to keep your production Linux username and passwords in this file, but for this example I’ve set them up here as the repository sample code can only be run against your own GCP org service, so remember, if you run this you’ve got public facing default linux account credentials exposed right here!

cluster_name = "ournewworld"
gcp_cluster_count = 1
linux_admin_username = "frankie"
linux_admin_password = "supersecretpassword"

Now for the meat of this effort. The kubernetes.tf file. The way I’ve set this file up is as shown.

resource "google_container_cluster" "gcp_kubernetes" {
  name               = "${var.cluster_name}"
  zone               = "us-west1-a"
  initial_node_count = "${var.gcp_cluster_count}"

  additional_zones = [

  master_auth {
    username = "${var.linux_admin_username}"
    password = "${var.linux_admin_password}}"

  node_config {
    oauth_scopes = [

    labels {
      this-is-for = "dev-cluster"

    tags = ["dev", "work"]

With all that setup I can now run the three commands to get everything built. The first command is terraform init. This is new with the latest releases of Terraform, which pulls down any of the respective providers that a Terraform execution will need. In this particular project it pulls down the GCP Provider. This command only needs to be run the first time before terraform plan or terraform apply are run, if you’ve deleted your .terraform directory, or if you’ve added configuration for something like Azure, Amazon Web Services, or Github that needs a new provider.


Now to ensure and determine what will be built, I’ll run terraform plan.


Since everything looks good, time to execute with terraform apply. This will display output similar to the terraform plan command but for creating the command, and then you’ll see the countdown begin as it waits for instances to start up and networking to be configured and routed.


While waiting for this to build you can also click back and forth and watch firewall rules, networking, external IP addresses, and instances start to appears in the Google Cloud Platform Console. When it completes, we can see the results, which I’ll step through here with some added notes about what is or isn’t happening and then wrap up with a destruction of the Kubernetes cluster. Keep reading until the end, because there are some important caveats about things that might or might not be destroyed during clean up. It’s important to ensure you’ve got a plan to review the cluster after it is destroyed to make sure resources and the respective costs aren’t still there.

Compute Engine View

In the console click on the compute engine option.


I’ll start with the Compute Engine view. I can see the individual virtual machine instances here and their respective zones.


Looking at the Terraform file confiugration I can see that the initial zone to create the cluster in was used, which is us-west1-a inside the us-west1 region. The next two instances are in the respective additional_zones that I marked up in the Terraform configuration.

additional_zones = [

You could even add additional zones here too. Terraform during creation will create an additional virtual machine instance to add to the Kubernetes cluster for each increment that initial_node_count is set to. Currently I set mine to a variable so I could set it and other things in my terraform.tfvars file. Right now I have it set to 1 so that one virtual machine instance will be created in the initial zone and in each of the designated additional_zones.

Beyond the VM instances view click on the Instance groups, Instance templates, and Disks to seem more items setup for each of the instances in the respective deployed zones.

If I bump my virtual machine instance count up to 2, I get 6 virtual machine instances. I did this, and took a screenshot of those instances running. You can see that there are two instances in each zone now.


Instance groups

Note that an instance group is setup for each zone, so this group kind of organizes all the instances in that zone.


Instance Templates

Like the instance groups, there is one template per zone. If I setup 1 virtual machine instance or 10 in the zone, I’ll have one template that describes the instances that are created.



To SSH into any of these virtual machine instances, the easiest way is to navigate into one of the views for the instances, such as under the VM instances section, and click on the SSH button for the instance.


Then a screen will pop up showing the session starting. This will take 10-20 seconds sometimes so don’t assume it’s broken. Then a browser based standard looking SSH terminal will be running against the instance.



This comes in handy if any of the instances ends up having issues down the line. Of all the providers GCP has made connecting to instances and such with this and tools like gcloud extremely easy and quick.

Container Engine View

In this view we have cluster specific information to check out.


Once the cluster view comes up there sits the single cluster that is built. If there are additional, they display here just like instances or other resources on other screens. It’s all pretty standard and well laid out in Google Cloud Platform fashion.


The first thing to note, in my not so humble opinion, is the Connect button. This, like on so many other areas of the console, has immediate, quick, easy ways to connect to the cluster.


Gaining access to the cluster that is now created with the commands available is quick. The little button in the top right hand corner copies the command to the copy paste buffer. The two commands execute as shown.

gcloud container clusters get-credentials ournewworld --zone us-west1-a --project thrashingcorecode

and then

kubectl proxy


With the URI posted after execution of kubectl proxy I can check out the active dashboard rendered for the container cluster at

IMPORTANT NOTE: If the kubectl version isn’t up to an appropriate parity version with the server then it may not render this page ok. To ensure that the version is at parity, run a kubectl version to see what versions are in place. I recently went through troubleshooting this scenario which rendered a blank page. After trial and error it came down to version differences on server and client kubectl.


I’ll dive into more of the dashboard and related things in a later post. For now I’m going to keep moving forward and focus on the remaining resources built, in networking.

VPC Network


Once the networking view renders there are several key tabs on the left hand side; External IP addresses, Firewall rules, and Routes.

External IP Addresses

Setting and using external IP addresses allow for routing to the various Kubernetes nodes. Several ephemeral IP addresses are created and displayed in this section for each of the Kubernetes nodes. For more information check out the documentation on reserving a static external IP address and reserving an internal IP address.


Firewall Rules

In this section there are several new rules added for the cluster. For more information specific to GCP firewall rules check out the documentation about firewall rules.



Routes are used to setup paths mapping an IP range to a destination. Routes setup a VPC Networks where to send packets for a particular IP address. For more information about documentation route details.


Each of these sections have new resources built and added as shown above. More than a few convention based assumptions are made with Terraform.

Next steps…

In my next post I’ll dive into some things to setup once you’ve got your Kubernetes cluster. Setting up users, getting a continuous integration and delivery build started, and more. I’ll also be writing up another entry, similar to this for AWS and Azure Cloud Providers. If you’d like to see Kubernetes setup and a tour of the setup with Terraform beyond the big three, let me know and I’ll add that to the queue. Once we get past that there are a number of additional Kubernetes, containers, and dev specific posts that I’ll have coming up. Stay tuned, subscribe to the blog feed or follow @ThrashingCode for new blog posts.



Just Another Sunday

I sit here at the moment watching two Kubernetes Clusters build. One is building on Azure and one on Google Cloud Platform (GCP). I’ve got a presentation coming up this Tuesday and Thursday, both I’ll be digging into Kubernetes, Terraform, and a number of other technologies. Those are the two hot technologies for the talks though. Albeit, the continuous integration, languages, and tooling that Terraform builds via configuration and Kubernetes runs in containers is what is actually the meat of this whole sandwich. Which is where I ponder what all of this goo is that wires together things in this virtual programmatic realm in which I’ll build something on top of.

It seems messy from inception. But then of course, all programming and related ecosystem elements in which programming takes place is a messy bag of guts.

Here I sit then, waiting the rather unknown pseudo random amount of time for the Kubernetes Clusters to finish building. A few moments pass and sure enough, as always, very inconsistent build times. The Azure Kubernetes Cluster took 7 minutes to build and the GCP Kubernetes Cluster took just 4 minutes. Last night the Azure cluster was taking 20 minutes or more while the GCP cluster was consuming about 3–4 minutes to build. I’m not sure, as I’ve not dug into the matter deep enough, but something seems awry within the way Azure needs to build out its instances, networking, and related cluster mechanisms. I’m not surprised though, Azure has always behaved and felt slow and cumbersome during the build out of infrastructure. GCP on the other hand clearly comes from Google’s thoroughbred engineering focus on things. It generally builds in a much smaller range of time, consuming much less time overall.

As I build all of this, to work out what will and won’t be in the demo, I find myself next fiddling with presentation material. I really don’t even like to have presentation material, I’d much rather have an interesting enough talk and respective code, samples, and demo to just show the whole thing. Presentation slide decks always fell like, and almost always are, just a crutch for the inability to form ideas, show concepts, or otherwise actually engage the audience around what is being presented. It’s a frustrating dichotomy to say the least. Eventually, with these latest efforts, I actually intend to get down to two slides: one for my information when ending the talk, the requisite contact information and such, then two would be the intro slide with a fancy title for whatever the meat of the talk will be about.

All of this work however is going to be interrupted by the dramatically more important bike ride I’ll take later to clear my thoughts and get the blood flowing through my veins. As things go, I actually dislike sitting still for more than a few hours. I like to chunk my time into brackets, get the work done, and then go for a ride, walk, or something to get my mind cleared back up. I hear it’s healthier for us humans too, but I’ve not set the research to memory to make that argument.

Until later… fini.

Oh the coding stories…

I’ve finally started writing steadily again, after stumbling through more than a few stages of writer’s block. The most recent escapades started with “Quick Start Connections with Terraform and Kubernetes” and “State for Terraform with Google Cloud Storage (GCS)”. But really, there are several types of articles I really need to kick into gear again.

  1. One type of writing is that where I pick apart what’s going on in the industry. This is something I ought to be doing for a number of strategic reasons. Many people should honestly, but it doesn’t happen. We’re often just limited to very specific analysts that may or may not have a solid grasp on what is going on. Since I’m often in the innards of industry projects and efforts, I have some fair insight to apply that isn’t at hands length, it’s from the bloody front lines.
  2. The second type of article is some type of disciplined approach to teach and to learn skills forgotten, skills needed, and hash through things that work into bigger projects.
  3. The third, which is fun and also usually useful are the use case, patterns, and practices articles. I like these, they’re often fun and sometimes even a little controversial.

These all wrap into a steady flow of thoughts, ideas, learning, teaching, and related things. So this is that restart. Against all suggestions to write this type of article, here I’ve done it anyway as a stake in the ground of where I’m officially starting these writing efforts today, Friday the 29th at 8:21pm while at Farley’s Easy in Oakland, California writing while waiting for the arrival of the Coast Starlight. So here’s to writing rhythms and exploring this medium as a medium in which to publish among the mediums! Cheers!