Tag Archives: go

New Trio of Series: Go, Bash, and Distributed Databases

I’m starting three new series and only making this post now that I’ve got the first round posted and live. Each of these I’ll be back posting so that I have a kind of linked list of blog entries for each of the series. So if you’re into learning Go, I’ve got that going, or you’re into bash hacking, got that too, and of course if you’re digging into distributed systems and databases, I’m tackling that too. It’s kind of the trio of core technologies I’m working on these days, enjoy:

The Nuances of Go – This is going to be a series where I go through some of the details of Go. It’s going to be kind of all over the board, but drill into usage of the language, the why, and what for of various features, capabilities, and related topics.

‘bash’ A.K.A. The Solution for Everything – Bash has been around for a while. But let’s not talk about how old it is, the shell has been used and is being used by about every single operating system on the planet. It’s hugely popular and it isn’t exactly being replaced. You can also basically do anything on a computer that you would want to or need to do with it. However there are lot’s of features and commands one ought to know, this series is going to tackle a new command every new post and go into details of how to use it, what it can be used for, and related tips n’ tricks.

Distributed Database Things to Know – This series is going to cover various features and nuances of the Cassandra distributed database cluster technology. I’ll be diving into a whole host of capabilities, code, and a pointer or three back to white papers when relevant that helped bring these distributed databases into existence.

My goal with each of these is to rise early Monday morning, and time box each article to about 30 minutes. 30 minutes for Go, 30 minutes for bash, and 30 for Cassandra. Then I’ll publish those throughout the week. I set this goal with the hilarious fact that I’m taking time to go record some LinkedIn learning courses on Go and Terraform. With that it might be a week before you see the next trio published. But they’re halfway written already so I might surprise myself. Happy thrashing code!

A Go Repo & Some Go Code

Wrapped up another Twitch stream (follow here) and streamed live via YouTube where the video is now (subscribe here, video here). However the more interesting part IMHO was where I broke down a few key parts of the application building features around file writes, reads, JSON marshalling, and some other functionality. I also put together a repo of the code I put together here on Github (code shown and explained below). I dove into this topic in the later part of the stream which I’ve time tagged below. For the full timeline and the rest of the video just watch from the beginning of the video. I do make some progress working on Colligere, but I’ll cover that topic in a subsequent blog entry and Twitch stream.

  • 1:32:40 – Creating a new application with Jetbrains Goland to show off how to use the Go core libraries around the user, JSON, and some basic file creation and writing.
  • 1:36:40 – At this point I start adding some basic code to pull the current user and collect some information about that user.
  • 1:40:42 – I extract the error code using the Goland refactoring feature and setup a func check() for error checking. That cleans up the inline code a bit.
  • 1:43:10 – Here I add to the user data retrieved some environment variables to that list of collected data. I also cover again, as I have a number of times, how the environment variables are pulled in IDE versus user session versus out of IDE.
  • 1:47:46 – Now I add a file exists check and start working on that logic.
  • 1:48:56 – Props to Edd Turtle on a solid site on Go. Here’s the blog entry, and respectively the path to more of Edd’s material @ https://golangcode.com/. Also, Edd seems like a good guy to follow @eddturtle.
  • 2:01:33 – Starting the JSON Work here to marshal and unmarshall.
  • 2:26:33 – At this point I push the code up to Github (repo here) using the built in Goland VCS features. I realize I’ve named the repo “adron” by accident so I rename it, close Goland, and then clone the code back down locally with Goland’s VCS features. It’s kind of interesting to see Goland go through the 2-factor auth for this too.
  • 2:58:56 – The Seattle Thrashing Code outtro!

There were a few notes I took during the session with my collected links and references for things I looked up. Those included the following:

The code for the Github repo, ended in this state. Just a single main.go file, which shows how to use several features of Go and capabilities of the core libraries.

package main

import (
	"encoding/json"
	"io/ioutil"
	"log"
	"os"
	"os/user"
	"strconv"
)

type UserInformation struct {
	Name     string   `json:"name"`
	UserId   int64    `json:"userid"`
	GroupId  int64    `json:"groupid"`
	HomeDir  string   `json:"homedir"`
	UserName string   `json:"username"`
	GroupIds []string `json:"groupids"`
	GoPath   string   `json:"gopath"`
	EnvVar   string   `json:"environmentvariable"`
}

func main() {
	currentUser, err := user.Current()
	check(err)
	changeDataForMarshalling(currentUser)
}

func changeDataForMarshalling(currentUser *user.User) {
	groupsIds, err := currentUser.GroupIds()
	check(err)
	userId, err := strconv.ParseInt(currentUser.Uid, 6, 64)
	check(err)
	groupId, err := strconv.ParseInt(currentUser.Gid, 6, 64)
	check(err)

	workingUserInformation := &UserInformation{
		Name:     currentUser.Name,
		UserId:   userId,
		GroupId:  groupId,
		HomeDir:  currentUser.HomeDir,
		UserName: currentUser.Username,
		GroupIds: groupsIds,
		GoPath:   os.Getenv("GOPATH"),
		EnvVar:   os.Getenv("NEW_STRING_ENVIRONMENT_VARIABLE"),
	}

	resultingUserInformation, _ := json.Marshal(workingUserInformation)

	filename := "collected_values.json"
	if _, err := os.Stat(filename); os.IsNotExist(err) {
		writeFileContents(err, filename, string(resultingUserInformation))
	} else {
		newUserInformation, err := openFileMakeChanges(filename)
		writeFileContents(err, filename, newUserInformation)
	}
}

func openFileMakeChanges(filename string) (string, error) {
	jsonFile, err := os.Open(filename)
	check(err)
	defer jsonFile.Close()
var changingUserInformation UserInformation
	byteValue, _ := ioutil.ReadAll(jsonFile)
	json.Unmarshal(byteValue, &changingUserInformation)
changingUserInformation.Name = "Adron Hall"
	changingUserInformation.GoPath = "/where/is/the/goland"
	newUserInformation, _ := json.Marshal(changingUserInformation)
return string(newUserInformation), err
}

func writeFileContents(err error, filename string, text string) {
	f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, 0600)
	check(err)
	defer f.Close()

	if _, err = f.WriteString(text); err != nil {
		panic(err)
	}
}

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

The import section includes several core libraries for the application “encoding/json”, “io/ioutil”, “log”, “os”, “os/user”, and “strconv”. Then I’ve got a structure declared that I use throughout the code with various fields.

Then just for the heck of it I created a changeDataForMarshalling function and passed in userInfo and parsed the string results of gid and uid to int data types. From there the remaining values are passed in then marshalled to JSON and written to a file, depending on if it’s a new file or existing. The writing however I broke out to a function dubbed writeFileContents. Definitely some more refactoring and tweaking to really make it functional and usable, but shows easily what strconv, marshalling, and other features of these core libraries do. This code provides some examples on what functionality Colligere will use to read in schema configuration, edit, and save that schema. More about that in the next Twitch few streams.

The Nuances of Go: Go Program Structure

Talking About Types, Variables, Pointers, and More

In my endeavor to get more of the specifics of Go figured out, I’ve been diving a bit more in depth to each of the specific features, capabilities, and characteristics of the language. I’ll be doing the same for Cassandra over the coming months and producing a blog entry series about it. This new series however is about Go, and this first in the series is about types, variables, pointers, and naming of these things. Enjoy.

Reserved Words, Keywords & Related Systemic Parts of Go

First off let’s talk about the reserved words in Go. The list includes 25 keywords that can’t be used as variables or names of functions or anything like that. This list includes:

if import interface map
select return range package
struct switch type var
for func go goto
default defer else fallthrough
case const chan continue
break

Beyond these 25 keywords there are also some predeclared names that can be but usually shouldn’t be used for naming. These include:

Constants

true false iota nil

Types

int int8 int16 int32
int64 uint uint8 uint16
uint32 uint64 uintptr float32
float64 complex128 complex64 bool
byte rune string error

Functions

make len cap new
append copy close delete
complex real imag panic
recover

When declaring an entity within a function, it’s scope is limited to that function. When declaring it outside of the function it is available to the contents of the package it is declared inside of. The case of the first letter also has the effect of allowing access within or across package boundaries. If the name begins with an upper case character it is exported for other packages. Package names themselves are always lower case however.

Camel Casing

It’s also important to know about the emphasis and standard within Go to use camel casing. Except it appears that for acronyms and such like; HTML, XML, JSON, or such things, it is routine to use names like HTMLthingy, XMLjunkHere, or rockStarJSON. Go is somewhat particular about these types of things, so it is extremely important to use the Go format standards. To help with this, there’s also the gofmt function, but it’s always good to just use the proper formatting during writing of the code anyway.

The Four Declarations

The four declarations are kind of like the four horsement, except not really. Forget that analogy as it was already going nowhere. The four declarations include:

var const type func

Here are some code samples of various declarations. First use of the func. In any new Go program there’s always the main() function. Declared like this.

func main() {
    // Stuff goes here.
}

Another example of a function declaration with parameters passed and expected results to return. The signature works like func followed by the name of the function, then in parenthesis parameter name followed by its type, for as may parameters as is needed. Then after the parenthesis the type of the expected result of the function. Of course, if no result is expected to be returned, then there would be no tpe specified here. Just as if no parameters, the parenthesis would still be used but no parameters specified.

func doSomeComputation(someValue float64, anotherFloaty float64) float64 {
    return someValue * anotherFloat
}

Using the var for declaration of variables looks like this.

var f = 212.0
var result = 2 + 2
var celsius = (f - 32) * 5 / 9

Variable types can be defined in this way. Using var keyword, followed by the variable name, then followed by its type, and then if desired assigned via = and then the specific value or function with a result to assign to the variable.

var name type = expression

Without an assignment, the variable is simply created with the intent of assigning a value later. Such as these variable declarations.

var f float32
var result float64
var celsius float64
var someVariable int
var junk string

A shorter version of the creation and assignment of variables looks like this. It however can only be used within a function.

name := expression

Actual use of this shorter version would look like this.

f := 212.0
randomStuff := rand.Float64()
wordContent := "This is some character string nonsense."

One can also do some other tricks with this shorter method of assignment. Let’s say some variables are needed that will have integers assigned to them. You could set up the variables with values like this. I’ve noticed some get a bit confused about this and tuple assignments, this is not the same thing. This merely gives you three variables: k, l, and m and assigns the values 0, 1, and 2 to the respective variable.

k, l, m := 0, 1, 2

It’s also important to note, a short declaration doesn’t always declare all of the variables on the left-hand side. Some, if already declared in the same lexical block, it then acts as an assignment to the variables instead.

Declaring a constant value with const would look like this.

const fahrenheitBoiling = 212.0

Pointers

A variable holds a particular value, where as a pointer holds the address in memory of a particular value. Sometimes, it gets a bit difficult to mentally keep track of pointers, so let’s look at a few examples. First I’ll start with a plain old variable.

someVariable := "The original value of this variable."

Then I’ll add another variable, which will point at the address and respective value of someVariable.

someVariable := "The original value of this variable."
anotherVariable := &someVariable
fmt.Println(*anotherVariable)

In this situation what does the fmt.Println(*anotherVariable) function print out? The result would be “The original value of this variable.” because anotherVariable is a variable that points to the value of someVariable. Which means anotherVariable doesn’t take the resources that someVariable does, but merely points to it.

The reasoning here is, if you have variable for example that stores a very large value in memory. One can merely point to that data with a pointer and the large value doesn’t have to change it’s memory space. Imagine if that variable were a gigabyte of data and it had to be moved every time something needed referenced or calculated with it, with a pointer we don’t have to move it. It can be used in various ways without it getting shifted around.

Now if I take the pointer value anotherVariable and assign it a value, like this.

*anotherVariable = "42"

Then what happens? The pointer now is re-allocated and assigned the new value of “42”. It no longer references the address space of the pointer and instead now has the value 42 assigned to the variable.

The Fancy new Function

There’s also another function, a built-in one, that creates a variable. Using the expression new(type) will create an unnamed variable of type, initializes it, and returns the address of the value of type. In pointer speak, it’d be *type. Here are some examples.

pinto := new(int)
fmt.Println(*pinto)

This code will create a variable of type int, then initialize the value to zero since it is an int. Let’s say we want to actually give it a value after it’s created though. We could assign pinto like this.

*pinto = 42
fmt.Println(*p)

This will then print out the value 42, of type integer. Note, this new is just sugar, a syntactical convenience feature really. They can be useful however, and there are a few caveats that need to be noted. For example, check out this gotcha.

If I declare two new values using new.

firstThing := new(int)
secondThing := new(int)
fmt.Println(firstThing == secondThing)

In this case, the firstThing and secondThing don’t actually match, because the comparison is on the address space, and they’re different address spaces!

Alright, that’s it for now, just a few important nuances around how variables, pointers, and Go features work.

Thrashing Code Twitch Schedule September 19th-October 2nd

I’ve got everything queued back up with some extra Thrashing Code Sessions and will have some on the rails travel streams. Here’s what the schedule looks like so far.

Today at 3pm PST (UPDATED: Sep 19th 2018)

thrashing-code-terraformUPDATED: Video available https://youtu.be/NmlzGKUnln4

I’m going to get back into the roll of things this session after the travels last week. In this session I’m aiming to do several things:

  1. Complete next steps toward getting a DataStax Enterprise Apache Cassandra cluster up and running via Terraform in Google Cloud Platform. My estimate is I’ll get to the point that I’ll have three instances that launch and will automate the installation of Cassandra on the three instances. Later I’ll aim to expand this, but for now I’m just going to deploy 3 nodes and then take it from there. Another future option is to bake the installation into a Packer deployed image and use it for the Terraform execution. Tune in to find out the steps and what I decide to go with.
  2. I’m going to pull up the InteroperabilityBlackBox and start to flesh out some objects for our model. The idea, is based around something I stumbled into last week during travels, the thread on that is here.

Friday (Today) @ 3pm PST

thrashing-code-gopherThis Friday I’m aiming to cover some Go basics before moving further into the Colligere CLI  app. Here are the highlights of the plan.

  1.  I’m going to cover some of the topics around program structure including: type declarations, tuple assignment, variable lifetime, pointers, and other variables.
  2.  I’m going to cover some basics on packages, initialization of packages, imports, and scope. This is an important aspect of ongoing development with Colligere since we’ll be pulling in a number of packages for generation of the data.
  3. Setting up configuration and schema for the Colligere application using Viper and related tooling.

Tuesday, October 2nd @ 3pm PST

thrashing-code-terraformThis session I’m aiming to get some more Terraform work done around the spin up and shutdown of the cluster. I’ll dig into some more specific points depending on where I progress to in sessions previous to this one. But it’s on the schedule, so I’ll update this one in the coming days.

 

Thrashing Code Sessions via Twitch & Kick Ass Dis-Sys Meetup

Got some excellent coding and systems setup coming up in the next few days. Also a meetup on the 28th with Tim Kellogg and Alena Hall presenting on some interesting topics around distributed database data working on Kubernetes and WebAssembly of the hot temperament type. A new surprise guest addition on my Twitch channel that is scheduled to swing into Valhalla and help build out a cluster and respective needed DHCP, DNS, and related configuration for a setup on the metal!

Schedule

 

Collecting Terraform Resources

I just finished and got a LinkedIn Learning (AKA Lynda.com) course published last month on Learning Terraform (LinkedIn Learning Course & Lynda.com Course). Immediately after posting that I spoke with my editor at LinkedIn Learning and agreed on the next two courses I’ll record: Terraform Essentials and Go for Site Reliability Engineers. Consider me stoked to be putting this material together and recording more video courses, this is a solid win, as the internet dogge would say, “much excite, very wow”!

The following are some recent materials I’ve dug up in regards to Terraform, Go, and Site Reliability work. Some of which will very likely find it’s way into influencing my courses. Good material here if you’re looking for some solid, and arguably more advanced approaches, to your Terraform work.

Advanced Terraform Materials

The HashiCorp Documentation Material

Writing customer providers

Running Terraform in Automation

Go Library Data Generation Timings

Recently I put together some quick code to give some timings on the various data generation libraries available for Go. For each library there were a few key pieces of data generation I wanted to time:

  • First Name – basically a first name of some sort, like Adam, Nancy, or Frank.
  • Full Name – something like Jason McCormick or Sally Smith.
  • Address – A basic street address, or whatever the generator might provide.
  • User Agent – Such as that which is sent along with the browser response.
  • Color – Something like red, blue, green, or other color beyond the basics.
  • Email – A fully formed, albeit faked email address.
  • Phone – A phone number, ideally with area code and prefix too.
  • Credit Card Number – Ideally a properly formed one, which many of the generators seem to provide based on VISA, Mastercard, or related company specifications.
  • Sentence – A stand multi-word lorem ipsum based sentence would be perfect.

I went through and searched for libraries that I wanted to try out. Of all the libraries I found I narrowed it down to three specific libraries. When I add the imports for these libraries, by way of how Go works, it gives you the repo locations:

  • “github.com/bxcodec/faker” – faker – Faker generates data based on a Struct, which is a pretty cool way to determine what type of data you want and to get it returned in a particularly useful format.
  • “github.com/icrowley/fake” – fake – Fake is a library inspired by the ffaker and forgery Ruby gems. Not that you’d be familiar with those, but if you are you have instant insight into how this library works.
  • “github.com/malisit/kolpa” – kolpa – This is another data generator that creates fake data for various types of data, structures, strings, sentences, and more.

Continue reading