Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation

Part 2 of 3 – Coding Session in Go – Cobra + Viper CLI for Parsing Text Files, Retrieval of Twitter Data, Exports to various file formats, and export to Apache Cassandra.

UPDATED PARTS:

  1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
  2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation (this post)
  3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

Updated links to each part will be posted at bottom of  this post when I publish them. For code, written walk through, and the like scroll down below the video and timestamps.

Hacking Together a CLI Installing Cassandra, Setting Up the Twitter API, ENV Vars, etc.

0:04 Kick ass intro. Just the standard rocking tune.

3:40 A quick recap. Check out the previous write “Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files” of this series.

4:30 Beginning of completion of twitz parse command for exporting out to XML, JSON, and CSV (already did the text export previous session). This segment also includes a number of refactorings to clean up the functions, break out the control structures and make the code more readable.

In the end of refactoring twitz parse came out like this. The completed list is put together by calling the buildTwitterList() function which is actually in the helpers.go file. Then prints that list out as is, and checks to see if a file export should be done. If there is a configuration setting set for file export then that process starts with a call to exportParsedTwitterList(exportFilename string, exportFormat string, ... etc ... ). Then a simple single level control if then else structure to determine which format to export the data to, and a call to the respective export function to do the actual export of data and writing of the file to the underlying system. There’s some more refactoring that could be done, but for now, this is cleaned up pretty nicely considering the splattering of code I started with at first.


var parseCmd = &cobra.Command{
Use: "parse",
Short: "This command will extract the Twitter Accounts form a text file.",
Long: `This command will extract the Twitter Accounts and clean up or disregard other characters
or text around the twitter accounts to create a simple, clean, Twitter Accounts only list.`,
Run: func(cmd *cobra.Command, args []string) {
completedTwittererList := buildTwitterList()
fmt.Println(completedTwittererList)
if viper.Get("fileExport") != nil {
exportParsedTwitterList(viper.GetString("fileExport"), viper.GetString("fileFormat"), completedTwittererList)
}
},
}
func exportParsedTwitterList(exportFilename string, exportFormat string, twittererList []string) {
if exportFormat == "txt" {
exportTxt(exportFilename, twittererList, exportFormat)
} else if exportFormat == "json" {
exportJson(exportFilename, twittererList, exportFormat)
} else if exportFormat == "xml" {
exportXml(exportFilename, twittererList, exportFormat)
} else if exportFormat == "csv" {
exportCsv(exportFilename, twittererList, exportFormat)
} else {
fmt.Println("Export type unsupported.")
}
}
func exportXml(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting xml export to %s.", exportFilename)
xmlContent, err := xml.Marshal(twittererList)
check(err)
header := xml.Header
collectedContent := header + string(xmlContent)
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportCsv(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting txt export to %s.", exportFilename)
collectedContent := rebuildForExport(twittererList, ",")
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportTxt(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting %s export to %s.", exportFormat, exportFilename)
collectedContent := rebuildForExport(twittererList, "\n")
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportJson(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting %s export to %s.", exportFormat, exportFilename)
collectedContent := collectContent(twittererList)
exportFile(string(collectedContent), exportFilename+"."+exportFormat)
}
func collectContent(twittererList []string) []byte {
collectedContent, err := json.Marshal(twittererList)
check(err)
return collectedContent
}
func rebuildForExport(twittererList []string, concat string) string {
var collectedContent string
for _, twitterAccount := range twittererList {
collectedContent = collectedContent + concat + twitterAccount
}
if concat == "," {
collectedContent = strings.TrimLeft(collectedContent, concat)
}
return collectedContent
}
func exportFile(collectedContent string, exportFile string) {
contentBytes := []byte(collectedContent)
err := ioutil.WriteFile(exportFile, contentBytes, 0644)
check(err)
}

view raw

parse.go

hosted with ❤ by GitHub

50:00 I walk through a quick install of an Apache Cassandra single node that I’ll use for development use later. I also show quickly how to start and stop post-installation.

Reference: Apache Cassandra, Download Page, and Installation Instructions.

53:50 Choosing the go-twitter API library for Go. I look at a few real quickly just to insure that is the library I want to use.

Reference: go-twitter library

56:35 At this point I go through how I set a Twitter App within the API interface. This is a key part of the series where I take a look at the consumer keys and access token and access token secrets and where they’re at in the Twitter interface and how one needs to reset them if they just showed the keys on a stream (like I just did, shockers!)

57:55 Here I discuss and show where to setup the environment variables inside of Goland IDE to building and execution of the CLI. Once these are setup they’ll be the main mechanism I use in the IDE to test the CLI as I go through building out further features.

1:00:18 Updating the twitz config command to show the keys that we just added as environment variables. I set these up also with some string parsing and cutting off the end of the secrets so that the whole variable value isn’t shown but just enough to confirm that it is indeed a set configuration or environment variable.

1:16:53 At this point I work through some additional refactoring of functions to clean up some of the code mess that exists. Using Goland’s extract method feature and other tooling I work through several refactoring efforts that clean up the code.

1:23:17 Copying a build configuration in Goland. A handy little thing to know you can do when you have a bunch of build configuration options.

1:37:32 At this part of the video I look at the app-auth example in the code library, but I gotta add the caveat, I run into problems using the exact example. But I work through it and get to the first error messages that anybody would get to pending they’re using the same examples. I get them fixed however in the next session, this segment of the video however provides a basis for my pending PR’s and related work I’ll submit to the repo.

The remainder of the video is trying to figure out what is or isn’t exactly happening with the error.

I’ll include the working findem code in the next post on this series. Until then, watch the wrap up and enjoy!

1:59:20 Wrap up of video and upcoming stream schedule on Twitch.

UPDATED SERIES PARTS

    1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
    2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation (this post)
    3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

 

New Live Coding Streams and Episodes!

I’ve been working away in Valhalla on the next episodes of Thrashing Code TV and subsequent content for upcoming Thrashing Code Sessions on Twitch (follow) and Youtube (subscribe). The following I’ve broken into the main streams and shows that I’ll be putting together over the next days, weeks, and months and links to sessions and shows already recorded. If you’ve got any ideas, questions, thoughts, just send them my way.

Colligere (Next Session)

Coding has been going a little slow, in light of other priorities and all, but it’ll still be one of the featured projects I’ll be working on. Past episodes are available here, however join in on Friday and I’ll catch everybody up, so you can skip past episodes if you aren’t after specific details and just want to join in on future work and sessions.

In this next session, this Friday the 9th at 3:33pm PST, I’m going to be working on reading in JSON, determining what type of structure the JSON should be unmarshalled into, and how best to make that determination through logic and flow.

Since Go needs something to unmarshall JSON into, a specific structure, I’ll be working on determining a good way to pre-read information in the schema configuration files (detailed in the issue listed below) so that a logic flow can be implemented that will then begin the standard Go JSON unmarshalling of the object. So this will likely end up including some hackery around reading in JSON without the assistance of the Go JSON library. Join in and check out what solution(s) I come up with.

The specific issue I’ll be working on is located on Github here. These sessions I’m going to continue working on, but will be a little vague and will start working on the Colligere CLI primarily on Saturday’s at 10am. So you can put that on your schedule and join me then for hacks. If you’d like to contribute, as always, reach out via here, @Adron, or via the Github Colligere Repository and let’s discuss what you’d like to add.

Getting Started with Go

This set of sessions, which I’ve detailed in “Getting Started with Go“, I’ll be starting on January 12th at 4pm PST. You can get the full outline and further details of what I’ll be covering on my “Getting Started with Go” page and of course the first of these sessions I’ve posted details on the Twitch event page here.

  • Packages & the Go Tool – import paths, package declarations, blank imports, naming, and more.
  • Structure – names, declarations, variables, assignments, scope, etc., etc.
  • Basic Types – integers, floats, complex numbers, booleans, strings, and constants.

Infrastructure as Code with Terraform and Apache Cassandra

I’ll be continuing the Terraform, bash, and related configuration and coding of using infrastructure as code practices to build out, maintain, and operate Apache Cassandra distributed database clusters. At some point I’ll likely add Kubernetes, some additional on the metal cluster systems and start looking at Kubernetes Operators and how one can manage distributed systems on Kubernetes using this on the metal environment. But for now, these sessions will continue real soon as we’ve got some systems to build!

Existing episodes of this series you can check out here.

Getting Started with Multi-model Databases

This set of sessions I’ve detailed in “Getting Started with a Multi-model Database“, and this one I’ll be starting in the new years also. Here’s the short run down of the next several streams. So stay tuned, subscribe or follow my Twitch and Youtube and of course subscribe to the Composite Code blog (should be to the left, or if on mobile click the little vertical ellipses button)

  1. An introduction to a range of databases: Apache Cassandra, Postgresql and SQL Server, Neo4j, and … in memory database. Kind of like 7 Databases in 7 Weeks but a bunch of databases in just a short session!
  2. An Introduction – Apache Cassandra and what it is, how to get a minimal cluster started, options for deploying something quickly to try it out.
  3. Adding to Apache Cassandra with DataStax Enterprise, gaining analytics, graph, and search. In this session I’ll dive into what each of these capabilities within DataStax Enterprise give us and how the architecture all fits together.
  4. Deployment of Apache Cassandra and getting a cluster built. Options around ways to effectively deploy and maintain Apache Cassandra in production.
  5. Moving to DataStax Enterprise (DSE) from Apache Cassandra. Getting a DSE Cluster up and running with OpsCenter, Lifecycle Manager (LCM), and getting some queries tried out with Studio.

New Trio of Series: Go, Bash, and Distributed Databases

I’m starting three new series and only making this post now that I’ve got the first round posted and live. Each of these I’ll be back posting so that I have a kind of linked list of blog entries for each of the series. So if you’re into learning Go, I’ve got that going, or you’re into bash hacking, got that too, and of course if you’re digging into distributed systems and databases, I’m tackling that too. It’s kind of the trio of core technologies I’m working on these days, enjoy:

The Nuances of Go – This is going to be a series where I go through some of the details of Go. It’s going to be kind of all over the board, but drill into usage of the language, the why, and what for of various features, capabilities, and related topics.

‘bash’ A.K.A. The Solution for Everything – Bash has been around for a while. But let’s not talk about how old it is, the shell has been used and is being used by about every single operating system on the planet. It’s hugely popular and it isn’t exactly being replaced. You can also basically do anything on a computer that you would want to or need to do with it. However there are lot’s of features and commands one ought to know, this series is going to tackle a new command every new post and go into details of how to use it, what it can be used for, and related tips n’ tricks.

Distributed Database Things to Know – This series is going to cover various features and nuances of the Cassandra distributed database cluster technology. I’ll be diving into a whole host of capabilities, code, and a pointer or three back to white papers when relevant that helped bring these distributed databases into existence.

My goal with each of these is to rise early Monday morning, and time box each article to about 30 minutes. 30 minutes for Go, 30 minutes for bash, and 30 for Cassandra. Then I’ll publish those throughout the week. I set this goal with the hilarious fact that I’m taking time to go record some LinkedIn learning courses on Go and Terraform. With that it might be a week before you see the next trio published. But they’re halfway written already so I might surprise myself. Happy thrashing code!

A Go Repo & Some Go Code

Wrapped up another Twitch stream (follow here) and streamed live via YouTube where the video is now (subscribe here, video here). However the more interesting part IMHO was where I broke down a few key parts of the application building features around file writes, reads, JSON marshalling, and some other functionality. I also put together a repo of the code I put together here on Github (code shown and explained below). I dove into this topic in the later part of the stream which I’ve time tagged below. For the full timeline and the rest of the video just watch from the beginning of the video. I do make some progress working on Colligere, but I’ll cover that topic in a subsequent blog entry and Twitch stream.

  • 1:32:40 – Creating a new application with Jetbrains Goland to show off how to use the Go core libraries around the user, JSON, and some basic file creation and writing.
  • 1:36:40 – At this point I start adding some basic code to pull the current user and collect some information about that user.
  • 1:40:42 – I extract the error code using the Goland refactoring feature and setup a func check() for error checking. That cleans up the inline code a bit.
  • 1:43:10 – Here I add to the user data retrieved some environment variables to that list of collected data. I also cover again, as I have a number of times, how the environment variables are pulled in IDE versus user session versus out of IDE.
  • 1:47:46 – Now I add a file exists check and start working on that logic.
  • 1:48:56 – Props to Edd Turtle on a solid site on Go. Here’s the blog entry, and respectively the path to more of Edd’s material @ https://golangcode.com/. Also, Edd seems like a good guy to follow @eddturtle.
  • 2:01:33 – Starting the JSON Work here to marshal and unmarshall.
  • 2:26:33 – At this point I push the code up to Github (repo here) using the built in Goland VCS features. I realize I’ve named the repo “adron” by accident so I rename it, close Goland, and then clone the code back down locally with Goland’s VCS features. It’s kind of interesting to see Goland go through the 2-factor auth for this too.
  • 2:58:56 – The Seattle Thrashing Code outtro!

There were a few notes I took during the session with my collected links and references for things I looked up. Those included the following:

The code for the Github repo, ended in this state. Just a single main.go file, which shows how to use several features of Go and capabilities of the core libraries.

[sourcecode language=”javascript”]
package main

import (
“encoding/json”
“io/ioutil”
“log”
“os”
“os/user”
“strconv”
)

type UserInformation struct {
Name string `json:”name”`
UserId int64 `json:”userid”`
GroupId int64 `json:”groupid”`
HomeDir string `json:”homedir”`
UserName string `json:”username”`
GroupIds []string `json:”groupids”`
GoPath string `json:”gopath”`
EnvVar string `json:”environmentvariable”`
}

func main() {
currentUser, err := user.Current()
check(err)
changeDataForMarshalling(currentUser)
}

func changeDataForMarshalling(currentUser *user.User) {
groupsIds, err := currentUser.GroupIds()
check(err)
userId, err := strconv.ParseInt(currentUser.Uid, 6, 64)
check(err)
groupId, err := strconv.ParseInt(currentUser.Gid, 6, 64)
check(err)

workingUserInformation := &UserInformation{
Name: currentUser.Name,
UserId: userId,
GroupId: groupId,
HomeDir: currentUser.HomeDir,
UserName: currentUser.Username,
GroupIds: groupsIds,
GoPath: os.Getenv(“GOPATH”),
EnvVar: os.Getenv(“NEW_STRING_ENVIRONMENT_VARIABLE”),
}

resultingUserInformation, _ := json.Marshal(workingUserInformation)

filename := “collected_values.json”
if _, err := os.Stat(filename); os.IsNotExist(err) {
writeFileContents(err, filename, string(resultingUserInformation))
} else {
newUserInformation, err := openFileMakeChanges(filename)
writeFileContents(err, filename, newUserInformation)
}
}

func openFileMakeChanges(filename string) (string, error) {
jsonFile, err := os.Open(filename)
check(err)
defer jsonFile.Close()
var changingUserInformation UserInformation
byteValue, _ := ioutil.ReadAll(jsonFile)
json.Unmarshal(byteValue, &changingUserInformation)
changingUserInformation.Name = “Adron Hall”
changingUserInformation.GoPath = “/where/is/the/goland”
newUserInformation, _ := json.Marshal(changingUserInformation)
return string(newUserInformation), err
}

func writeFileContents(err error, filename string, text string) {
f, err := os.OpenFile(filename, os.O_CREATE|os.O_WRONLY, 0600)
check(err)
defer f.Close()

if _, err = f.WriteString(text); err != nil {
panic(err)
}
}

func check(err error) {
if err != nil {
log.Fatal(err)
}
}
[/sourcecode]

The import section includes several core libraries for the application “encoding/json”, “io/ioutil”, “log”, “os”, “os/user”, and “strconv”. Then I’ve got a structure declared that I use throughout the code with various fields.

Then just for the heck of it I created a changeDataForMarshalling function and passed in userInfo and parsed the string results of gid and uid to int data types. From there the remaining values are passed in then marshalled to JSON and written to a file, depending on if it’s a new file or existing. The writing however I broke out to a function dubbed writeFileContents. Definitely some more refactoring and tweaking to really make it functional and usable, but shows easily what strconv, marshalling, and other features of these core libraries do. This code provides some examples on what functionality Colligere will use to read in schema configuration, edit, and save that schema. More about that in the next Twitch few streams.

The Nuances of Go: Go Program Structure

Talking About Types, Variables, Pointers, and More

In my endeavor to get more of the specifics of Go figured out, I’ve been diving a bit more in depth to each of the specific features, capabilities, and characteristics of the language. I’ll be doing the same for Cassandra over the coming months and producing a blog entry series about it. This new series however is about Go, and this first in the series is about types, variables, pointers, and naming of these things. Enjoy.

Reserved Words, Keywords & Related Systemic Parts of Go

First off let’s talk about the reserved words in Go. The list includes 25 keywords that can’t be used as variables or names of functions or anything like that. This list includes:

if import interface map
select return range package
struct switch type var
for func go goto
default defer else fallthrough
case const chan continue
break

Beyond these 25 keywords there are also some predeclared names that can be but usually shouldn’t be used for naming. These include:

Constants

true false iota nil

Types

int int8 int16 int32
int64 uint uint8 uint16
uint32 uint64 uintptr float32
float64 complex128 complex64 bool
byte rune string error

Functions

make len cap new
append copy close delete
complex real imag panic
recover

When declaring an entity within a function, it’s scope is limited to that function. When declaring it outside of the function it is available to the contents of the package it is declared inside of. The case of the first letter also has the effect of allowing access within or across package boundaries. If the name begins with an upper case character it is exported for other packages. Package names themselves are always lower case however.

Camel Casing

It’s also important to know about the emphasis and standard within Go to use camel casing. Except it appears that for acronyms and such like; HTML, XML, JSON, or such things, it is routine to use names like HTMLthingy, XMLjunkHere, or rockStarJSON. Go is somewhat particular about these types of things, so it is extremely important to use the Go format standards. To help with this, there’s also the gofmt function, but it’s always good to just use the proper formatting during writing of the code anyway.

The Four Declarations

The four declarations are kind of like the four horsement, except not really. Forget that analogy as it was already going nowhere. The four declarations include:

var const type func

Here are some code samples of various declarations. First use of the func. In any new Go program there’s always the main() function. Declared like this.

[sourcecode language=”javascript”]
func main() {
// Stuff goes here.
}
[/sourcecode]

Another example of a function declaration with parameters passed and expected results to return. The signature works like func followed by the name of the function, then in parenthesis parameter name followed by its type, for as may parameters as is needed. Then after the parenthesis the type of the expected result of the function. Of course, if no result is expected to be returned, then there would be no tpe specified here. Just as if no parameters, the parenthesis would still be used but no parameters specified.

[sourcecode language=”javascript”]
func doSomeComputation(someValue float64, anotherFloaty float64) float64 {
return someValue * anotherFloat
}
[/sourcecode]

Using the var for declaration of variables looks like this.

[sourcecode language=”javascript”]
var f = 212.0
var result = 2 + 2
var celsius = (f – 32) * 5 / 9
[/sourcecode]

Variable types can be defined in this way. Using var keyword, followed by the variable name, then followed by its type, and then if desired assigned via = and then the specific value or function with a result to assign to the variable.

var name type = expression

Without an assignment, the variable is simply created with the intent of assigning a value later. Such as these variable declarations.

[sourcecode language=”javascript”]
var f float32
var result float64
var celsius float64
var someVariable int
var junk string
[/sourcecode]

A shorter version of the creation and assignment of variables looks like this. It however can only be used within a function.

name := expression

Actual use of this shorter version would look like this.

[sourcecode language=”javascript”]
f := 212.0
randomStuff := rand.Float64()
wordContent := “This is some character string nonsense.”
[/sourcecode]

One can also do some other tricks with this shorter method of assignment. Let’s say some variables are needed that will have integers assigned to them. You could set up the variables with values like this. I’ve noticed some get a bit confused about this and tuple assignments, this is not the same thing. This merely gives you three variables: k, l, and m and assigns the values 0, 1, and 2 to the respective variable.

[sourcecode language=”javascript”]
k, l, m := 0, 1, 2
[/sourcecode]

It’s also important to note, a short declaration doesn’t always declare all of the variables on the left-hand side. Some, if already declared in the same lexical block, it then acts as an assignment to the variables instead.

Declaring a constant value with const would look like this.

[sourcecode language=”javascript”]
const fahrenheitBoiling = 212.0
[/sourcecode]

Pointers

A variable holds a particular value, where as a pointer holds the address in memory of a particular value. Sometimes, it gets a bit difficult to mentally keep track of pointers, so let’s look at a few examples. First I’ll start with a plain old variable.

[sourcecode language=”javascript”]
someVariable := “The original value of this variable.”
[/sourcecode]

Then I’ll add another variable, which will point at the address and respective value of someVariable.

[sourcecode language=”javascript”]
someVariable := “The original value of this variable.”
anotherVariable := &someVariable
fmt.Println(*anotherVariable)
[/sourcecode]

In this situation what does the fmt.Println(*anotherVariable) function print out? The result would be “The original value of this variable.” because anotherVariable is a variable that points to the value of someVariable. Which means anotherVariable doesn’t take the resources that someVariable does, but merely points to it.

The reasoning here is, if you have variable for example that stores a very large value in memory. One can merely point to that data with a pointer and the large value doesn’t have to change it’s memory space. Imagine if that variable were a gigabyte of data and it had to be moved every time something needed referenced or calculated with it, with a pointer we don’t have to move it. It can be used in various ways without it getting shifted around.

Now if I take the pointer value anotherVariable and assign it a value, like this.

[sourcecode language=”javascript”]
*anotherVariable = “42”
[/sourcecode]

Then what happens? The pointer now is re-allocated and assigned the new value of “42”. It no longer references the address space of the pointer and instead now has the value 42 assigned to the variable.

The Fancy new Function

There’s also another function, a built-in one, that creates a variable. Using the expression new(type) will create an unnamed variable of type, initializes it, and returns the address of the value of type. In pointer speak, it’d be *type. Here are some examples.

[sourcecode language=”javascript”]
pinto := new(int)
fmt.Println(*pinto)
[/sourcecode]

This code will create a variable of type int, then initialize the value to zero since it is an int. Let’s say we want to actually give it a value after it’s created though. We could assign pinto like this.

[sourcecode language=”javascript”]
*pinto = 42
fmt.Println(*p)
[/sourcecode]

This will then print out the value 42, of type integer. Note, this new is just sugar, a syntactical convenience feature really. They can be useful however, and there are a few caveats that need to be noted. For example, check out this gotcha.

If I declare two new values using new.

[sourcecode language=”javascript”]
firstThing := new(int)
secondThing := new(int)
fmt.Println(firstThing == secondThing)
[/sourcecode]

In this case, the firstThing and secondThing don’t actually match, because the comparison is on the address space, and they’re different address spaces!

Alright, that’s it for now, just a few important nuances around how variables, pointers, and Go features work.