Tag: go

Language Stack Installation for Python & Go

Previously I’ve gone through the steps I take to get a solid development machine setup. From the base operating system load, to the browser and basic IDEs I install. Now I’ve got more videos and the respective notes and details about what two language stacks I setup next; Python and Go.

Python

In regards to the Python stack, this one can often be somewhat confusing. Depending on the operating system I setup the stack a little differently.

MacOS

For MacOS I’ve written two posts about this previously, one titled “Getting Started with Python Right!” and one “Unbreaking Python Through Virtual Environments“. Those two posts cover most of the nuance to getting a base Python stack installed on MacOS and then using virtual environments to manage project specific versions per repository.

Linux

For the Linux OS, usually a debian variant, the systems tend to have Python 3 installed by default. I then take the next step of installing pip3 and work from there. The IDE, PyCharm from Jetbrains uses virtualenv to setup virtual environments per repository from that point forward.

Python Setup && Reasons

For more details about the specific walk through, I’ve created this video to walk through setting up Python 3 on Ubuntu and verifying, and also by use of PyCharm to setup a small verification app it shows how the virtualenv sets up a specific environment for the new verification project.

The reasons for installing Python first are numerous. One of the first reasons is that Python is required for installing and using numerous Python related CLIs, such as AWS’s CLI, among many others. It pays off to just have a good install at the system level (i.e. not particular just in a virutal environment, but executable at the terminal on system) to ensure it is available for any and all CLIs that would need it. If you’re into data science work, that’s a huge second reason, because Python is used in about every aspect of data science work, machine learning, and related endeavors.

Go Setup && Reasons

The reason I go for Go as my second language stack install is driven by two primary reasons:

  1. I like writing Go and use it myself for a number of reasons. Such as, it is ridiculously quick and minimal work to build a CLI for use in systems that requires only a single binary executable for use.
  2. I use Go for a lot of other work-related efforts, around Kubernetes, Docker, Terraform, and others.

With that, here’s the quick install and initial project for verification setup.

If you’d like to take a few other quick tours of Go, here are some posts, with videos, putting together a Go module project and writing an initial test in under 3 minutes and setting up an HTTP server in about 15 minutes.

That’s it for now. However, if you’re interested in joining me for next steps, language stack setup, and more in addition to writing some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and all sorts of coding I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!

For more on my open source efforts and related projects, sign up for the Thrashing Code Newsletter!

My Top 2 Reading List for Go & Data

Right now I’ve got a couple books in queue.

“Black Hat Go” Go Programming for Hackers and Pentesters by Tom Steele, Chris Patten, and Dan Kottmann.

Alt Text

I picked this book up after a good look at the index. There is a basic intro to Go at the beginning of the book to cover language fundamentals, but immediately after that dives into the meat of the topic. Immediately getting into understanding the TCP handshake, TCP itself, writing a scanner, and a proxy. There is then some basics about HTTP Servers, routers, and middleware in chapter 3 and 4, but returns immediately into topics of interest in chapter 5 around exploiting DNS, and writing DNS clients. I perused the index a bit further to note it covers SMB and NTLM, a chapter on “Abusing Databases and Filesystems”, then on to raw packet processing, writing and porting exploit code, and a host of other topics. This is going to be an interesting book to dig into and write about in the coming weeks. I’m pretty excited about it and hope to write a thorough review upon completion.

“Designing Data-Intensive Applications” by Martin Kleppmann

Alt Text

This book is familiar territory, as I’ve spent much of my career working on similar topics that this book covers. What I suspect is that I’ll enjoy reading the material presented in an orderly and concise way versus the chaos and disruptive way I’ve acquired the knowledge on these topics.

From the index, the book starts off with foundations of data systems and the ideas around building reliable, scalable, and maintainable applications. This provides a good basis in which to dive into the other topics. From there it looks like we’ll get a run into the birth of NoSQL, object-relational mismatches and the related insanity that this has bred in the industry! Then a solid dive into graph-like, traditional, and multi-model data modeling. With the beginning quarter of the book then covering everything from hash indexes, SSTables (another familiar topic), LSM-Trees, B-Trees, and related indexing structures before wrapping up this first 25% of the book with stars and snowflake topics for analytics and column-oriented storage, compression, sort orders in column storage, and related material on aggregation in data cubes and materialized views.

That’s just the first 25%! From there Martin covers a wide range of topics, that if you’re in the industry and plan to deal with large scale data-intensive applications these are topics you need to be intimately familiar with!

Reading

Over the next few months while I read through these books I hope to provide summaries and related notes on the material. Who knows, maybe you’ll want to dive into the material yourself! Until then happy thrashing code and may you have high retention and comprehension in reading!

Cobra + Viper Go CLI Sample

I just released a Go Cobra + Viper library CLI App sample app here. Here are the main up to date details as of release today.

The app example is using the Cobra and Viper libraries for Go. This application provides commands (per Cobra) to read, write, update, and delete configuration records from a configuration file using Viper.

Using the example CLI App.

<cli> config -h provides the following documentation for using this CLI App.

Available Commands:
  add         The 'add' subcommand will add a passed in key value pair to the application configuration file.
  delete      The 'delete' subcommand removes a key value pair from the configuration file. 
  update      The 'update' subcommand will update a passed in key value pair for an existing set of data to the application configuration file.
  view        The 'view' subcommand will provide a list of keys and a map of the values.

Flags:
  -h, --help           help for config
  -k, --key string     The key for the key value set to add to the configuration.
  -v, --value string   The value for the key value set to add to the configuration.

Currently I got this to work but it isn’t in the best shape. Check out the code here.

Get Involved, Add Samples, Make Requests

If you’d like to get involved and add samples or make request for additional samples please file an issue here!

If you’ve found any bugs or issues with the code please file a bug report here!

Examples, The CRUD!

./cli config add -k "blog" -v "https://compositecode.blog/" example writes a record to the configuration file with a key of “blog” and a value of “https://compositecode.blog/“.

./cli config view displays the contents of the configuration file and CLI specific environment variables. These are the configuration files located in the .cobrae-cli-samples.yml and environment variables prefaced with COBRACLISAMPLES. The list of keys is displaced first and then the keys and values are displayed below that.

./cli config update -k "blog" -v "not found" will update the blog entry in the configuration to read not found for the value.

./cli config delete ... will delete the key and value from the configuration file.

Building the Project

Following a fairly standard clone, one can build this project with a single step using the ./build.sh file. If you’d like to contribute the same for Windows, feel free I’d be happy to pull that PR in. Once the project is built use the CLI as defined above.

Installing the CLI App

This application can be installed as a CLI app by referencing it’s location in your bash (powershell? etc) startup script.

Schedule Updates: Coding, Meetups, Twitch Streams, Etc.

Here is my updated schedule for the two weeks starting August 12th for my Twitch streams, the basic topics, meetups (a little beyond two weeks), and related events coming up.

Series: Thrashing Code General Calamity

This is going to be a mix of tech this week. Probably some database hacking, code hacking, samples, and setup of even more examples for use throughout your coding week!

Series: Meetups

This is a little further out than the next few weeks, but in August we’re having another meetup!

Series: Emerald City Tech Conversations

Join me and several guests, on several different days to get into particulars about programming, coding, hacking, ops, devops, or whatever comes up. Ask questions, inquire, infer, or inform we’re here for the conversation. Join my guests and I!

Series: DataStax’s Apache Cassandra & C# Hour (C# Schema Migration builder, C# driver, and additional content). Navigate through to the DataStax Developers Channel to check out the event times.

The regularly occurring DataStax Devs series are continuing too, so join me, Cristina, Eric, and others on the team for these episodes and more. Check out the channel here!

Learning Go Episode 5 – Functions (and Methods and lots of other things)

Episode Post & Video Links:  1, 2, 3, 4, 5 (this post), 6, 7, and 8. Non-linked are in the works! Videos available now on Youtube however, so check em’ out!

In this session we covered a host of topics around Go Functions. Along with some troubleshooting, debugging, and other features in Jetbrains Goland IDE.

If you’d like to go through this material too in book form, I highly suggest “The Go Programming Language” by Alan A.A. Donovan & Brian W. Kernighan as a starting point. I’m using it as a simple guideline, but also doing a lot more in each stream that includes ecosystem, dependency management with godep, IDE use of Goland from Jetbrains, and more. In this session I get specifically into: functions, signatures, declarations, recursion, return values, and more.

Video Time Points & Topics

2:50 – Introduction to the snowy wonderland of Seattle and episode 5 of the Learning Go series. Introduction to the various screen transitions and such.
6:40 – Getting started, opening up JetBrains Goland and creating a new project. The project exists on Github as https://github.com/adron/learning-go-….
12:18 – Starting with functions in Go. See the blog entry I wrote on the topic for more additional information around this first code session within the episode 5 session.

Code – This first example I setup a basic function in Go that is called by the main function. The sample function below I’ve named exampleExecutor, and the signature is made up of an int parameter called this, a string parameter called that, and a return parameter of type int and one of type string. In summary for the function signature we have two input parameters going in and two return parameters coming out.

The function does very little besides print the parameters passed in and then return the parameters back out as the return parameters.


package main
import "fmt"
func main() {
var this, result int
var that, message string
this = 2
that = "42"
result, message = exampleExecutor(this, that)
fmt.Printf("%s\n%d", message, result)
}
func exampleExecutor(this int, that string) (int, string) {
fmt.Printf("Numbers: %d, %s\n", this, that)
return this, "This is the result: " + that
}

view raw

func_main_plus_example_executor.go

hosted with ❤ by GitHub

Recursion with Go & HTML Parsing

28:10 – Here I get into recursion and the application example, largely taken from the book but with some very distinctive modifications, that parses HTML and the various nodes within an HTML document.

For the recursion section I use an example from the book with an expanded sample set of HTML. The HTML is included in the repo under the function-recursion branch. For this example I setup a set of types and variables up that are needed throughout the code.

First a type setup called NodeType of type int32. A constant array of ErrorNode, TextNode, DocumentNode, ElementNode, CommentNode, and DoctypeNode for determining the different nodes within an HTML document. Then a general struct called Node with Type, Data, Attr for attribute, and FirstChild with NextSibling setup as a pointer to *Node, which gives a type of memory recursion to the underlying type. Then finally the Attribute struct with a Key a Value.


type NodeType int32
const (
ErrorNode NodeType = iota
TextNode
DocumentNode
ElementNode
Commentnode
DoctypeNode
)
type Node struct {
Type NodeType
Data string
Attr []Attribute
FirstChild, NextSibling *Node
}
type Attribute struct {
Key, Val string
}

view raw

types_nodes_etc.go

hosted with ❤ by GitHub

One of the first functions I then end up with is the visit function. It turns out as shown below. Here the function takes a links parameter that is of type string array, a parameter name n that is a pointer reference to an a node within html, and then the function returns a parameter of type string array.


func visit(links []string, n *html.Node) []string {
if n.Type == html.ElementNode && n.Data == "a" {
for _, a := range n.Attr {
if a.Key == "href" {
links = append(links, a.Val)
}
}
}
for c := n.FirstChild; c != nil; c = c.NextSibling {
links = visit(links, c)
}
return links
}

view raw

visit_func.go

hosted with ❤ by GitHub

After that function I worked through and created two additional functions, one for countsWordsAndImages and one called CountWordsAndImages. The casing being specific to scope and use of the functions, and they respectively look like this in completion.


func CountWordsAndImages(url string) (words, images int, err error) {
resp, err := http.Get(url)
if err != nil {
return
}
doc, err := html.Parse(resp.Body)
resp.Body.Close()
if err != nil {
err = fmt.Errorf("parsing HTML: %s", err)
}
words, images = countWordsAndImages(doc)
return
}
func countWordsAndImages(n *html.Node) (words, images int) {
if n.Type == html.TextNode {
words += len(strings.Split(n.Data, " "))
return
}
if n.Data == "img" {
images++
} else {
if n.FirstChild != nil {
w, i := countWordsAndImages(n.FirstChild)
words += w
images += i
}
if n.NextSibling != nil {
w, i := countWordsAndImages(n.NextSibling)
words += w
images += i
}
}
return
}

view raw

counting.go

hosted with ❤ by GitHub

Then all that is wrapped up, with recursive calls and more, in the main function for program execution.


func main() {
htmlContent, err := ioutil.ReadFile("compositecode.html")
if err != nil {
fmt.Println(err)
}
htmlData := string(htmlContent)
r := strings.NewReader(htmlData)
doc, err := html.Parse(r)
if err != nil {
fmt.Fprintf(os.Stderr, "find links: %v\n", err)
os.Exit(1)
}
for _, link := range visit(nil, doc) {
fmt.Println(link)
}
w, i, _ := CountWordsAndImages("https://compositecode.blog&quot;)
fmt.Printf("Words: %d\nImages: %d\n", w, i)
}

view raw

TheCountingRecursionOfThings.go

hosted with ❤ by GitHub

Starting Error Handling && Anonymous Functions

1:32:40 – At this point in the episode 5 session I get into a simple Error handling function, and further into function signatures and how to set them up.
1:52:24 – Setting up some anonymous functions and reviewing what they are.
1:59:00 – Introduction to panics in Go. After this short introduction I also discuss some of the pedantic specifics of methods vs functions and related verbiage around the Go language. Additionally I provide more examples around these specifics for declaring functions, various scope, and other types for function calls and related usage.

With that done the wrap up of the session is then a short introduction to anonymous functions.