Language Stack Installation for Python & Go

Previously I’ve gone through the steps I take to get a solid development machine setup. From the base operating system load, to the browser and basic IDEs I install. Now I’ve got more videos and the respective notes and details about what two language stacks I setup next; Python and Go.

Python

In regards to the Python stack, this one can often be somewhat confusing. Depending on the operating system I setup the stack a little differently.

MacOS

For MacOS I’ve written two posts about this previously, one titled “Getting Started with Python Right!” and one “Unbreaking Python Through Virtual Environments“. Those two posts cover most of the nuance to getting a base Python stack installed on MacOS and then using virtual environments to manage project specific versions per repository.

Linux

For the Linux OS, usually a debian variant, the systems tend to have Python 3 installed by default. I then take the next step of installing pip3 and work from there. The IDE, PyCharm from Jetbrains uses virtualenv to setup virtual environments per repository from that point forward.

Python Setup && Reasons

For more details about the specific walk through, I’ve created this video to walk through setting up Python 3 on Ubuntu and verifying, and also by use of PyCharm to setup a small verification app it shows how the virtualenv sets up a specific environment for the new verification project.

The reasons for installing Python first are numerous. One of the first reasons is that Python is required for installing and using numerous Python related CLIs, such as AWS’s CLI, among many others. It pays off to just have a good install at the system level (i.e. not particular just in a virutal environment, but executable at the terminal on system) to ensure it is available for any and all CLIs that would need it. If you’re into data science work, that’s a huge second reason, because Python is used in about every aspect of data science work, machine learning, and related endeavors.

Go Setup && Reasons

The reason I go for Go as my second language stack install is driven by two primary reasons:

  1. I like writing Go and use it myself for a number of reasons. Such as, it is ridiculously quick and minimal work to build a CLI for use in systems that requires only a single binary executable for use.
  2. I use Go for a lot of other work-related efforts, around Kubernetes, Docker, Terraform, and others.

With that, here’s the quick install and initial project for verification setup.

If you’d like to take a few other quick tours of Go, here are some posts, with videos, putting together a Go module project and writing an initial test in under 3 minutes and setting up an HTTP server in about 15 minutes.

That’s it for now. However, if you’re interested in joining me for next steps, language stack setup, and more in addition to writing some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and all sorts of coding I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!

For more on my open source efforts and related projects, sign up for the Thrashing Code Newsletter!

My Top 2 Reading List for Go & Data

Right now I’ve got a couple books in queue.

“Black Hat Go” Go Programming for Hackers and Pentesters by Tom Steele, Chris Patten, and Dan Kottmann.

Alt Text

I picked this book up after a good look at the index. There is a basic intro to Go at the beginning of the book to cover language fundamentals, but immediately after that dives into the meat of the topic. Immediately getting into understanding the TCP handshake, TCP itself, writing a scanner, and a proxy. There is then some basics about HTTP Servers, routers, and middleware in chapter 3 and 4, but returns immediately into topics of interest in chapter 5 around exploiting DNS, and writing DNS clients. I perused the index a bit further to note it covers SMB and NTLM, a chapter on “Abusing Databases and Filesystems”, then on to raw packet processing, writing and porting exploit code, and a host of other topics. This is going to be an interesting book to dig into and write about in the coming weeks. I’m pretty excited about it and hope to write a thorough review upon completion.

“Designing Data-Intensive Applications” by Martin Kleppmann

Alt Text

This book is familiar territory, as I’ve spent much of my career working on similar topics that this book covers. What I suspect is that I’ll enjoy reading the material presented in an orderly and concise way versus the chaos and disruptive way I’ve acquired the knowledge on these topics.

From the index, the book starts off with foundations of data systems and the ideas around building reliable, scalable, and maintainable applications. This provides a good basis in which to dive into the other topics. From there it looks like we’ll get a run into the birth of NoSQL, object-relational mismatches and the related insanity that this has bred in the industry! Then a solid dive into graph-like, traditional, and multi-model data modeling. With the beginning quarter of the book then covering everything from hash indexes, SSTables (another familiar topic), LSM-Trees, B-Trees, and related indexing structures before wrapping up this first 25% of the book with stars and snowflake topics for analytics and column-oriented storage, compression, sort orders in column storage, and related material on aggregation in data cubes and materialized views.

That’s just the first 25%! From there Martin covers a wide range of topics, that if you’re in the industry and plan to deal with large scale data-intensive applications these are topics you need to be intimately familiar with!

Reading

Over the next few months while I read through these books I hope to provide summaries and related notes on the material. Who knows, maybe you’ll want to dive into the material yourself! Until then happy thrashing code and may you have high retention and comprehension in reading!

Development Machine Environment Build and Base OS Load

For the next post in this series building a development machine environment check out the browser & IDEs post.

Development Machine Environment Build & Language Stack Installations – Getting a Base OS Load

Operating System Notes

When choosing an operating system to work with there are a number of factors to take into account. Some of these factors include;

  1. What language stack and tooling will you need to use?
  2. What focus beyond software development will the machine have?
  3. What apps and tooling do you already have?

There are many other questions, around costs and licensing, and other errata that you’ll need to take into account. One thing however, regardless of what you choose, the bulk of software development with most software languages, their respective tooling stacks work on almost any operating system you’d choose.

In this particular example and the following material I’ve chosen Ubuntu 20.04 LTS (Long Term Support) to use as my operating system. I’ll build a virtualization image of the operating system so that I can reuse time and time again. The virtualization software I’m using is VMWare’s VMWare Fusion software, but just like the operating system you can use most virtualization software and it will work just fine.

Getting a Base OS Loaded

It is often important to load a new operating system installation onto a machine for use in application (re: web/software/IoT/mobile/etc) development. This can help us better understand what is installed and effecting various affects on the machine itself, it can insure that we have known assets that aren’t influencing server or other manipulations, and a long list of positives over taking whatever one is given from the various computer system manufacturers.

This post is the first in a series (here on Dev.to and on my core blog https://compositecode.blog). I’m going through precise steps, but I’m keeping it general enough with pointers and related tips for MacOS and Windows that the series will be useful for anyone building their first or tenth or twentieth development machine regardless of operating system or hardware.

My Installation Specifics

I’m using a Macbook Pro 16″ w/ enough RAM and compute cores that I split off a solid 4-8GB of RAM dedicated to the VM that has the host OS. In addition, I set the compute

To do this, in VMWare Fusion pull up the settings for the Virtual Machine that was created. To note, when the virtual machine is shutdown you can change these settings to add or decrease memory, compute cores, etc. But do note, sometimes this will make an operating system unstable. The rule to follow is, set the resources at time of creation and leave them set.

Once the settings are open, click on the processors and memory option.

Settings Screen for the Virtual Machine

That will display this screen where the values can be set. I also show stepping through this process in the video.

Alt Text

That’s it for now. However, if you’re interested in joining me for next steps, language stack setup, and more in addition to writing some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and all sorts of coding I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!

For more Thrashing Code Newsletter for more details about open source projects and related efforts I work on, sign up for it here!

Simple Go HTTP Server Starter in 15 Minutes

This is a quick starter, to show some of the features of Go’s http library. (sample repo here) The docs for the library are located here if you want to dig in deeper. In this post however I cover putting together a simple http server with a video on creating the http server, setting a status code, which provides the basic elements you need to further build out a fully featured server. This post is paired with a video I’ve put together, included below.

00:15 Reference to the Github Repo where I’ve put the code written in this sample.
https://github.com/Adron/coro-era-coding-go-sample
00:18 Using Goland IDE (Jetbrains) to clone the repository from Github.
00:40 Creating code files.
01:00 Pasted in some sample code, and review the code and what is being done.
02:06 First run of the web server.
02:24 First function handler to handle the request and response of an HTTP request and response cycle.
04:56 Checking out the response in the browser.
05:40 Checking out the same interaction with Postman. Also adding a header value and seeing it returned via the browser & related header information.
09:28 Starting the next function to provide further HTTP handler functionality.
10:08 Setting the status code to 200.
13:28 Changing the status code to 500 to display an error.

Getting a Server Running

I start off the project by pulling an empty repository that I had created before starting the video. In this I use the Jetbrains Goland IDE to pull this repository from Github.

1

Next I create two files; main.go and main_test.go. We won’t use the main_test.go file right now, but in a subsequent post I’ll put together some unit tests specifically to test out our HTTP handlers we’ll create. Once those are created I pasted in some code that just has a basic handler, using an anonymous function, provides some static file hosting, and then sets up the server and starts listening. I make a few tweaks, outlined in the video, and execute a first go with the following code.

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", func (w http.ResponseWriter, r *http.Request) {
        fmt.Fprintf(w, "Welcome to my website!")
    })

    http.ListenAndServe(":8080", nil)
}

When that executes, opening a browser to localhost:8080 will bring up the website which then prints out “Welcome to my website!”.

2

Adding a Function as an HTTP Handler

The next thing I want to add is a function that can act as an HTTP handler for the server. To do this create a function just like we’d create any function in Go. For this example, the function I built included several print line calls to the ResponseWriter with Request properties and a string passed in.

func RootHandler(w http.ResponseWriter, r *http.Request){
    fmt.Fprintln(w, "This is my content.")
    fmt.Fprintln(w, r.Header)
    fmt.Fprintln(w, r.Body)
}

In the func main I changed out the root handler to use this newly created handler instead of the anonymous function that it currently has in place. So swap out this…

http.HandleFunc("/", func (w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Welcome to my website!")
})

with this…

http.HandleFunc("/", RootHandler)

Now the full file reads as shown.

package main

import (
    "fmt"
    "net/http"
)

func main() {
    http.HandleFunc("/", RootHandler)
    http.ListenAndServe(":8080", nil)
}

Now executing this and navigating to localhost:8080 will display the following.

3

The string is displayed first “This is my content.”, then the header, and body respectively. The body, we can see is empty. Just enclosed with two braces {}. The header is more interesting. It is returned as a map type, between the brackets []. Showing an accept, accept-encoding, accept-language, user-agent, and other header information that was passed.

This is a good thing to explore further, check out how to view or set the values associated with the header values in HTTP responses, requests, and their related metadata. To go a step further, and get into this metadata a tool like Postman comes in handy. I open this tool up, setup a GET request and add an extra header value just to test things out.

4

Printing Readable Body Contents

For the next change I wanted to get a better print out of body contents, as the previous display was actually just attempting to print out the body in an unreadable way. In this next section I used an ioutil function to get the body to print out in a readable format. The ioutil.ReadAll function takes the body, then I retrieve a body variable with the results, pending no error, the body variable is then cast as a string and print out to the ResponseWriter on the last line. The RootHandler function then reads like this with the changes.

func RootHandler(w http.ResponseWriter, r *http.Request){
    fmt.Fprintln(w, "This is my content.")
    fmt.Fprintln(w, r.Header)

    defer r.Body.Close()

    body, err := ioutil.ReadAll(r.Body)
    if err != nil {
        fmt.Fprintln(w, err)
    }
    fmt.Fprintln(w, string(body))
}

If the result is then requested using Postman again, the results now display appropriately!

5

Response Status Codes!

HTTP Status codes fit in to a set of ranges for various categories of responses. The most common code is of course the success code, which is 200 “Status OK”. Another common one is status code 500, which is a generic catch all for “Server Error”. The ranges are as follows:

  • Informational responses (100–199)
  • Successful responses (200–299)
  • Redirects (300–399)
  • Client errors (400–499)
  • and Server errors (500–599)

For the next function, to get an example working of how to set this status code, I added the following function.

func ResponseExampleHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(200)
    fmt.Fprintln(w, "Testing status code. Manually added a 200 Status OK.")
    fmt.Fprintln(w, "Another line.")
}

With that, add a handler to the main function.

http.HandleFunc("/response", ResponseExampleHandler)

Now we’re ready to try that out. In the upper right of Postman, the 200 status is displayed. The other data is shown in the respective body & header details of the response.

6

Next up, let’s just write a function specifically to return an error. We’ll use the standard old default 500 status code.

func ErrorResponseHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(500)
    fmt.Fprintln(w, "Server error.")
}

Then in main, as before, we’ll add the http handle for the function handler.

http.HandleFunc("/errexample", ErrorResponseHandler)

Now if the server is run again and an HTTP request is sent to the end point, the status code changes to 500 and the message “Server error.” displays on the page.

7

Summary

That’s a quick intro to writing an HTTP server with Go. From here, we can take many next steps such as writing tests to verify the function handlers, or setup a Docker image in which to deploy the server itself. In subsequent blog entries I’ll write up just those and many other step by step procedures. For now, a great next step is to expand on trying out the different functions and features of the http library.

That’s it for now. However, if you’re interested in joing me to write some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and coding in general I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!

Creating a Go Module & Writing Tests in Less Than 3 Minutes

In this short (less than 4 minutes) video I put together a Go module library, then setup some first initial tests calling against functions in the module.

 

00:12 – Writing unit tests.

00:24 – Create the first go file.

00:40 – Creating the Go go.mod file. When creating the go.mode file, note the command is go mod init [repopath] where repopath is the actual URI to the repo. The go.mod file that is generated would look like this.

module github.com/adron/awesomeLib

go 1.13

00:56 Enabling Go mod integration for Goland IDE. This dialog has an option to turn off the proxy or go direct to repo bypassing the proxy. For more details on the Go proxy, check out goproxy.io and the Go docs for more details.

01:09Instead of TDD, first I cover a simple function implementation. Note the casing is very important in setting up a test in Go. The test function needs to be public, thus capitalized, and the function needs accessible, thus also capitalized. The function in the awesomeLib.go file, just to have a function that would return some value, looks like this.

package awesomeLib

func GetKnownResult() string {
    return "known"
}

01:30Creating the test file, awesomeLib_test.go, then creating the test. In this I also show some of the features of the Goland IDE that extracts and offers the function names of a module prefaced with Test and other naming that provides they perform a test, performance, or related testing functionality.

package awesomeLib

import "testing"

func TestGetKnownResult(t *testing.T) {
    got := GetKnownResult()
    if got != "known" {
        t.Error("Known result not received, test failed.")
    }
}

02:26Running the standard go test to execute the test. To run tests for all files within this module, such as if we’ve added multiple directories and other files, would be go test ./....

02:36Using Goland to run the tests with singular or multiple tests being executed. With Goland there are other capabilities to show test coverage, what percentage of functionality is covered by tests, and other various code metrics around testing, performance, and other telemetry.

That’s it, now the project is ready for elaboration and is setup for a TDD, BDD, or implement and test style approach to development.

For JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and coding in general I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode.