Previously I’ve gone through the steps I take to get a solid development machine setup. From the base operating system load, to the browser and basic IDEs I install. Now I’ve got more videos and the respective notes and details about what two language stacks I setup next; Python and Go.
Python
In regards to the Python stack, this one can often be somewhat confusing. Depending on the operating system I setup the stack a little differently.
MacOS
For MacOS I’ve written two posts about this previously, one titled “Getting Started with Python Right!” and one “Unbreaking Python Through Virtual Environments“. Those two posts cover most of the nuance to getting a base Python stack installed on MacOS and then using virtual environments to manage project specific versions per repository.
Linux
For the Linux OS, usually a debian variant, the systems tend to have Python 3 installed by default. I then take the next step of installing pip3 and work from there. The IDE, PyCharm from Jetbrains uses virtualenv to setup virtual environments per repository from that point forward.
Python Setup && Reasons
For more details about the specific walk through, I’ve created this video to walk through setting up Python 3 on Ubuntu and verifying, and also by use of PyCharm to setup a small verification app it shows how the virtualenv sets up a specific environment for the new verification project.
The reasons for installing Python first are numerous. One of the first reasons is that Python is required for installing and using numerous Python related CLIs, such as AWS’s CLI, among many others. It pays off to just have a good install at the system level (i.e. not particular just in a virutal environment, but executable at the terminal on system) to ensure it is available for any and all CLIs that would need it. If you’re into data science work, that’s a huge second reason, because Python is used in about every aspect of data science work, machine learning, and related endeavors.
Go Setup && Reasons
The reason I go for Go as my second language stack install is driven by two primary reasons:
I like writing Go and use it myself for a number of reasons. Such as, it is ridiculously quick and minimal work to build a CLI for use in systems that requires only a single binary executable for use.
I use Go for a lot of other work-related efforts, around Kubernetes, Docker, Terraform, and others.
With that, here’s the quick install and initial project for verification setup.
That’s it for now. However, if you’re interested in joining me for next steps, language stack setup, and more in addition to writing some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and all sorts of coding I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!
“Black Hat Go” Go Programming for Hackers and Pentesters by Tom Steele, Chris Patten, and Dan Kottmann.
I picked this book up after a good look at the index. There is a basic intro to Go at the beginning of the book to cover language fundamentals, but immediately after that dives into the meat of the topic. Immediately getting into understanding the TCP handshake, TCP itself, writing a scanner, and a proxy. There is then some basics about HTTP Servers, routers, and middleware in chapter 3 and 4, but returns immediately into topics of interest in chapter 5 around exploiting DNS, and writing DNS clients. I perused the index a bit further to note it covers SMB and NTLM, a chapter on “Abusing Databases and Filesystems”, then on to raw packet processing, writing and porting exploit code, and a host of other topics. This is going to be an interesting book to dig into and write about in the coming weeks. I’m pretty excited about it and hope to write a thorough review upon completion.
“Designing Data-Intensive Applications” by Martin Kleppmann
This book is familiar territory, as I’ve spent much of my career working on similar topics that this book covers. What I suspect is that I’ll enjoy reading the material presented in an orderly and concise way versus the chaos and disruptive way I’ve acquired the knowledge on these topics.
From the index, the book starts off with foundations of data systems and the ideas around building reliable, scalable, and maintainable applications. This provides a good basis in which to dive into the other topics. From there it looks like we’ll get a run into the birth of NoSQL, object-relational mismatches and the related insanity that this has bred in the industry! Then a solid dive into graph-like, traditional, and multi-model data modeling. With the beginning quarter of the book then covering everything from hash indexes, SSTables (another familiar topic), LSM-Trees, B-Trees, and related indexing structures before wrapping up this first 25% of the book with stars and snowflake topics for analytics and column-oriented storage, compression, sort orders in column storage, and related material on aggregation in data cubes and materialized views.
That’s just the first 25%! From there Martin covers a wide range of topics, that if you’re in the industry and plan to deal with large scale data-intensive applications these are topics you need to be intimately familiar with!
Reading
Over the next few months while I read through these books I hope to provide summaries and related notes on the material. Who knows, maybe you’ll want to dive into the material yourself! Until then happy thrashing code and may you have high retention and comprehension in reading!
Yesterday we finally went full GA (General Availability) with DataStax Astra. For the quick TLDR think of it as Apache Cassandra that you can spin up as a service and use in about a minute. I, as I wrote about some months ago, joined the engineering team to help build out the system! I quickly got to reconnoitering the role and working toward build out of features, which now are available to you!
With Astra, if you’ve used Apache Cassandra or DataStax Enterprise you can use the same drivers or CQL you’re familiar with. But with Astra there are two additional capabilities we’ve just released to use in connecting to and working with your databases:
Astra REST API
Astra GraphQL API
With the REST API there are a number of capabilities to add a table, return a list of all the tables, return content of a table, and delete a table. In addition to tables, there is functionality to retrieve, retrieve all, add, update, and delete columns. All of the standard CRUD (Create, Read, Update, and Delete) commands can also be performed.
For the GraphQL API it gives you the ability to perform CRUD actions and query with filters using the GraphQL syntax.
Authorization Token
To use either of these services, the first thing you’ll need is to create one of Astra’s time based authorization tokens. These tokens work until 30 minutes after the last call made with the token. Once expired a new token must be created. To create a token an HTTP POST to the API can be made, passing several header values, and username and password in the body of a POST request.
For an example of retrieving an authorization token I’ve put together a cURL request below. To get the URL for your database navigate to the Astra dashboard, and on the summary screen of any database the API Access URL’s are listed.
With that authorization token we can now call actions against the REST, or GraphQL APIs.
Creating a Table via the Astra REST API
To create a table, we need a few key elements: The table name, whether it should create if a table exists or not, and column definitions with at least one column as a primary key. This is done by using JSON to pass this schema to the REST API. Here’s an example of some JSON that can be used to create a table.
To use this JSON to create a table, just add the pertinent headers, insert your keyspace into the URL, and the x-cassandra-token and POST this data to the REST API end point. A cURL request to create the table would look like this.
At this point, with a data created, we can add, update, or delete data. The sample curl statement I’ve put together here is a sample GraphQL mutation to add a record to the products table.
With that short tour, check out your free database today @ https://astra.datastax.com/register! Feel free to ping me on Twitter @Adron or here in comments, I’m open to and would love to discuss your experience!
This is a quick starter, to show some of the features of Go’s http library. (sample repo here) The docs for the library are located here if you want to dig in deeper. In this post however I cover putting together a simple http server with a video on creating the http server, setting a status code, which provides the basic elements you need to further build out a fully featured server. This post is paired with a video I’ve put together, included below.
00:15 Reference to the Github Repo where I’ve put the code written in this sample. https://github.com/Adron/coro-era-coding-go-sample 00:18 Using Goland IDE (Jetbrains) to clone the repository from Github. 00:40 Creating code files. 01:00 Pasted in some sample code, and review the code and what is being done. 02:06 First run of the web server. 02:24 First function handler to handle the request and response of an HTTP request and response cycle. 04:56 Checking out the response in the browser. 05:40 Checking out the same interaction with Postman. Also adding a header value and seeing it returned via the browser & related header information. 09:28 Starting the next function to provide further HTTP handler functionality. 10:08 Setting the status code to 200. 13:28 Changing the status code to 500 to display an error.
Getting a Server Running
I start off the project by pulling an empty repository that I had created before starting the video. In this I use the Jetbrains Goland IDE to pull this repository from Github.
Next I create two files; main.go and main_test.go. We won’t use the main_test.go file right now, but in a subsequent post I’ll put together some unit tests specifically to test out our HTTP handlers we’ll create. Once those are created I pasted in some code that just has a basic handler, using an anonymous function, provides some static file hosting, and then sets up the server and starts listening. I make a few tweaks, outlined in the video, and execute a first go with the following code.
packagemainimport("fmt""net/http")funcmain(){http.HandleFunc("/",func(whttp.ResponseWriter,r*http.Request){fmt.Fprintf(w,"Welcome to my website!")})http.ListenAndServe(":8080",nil)}
When that executes, opening a browser to localhost:8080 will bring up the website which then prints out “Welcome to my website!”.
Adding a Function as an HTTP Handler
The next thing I want to add is a function that can act as an HTTP handler for the server. To do this create a function just like we’d create any function in Go. For this example, the function I built included several print line calls to the ResponseWriter with Request properties and a string passed in.
funcRootHandler(whttp.ResponseWriter,r*http.Request){fmt.Fprintln(w,"This is my content.")fmt.Fprintln(w,r.Header)fmt.Fprintln(w,r.Body)}
In the func main I changed out the root handler to use this newly created handler instead of the anonymous function that it currently has in place. So swap out this…
http.HandleFunc("/",func(whttp.ResponseWriter,r*http.Request){fmt.Fprintf(w,"Welcome to my website!")})
Now executing this and navigating to localhost:8080 will display the following.
The string is displayed first “This is my content.”, then the header, and body respectively. The body, we can see is empty. Just enclosed with two braces {}. The header is more interesting. It is returned as a map type, between the brackets []. Showing an accept, accept-encoding, accept-language, user-agent, and other header information that was passed.
This is a good thing to explore further, check out how to view or set the values associated with the header values in HTTP responses, requests, and their related metadata. To go a step further, and get into this metadata a tool like Postman comes in handy. I open this tool up, setup a GET request and add an extra header value just to test things out.
Printing Readable Body Contents
For the next change I wanted to get a better print out of body contents, as the previous display was actually just attempting to print out the body in an unreadable way. In this next section I used an ioutil function to get the body to print out in a readable format. The ioutil.ReadAll function takes the body, then I retrieve a body variable with the results, pending no error, the body variable is then cast as a string and print out to the ResponseWriter on the last line. The RootHandler function then reads like this with the changes.
funcRootHandler(whttp.ResponseWriter,r*http.Request){fmt.Fprintln(w,"This is my content.")fmt.Fprintln(w,r.Header)deferr.Body.Close()body,err:=ioutil.ReadAll(r.Body)iferr!=nil{fmt.Fprintln(w,err)}fmt.Fprintln(w,string(body))}
If the result is then requested using Postman again, the results now display appropriately!
Response Status Codes!
HTTP Status codes fit in to a set of ranges for various categories of responses. The most common code is of course the success code, which is 200 “Status OK”. Another common one is status code 500, which is a generic catch all for “Server Error”. The ranges are as follows:
Informational responses (100–199)
Successful responses (200–299)
Redirects (300–399)
Client errors (400–499)
and Server errors (500–599)
For the next function, to get an example working of how to set this status code, I added the following function.
funcResponseExampleHandler(whttp.ResponseWriter,r*http.Request){w.WriteHeader(200)fmt.Fprintln(w,"Testing status code. Manually added a 200 Status OK.")fmt.Fprintln(w,"Another line.")}
Now we’re ready to try that out. In the upper right of Postman, the 200 status is displayed. The other data is shown in the respective body & header details of the response.
Next up, let’s just write a function specifically to return an error. We’ll use the standard old default 500 status code.
Now if the server is run again and an HTTP request is sent to the end point, the status code changes to 500 and the message “Server error.” displays on the page.
Summary
That’s a quick intro to writing an HTTP server with Go. From here, we can take many next steps such as writing tests to verify the function handlers, or setup a Docker image in which to deploy the server itself. In subsequent blog entries I’ll write up just those and many other step by step procedures. For now, a great next step is to expand on trying out the different functions and features of the http library.
That’s it for now. However, if you’re interested in joing me to write some JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and coding in general I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode. Feel free to check out a coding session, ask questions, interject, or just come and enjoy the tunes!
In this short (less than 4 minutes) video I put together a Go module library, then setup some first initial tests calling against functions in the module.
00:40 – Creating the Go go.mod file. When creating the go.mode file, note the command is go mod init [repopath] where repopath is the actual URI to the repo. The go.mod file that is generated would look like this.
01:09 – Instead of TDD, first I cover a simple function implementation. Note the casing is very important in setting up a test in Go. The test function needs to be public, thus capitalized, and the function needs accessible, thus also capitalized. The function in the awesomeLib.go file, just to have a function that would return some value, looks like this.
01:30 – Creating the test file, awesomeLib_test.go, then creating the test. In this I also show some of the features of the Goland IDE that extracts and offers the function names of a module prefaced with Test and other naming that provides they perform a test, performance, or related testing functionality.
packageawesomeLibimport"testing"funcTestGetKnownResult(t*testing.T){got:=GetKnownResult()ifgot!="known"{t.Error("Known result not received, test failed.")}}
02:26 – Running the standard go test to execute the test. To run tests for all files within this module, such as if we’ve added multiple directories and other files, would be go test ./....
02:36 – Using Goland to run the tests with singular or multiple tests being executed. With Goland there are other capabilities to show test coverage, what percentage of functionality is covered by tests, and other various code metrics around testing, performance, and other telemetry.
That’s it, now the project is ready for elaboration and is setup for a TDD, BDD, or implement and test style approach to development.
For JavaScript, Go, Python, Terraform, and more infrastructure, web dev, and coding in general I stream regularly on Twitch at https://twitch.tv/adronhall, post the VOD’s to YouTube along with entirely new tech and metal content at https://youtube.com/c/ThrashingCode.
You must be logged in to post a comment.