‘bash’ A.K.A. The Solution for Everything – Passed Variables & The Script Filename

When writing a script in bash you can pass parameters into that script to work with. For example, let’s say I have a script file called runme.sh and I want to pass in my name and today’s date. I could do that like this.

./runme.sh "Adron Hall" "12/27/2018"

Inside the script I can get access to the parameters by using the bash param variables of $1, $2, and so on. With a little bash code written up like this.

 


echo "The first parameter $1"
echo "The second parameter is $2"
echo "The \$0 parametere returns the name of the file! This file is named $0."

view raw

bashParams.sh

hosted with ❤ by GitHub

dghubble:go-twitter- Go Twitter REST and Streaming API v1.1 2

It’s 2019. Resolutions.

2018 might have been the first year I setup some resolutions. I honestly find them pretty silly. However I have joined the band wagon, and by doing so here are my thoughts on resolutions for 2019.

New Resolutions

Alright, these are entirely new for this year. It seems wise that I’ll just put together three, listed below, since I have a number of failings from 2018 that I’ll need to slot into my continuing resolutions list, and of course turn those into successes this year!

  1. Increase my streaming (specifically with Twitch) and the correlative videos around coding, systems, and related technologies. Possibly, even throw in some *gaming*!
  2. Go office-less again. My aim is to not be tied to a desktop, an office, or related physical place and again go roaming coder again (also one of the inspirations behind getting a new laptop). This way I can also ensure I get out and to more meetups, hackathons, or whatever events that look good.
  3. Compose and record some music. I would say record more music, but I haven’t recorded any music in ages so I’m going to remedy this at some point this year.

A few other things, which I wouldn’t declare as failures if I don’t get them finished this year, but the likelihood I start writing a book is pretty high. I’ll also organize more than a few larger type events (i.e. beyond a user group meetup, such as more conferences like ML4ALL or maybe a open spaces conf).

Continuing Resolutions

Heyo I sound like Congress now! The resolutions are from the previous year that I will be continuing to either succeed at or aim to succeed at. For specifics on each of these just navigate back to yesterday’s retrospective.

  1. Write More Code, Build Patterns, & Algorithms (Previous Year‘s Status: failed)
  2. Make OSS Workable for Me (Previous Year‘s Status: failed)
  3. Get More Active (Previous Year‘s Status: failed)
  4. Self Health (Previous Year‘s Status: failed)
  5. Improve Local Communities (Previous Year‘s Status: failed)
  6. Beer (Previous Year‘s Status: Succes!)
  7. Communications Improvements (Previous Year‘s Status: Succes!)
  8. Cell Phone Disruption (Previous Year‘s Status: Succes!)

Done Resolutions

These resolutions are resolutions that I’ve just decided I’m done with. Not going to do anymore or maybe they were just one offs, things that were a do once thing, such as “see Within Temptation play a show“.

  1. Re-initiate in Industry – This was really the only one off resolution from 2018. I managed to really get it done too. As for continuing to be involved in industry that’s kind of covered, easily, by continuing resolutions above.

With that, whether we’re ready for it or not, here’s 2019! Cheers!

 

 

 

My 2018 Retrospective

Alright, with that 2018 is a wrap. Christmas is a wrap. New Years is a wrap. It’s all done, wrapped up, and time to move on. Well, ok, so maybe a small retrospective! Early in 2018 I wrote a list of resolutions for 2018. How did I do? First I’m going to lead off with the things I have found myself failing miserably at.

Failed Resolution: Write More Code, Build Patterns & Algorithms

Write More Code, Build Patterns & Algorithms: I want to review and go back to some of the coding roots that I haven’t hit upon in a long time. It’s odd, when coding day in and day out one tends to not touch upon a lot of the fundamental basics. I want to start writing about and reviewing this again, keep it fresh in mind so it’s always easy to reach into my mind and explain how things work to who may ask. Goal: Write 0.5 blog entry per week on coding algorithms, patterns, or related core coding concepts and skills.

Oh dear I had a simple goal of writing 0.5 blog entries per week on coding algorithms, patterns, or related core coding concepts and skills. Yeah, I didn’t get anywhere near that, even though I increased my rate of blog posts somewhere around 6x what it was in 2017! Overall, yeah, groovy, I wrote more and that’s great, but I didn’t hit on the topics I really wanted to that I knew would provide more value!

Failed Resolution: Make OSS Workable for Me

Make OSS Workable for Me: Get the OSS projects I’ve started in the last 2–3 years into a more workable state, insure others can take them, build, and get running without issue. It’s been a few years since I’ve worked on and helped with any OSS projects that are actually used, it’s a bummer and I’m going to resolve that this year. Goal: Get two projects into a workable state so others can use them and I can use them for their intended purpose and for teaching and blogging purposes.

I’m so close on this goal. I’ve got the project into a little bit better of a state, but overall they’re still kind of rough. At the rate I’m going among the Twitch streams and such, these projects will be in a nicely usable state soon however! Probably by March or April I’d suspect I’ll have 2 of the projects (Colligere and Twitz) into  a usable state! But from the goal point of view, this is a failed resolution.

Failed Resolution: Get More Active

Get More Active: Take more bike rides, train trips, and spend more time with friends and family in Portland. Goal 1: Spend at least 3–4 days at Pedalpalooza this year, and take at least 4 trips (1 per quarter at minimum) of about ~2 days each in 2018. Goal 2: Participate in at least 1 group rides per quarter in Seattle.

I failed in this resolution miserably. In all honesty, it’s hard to even talk about. It fills me with rage that I failed at this fundamentally important resolution badly. There’s not a lot of things that I enjoy anywhere close to a good group ride, hanging out at Pedalpalooza, and generally being involved in my local community this way and it came all apart on me this year. It even fills me with rage, anger, and frustration with Seattle’s communities and how uninvolved most people are in Seattle itself. After coming from Portland I have tons of frustration with this, but I’m slowly learning to just live with it. Maybe this year, maybe not, I’ve no idea how I’m going to pull this together. I’m going to need to however as my mental and physical health largely rests on being involved in the biking community, with this involvement I generally falter into an angry, apathetic, disenfranchised angry person.

Failed Resolution: Self Health

Self Health: Take more time for myself, allocate appropriate me time to make sure I can keep my sanity. Goal: Write, ponder, introspect, and watch the world turn more often. Blog on the park, lake, boat, train, or place and moment of writing, pondering, introspecting, and watching the world turn by writing about it.

The first part of this year was catastrophically bad for me. However between joining a truly amazing team at DataStax, starting to finally get back to riding around Seattle, exploring, and producing content and learning new tech, languages, and the like I’m doing a lot better. I’ve got a long ways to go still before I’m back to 100%, but I think 2019 might be that year. At least, ending 2018 and going into 2019 I’m not as angry, depressed, and perturbed at society as I was at the beginning of this year. Now I’m just kind of euphorically disconnected and apathetic about the state of the world. It’s a major improvement!

Failed Resolution: Improve Local Communities

Improve Local Communities: Stop getting involved in Internet politics I can’t improve. Get more focused in local politics. Help more people in Seattle, help more people however I can. Help in the fight for better and more housing for all people. Help in the fight for better and more advanced and modern transportation options in Seattle, Portland, and wherever I can. Goal 1: Join Cascade Bicycle Club and the monthly urbanist meets. Goal 2: Keep up with the respective communities ala Bike Portland, The Urbanist, Seattle Transit Blog, Seattle Bike Blog, WA Bikes, and Cascade Bicycle Club.

Alright, I’m angry at myself about this one, but also realize I just couldn’t get to it. Don’t get me wrong, I attended some city council meetings and got involved some. But nowhere near as much as I’d prefer to have done. The simple fact too, is that Seattle political mechanisms are just garbage compared to Portland. It’s hard to know what’s going on let alone how the hell to get something done or improve one’s community, lot in life, or environment here. Seattle is a beast! One day, if we survive that long, it’ll make a great “Manhatten” of the northwest! (Ironically I’ve learned, it once was indeed called “New York City”, which to me seems like the dumbest name since New York was thus dubbed after “York” and before that was called New Amsterdam after Amsterdam, which Seattle has no real relation to in any way whatsoever)

But now, failures aside I digress.

TIME FOR VICTORIES!

Now it’s time for the successes. Some of these I barely passed, some I beautifully knocked out of the park!

Victory Resolution, “Beer”

Beer: Live a little, treat yo’ self, and hit up at least two happy hours or other meets with friends and coder/tech/hacker crews per month. The Ballard Happy Hour has been a good one so far, and of course the exceptional Seattle Scalability Group I help organize is pretty epic unto itself. The latter often having free beer post meet up. Both meets are good conversations, great beer, and a great chance to just unwind with smart folk. Goal: Attend two meets per month.

Ok, this one wasn’t purely about beer itself, but beer tends to play a large part of the activity. The core of it all was insuring that I get more involved in the local “tech” community here in Seattle. In this I have succeeded! Currently I’m leading the return of the Seattle Scalability Group with my meetup cohort plus the addition of the DataStax team having my back. This is of course only one of the activities I’ve gotten myself involve in again. I hope to also help out and get involved in other meetups around Seattle, and have started plotting the return of ML4ALL (v2) with Troy and Alena! I’m looking forward to this being a continuing resolution, and continually succeeding at meeting this resolution throughout 2019!

Victory Resolution: Communications Improvements

Communication Improvements: Find a new and more efficient way to increase the throughput of my follow ups with everybody. Whether by email, phone, or whatever it might be. Goal: Increase rate of follow ups by 15%.

Oh jeez, this one was easier to accomplish than I thought it was going to be. I’ve since gone through and reviewed. I managed to only drop the ball on ~3 of the over 340 important productive conversation threads I initiated! I’m also impressed with my ability to actually keep track of those thanks to my combination of software tools I started using and will be using for the ongoing future. That puts the increase rate somewhere around 99% and not merely 15%! Major victory on this accomplishment!

Victory Resolution: Cell Phone Disruption

Cell Phone Disruption: Decrease my actual cell phone usage making *phone calls* even further. Regain the privacy, personal time, and focus time that one traditionally had before the time of cell phones invading every corner of existence. How I do this I’m not entirely sure, but I’m betting when I figure it out I’ll blog it. Goal: Make it so that I don’t look at or need my cell phone for anything more than 1 phone call per week on average (vid chat, etc, excluded).

I was spectacularly victories in this goal. I think I’ve taken approximately 3 work calls for the entire year via the cell phone. I’ve decreased my screen time by almost 30%! Overall, I use my device dramatically less than I have in the past and it’s helped me in a pretty significant way. Maybe I could knock it down another 5 or 10% this year eh?

Victory Resolution: Re-initiate in Industry

Reinitiate in Industry: Kick back off and give some dedicated presentations at meet ups and conferences. Goal: Give at least 4–6 talks that are dedicated, focused, mid-level or advanced topics following my standards for speaking, follow up, and improving.

Knocked out three talks: Systemic Deployments, DataStax Developer Day: Operations & Security, and Node Systems for Node.js Services on Nodes of Systemic Nodal Systems. Then, to boot, I allocated some of my talks to Twitch streams and am continuing to step through these on a daily basis. Hope y’all will join me on a future stream, they’re fun! Overall though, lot’s of moving parts going on, and I’m going to likely double my speaking schedule this year. I’ve got a lot of cool things to show ya!

Next up, 2019.

Let’s go!

Got a New Laptop, Here’s The Review

A few past reviews just for context of my general working fleet of computers and why and what led me to this review and this new laptop purchase.

Important! Do take note, I’m not paid by Dell, or System76, or anybody to write up reviews of laptops or hardware for that matter. These are my observations with these systems. I’m also not paid to use these systems for software development, but am only paid for the software development I produce with these machines. In other words, I very much roll Bring Your Own Device (BYOD) style and develop routinely without an assigned machine. I do what I can to stick to BYOD and such as it is, write up reviews of what I choose to use.

The Setting & Context for Purchase

system76-leopard-16Over the last year I’ve been pondering getting a Linux based laptop. At least a laptop that can run Linux native on the metal as the sole OS or at least a clean dual boot option. I wanted this for several specific reasons. The final decision to move forward occurred at this very tweet moment.

Here’s the short list of why I wanted a new laptop, that has good Linux support.

  1. Most of my gaming is in Linux these days. Thanks Steam OS!
  2. Most of my server workloads, server coding, internet coding, back-end work, and related coding is all for Linux. I haven’t deployed something to another OS in production for at least a decade now. As for front end apps, that’s also basically stuff that runs on Linux or MacOS. Web interfaces or usually just some simple CLI’s. I did write a Windows 8 “Metro UI” App, but it’s long gone and dead already along with the database (Riak) that it was an interface for.
  3. Most of my automation work and related site reliability coding, ops hacking, my metasoftware engineering (great words for a title from Katie Keel @akatiekeel, see tweet below), and all that good fun is often done on MacOS but always ends up being deployed to run on a Linux machine in the end.
  4. I’ve already got two Linux machines that I do a huge percentage of work on. The Dell XPS 13 and System 76 Leopard Workstation. However, the Leopard is in a bit of disrepair after a disturbingly wicked power surge here in Ballard (Seattle). The XPS 13 is just a bit weak, old, and the keyboard is still the crappy keyboard I detailed in the past review.
  5. One of the big demands for this new laptop was that I wanted to be able to – at least with a somewhat efficient hardware performance level – edit video, stream video, run the virtual machines, the pertinent container ecosystems (i.e. distributed database stuff), of course lots of code, and play the few games I do play. This meant at basic some decent video – albeit I knew it wasn’t going to be what I had/have in my System76 machine – at least a terabyte of storage on my main drive, and 32 GB of RAM.

Buy Time

huawei-matebook-x-pro-nvidia-geforce-mx-150Alright, that was the setting, so I went about searching through a lot of different laptop options. One of the most promising was this Huawai Matebook that Jeff & Adam pointed me at. It looked so good from the website that I decided I wanted to go check out the physical Matebook Pro somewhere, if possible, and found that option here in Seattle at the Microsoft store in University Park (It’s an outdoor mall, yuck, but whatevers).

huawei-matebook-x-pro-multiple-portsI rolled over via bus and a short walk, walked into the Microsoft store and made a beeline right to where one of the Matebooks sat. It was gorgeous, absolutely beautiful, flawless, and outright better bang for the hardware buck than one of the Apple products from across the street! I was instantly sold.

huawei-matebook-x-pro-with-intel-cpuBut there was an issue. Hardware specs for this thing sit at 2GB Video, 8 GB RAM, and a 512 GB SSD. That’s a problem. I checked the site again to make sure there weren’t other options. Nope, it didn’t get much more built up than that. It just wouldn’t do.

huawei-matebook-x-pro-thin-bodyI felt pained, annoyed, and frustrated. Does anybody actually want some decent power in a slim, elegant, and easy to carry laptop? Am I the only one wanting something like this? I started strolling around the floor of the Microsoft store. Looking at hard drives and Xbox stuff. Which just to point out, these Microsoft stores really are Xbox stores as much or more than they are anything else!

NOTE: All Huawai images copyright of Huawai. I’m hoping they’re cool since I’m pointing out their awesome laptop.

The reason I bring up the Matebook, is because I really was impressed by the build quality. It exceeded my expectation and based on this research, trying it out, I would happily suggest this laptop as a prime choice if the specs meet what you need. For me, sadly, I wanted and needed a bit more RAM and some more oomf in other categories.

The Final Decision

I walked around the Microsoft store checking out the Lenovo and a number of other laptops. I played some racing game thing on Xbox for a second. I wasn’t in so much of a hurry that I just had to buy something right then. I had after all waited almost a year to get to this point. Maybe I’d just save the cash and wait a little longer? Surely something would come along.

Then I walked up to another table. I first looked at the spec list, which I had been doing at every table except when I had walked up to the Matebook. I see 1 TB option on this machine. That’s cool. Then I see 32 GB of RAM. Holy shit when did the selection on the floor leap out of the piddly 8 GB range?! Then I see 4 GB Video! Specifically a NVIDIA® GeForce® GTX 1050Ti with 4GB GDDR5! Hell yeah. Alright, I’d looked and here’s the laptop that after months seemed to be the only ongoing choice to get this kind of specs in something that I sort of trusted the support for. So I started to play around with the keyboard and oh, looky there, it’s a touch screen too. Not that I cared, but it’s not a bad thing to have really, albeit a waste as I’ll likely never touch the screen.

So there it was, the decision was made, bagged, paid for, and out I walked with a brand new Dell XPS 15. Maybe I should have bought it from Dell, but meh, this will work. Support plan is nice, if anything fails I bring it into the store and they get me a new laptop too. Overall price $2499 for 3 years of coverage plus laptop. Also, yes, considering my unfortunate luck with hardware over the years – unless it’s a machine I’ve built myself – I get the coverage because I’ve got all of zero time to mess around with being my own technician.

The Dell XPS 15

Alright, so I set out to put this thing through some tasks as soon as I got home. Well, ok, not immediately cuz I had to shoot and put together this unboxing video. Here ya go, it’s a little long, but I also cover a lot of related topics of interest related to this machine.

First Test – Application Load

My first test was simply installing and setup of some standard software I use. That included Creative Cloud from Adobe, Jetbrains IDE’s and tooling, OBS for video streaming, Steam so I could pull down Cities: Skylines and Transport Fever, and some other miscellaneous software I would need. All this I would install while I get my Dropbox downloads going. With that I set out to install all of this stuff.

First I got Dropbox installed and began syncing. This would easily be about 380-400 Gig of files and video files. With that started I set out to install things I knew – or hoped – wouldn’t incur one of the required Windows reboots. First was Jetbrains IDE’s, which involves downloading their Toolbox and then clicking through every tool I’d want and letting it queue them all up. Then the same thing with Creative Cloud. Altogether that includes:

  • IntelliJ
  • WebStorm
  • Goland
  • Pycharm
  • ReSharper Ultimate w/ the pertinent tools it downloads.
  • Photoshop
  • Illustrator
  • Indesign
  • Premier Pro
  • Premier Rush (eh? Never used it, but I’ll download it)
  • After Effects
  • Character Animator
  • Audition
  • Media Encoder
  • InCopy
  • Prelude
  • Fuse CC (Another tool I don’t know but hey, it’s there now!)

All of that downloaded and installed quickly. Having a gigabit connect really, REALLY, REALLY helps out massively to get this done! Between the solid connection, the SDD being pretty lightning fast, and the proc and memory not being a bottleneck, I lost track of timing this. Suffice it to say the hardware combination is pretty quick.

Second Test – Premiere Pro Rendering

With Premiere Pro installed I set about recollecting how to edit with this software. I’ve been using Screenflow for Mac and Kdenlive on Linux for so long I don’t remember a thing about how Premiere Pro works. However as I worked through transitions (building the above unboxing video) I started to recall how much power is in this software, and I also really got a feel for another thing, the trackpad, which I’ll add more about in a moment.

The rendering for MPG4 was a little faster than the Mac Book Pro I’ve got from almost ~5 years ago and above equivalent to performance with the next to latest generation Mac Book Pro laptops (i.e. It’s about ~2 years old that I’m comparing performance on). Overall, the performance of video rendering wasn’t super impressive. At least not like the leap from a MBP to my System76 Leopard Workstation, which screams through rendering at a reasonably large percentage (~25-40%) faster than my laptop machines. So the XPS 15 really is close or better – but just a little. I’d be curious to get hold of the latest MBP and compare the performance. Considering this has dedicated video, it has dedicated video, and both have similar processors and pretty hefty RAM.

Third Test – Trackpad

Ok, this is the closest I’ve ever used in usefulness, capability, and multi-finger touch as compared to an Apple product. The tactile feel in my opinion is better too with this fiber type material that Dell has used on the trackpad and the surrounding area around the trackpad and keyboard.

The first real test was the maneuvering one has to do when video editing with Premiere Pro. Lot’s of right clicking (double finger tap or the traditional bottom right of the trackpad!) swiping up and down but also side to side, and this Dell trackpad performed exceptionally well. Still not as transparently smooth as an Apple’s trackpad is, but it gets the job done and once I’m used to it, I’ll easily be just as fast as I was with an Apple trackpad. I give myself a day or two and I’ll be up to speed with it.

Fourth Test – Weight and Sizing

Here’s a few images of the XPS 15 compared to an older model (circa 2015) Mac Book Pro.

The build quality of the XPS 15, just like the XPS 13 I have from circa ~2015 is really good. There’s elements of it that don’t seem like they’re aligned correctly compared to the smooth feel and look of some of the other laptops, but overall it feels, and appears to be more egalitarian and functional focused versus many of the other laptop options. The edges are very smooth and the cover of the laptop is a single plate of aluminum, which gives it that feel when carrying it around similar to most of the modern laptops. The edges however aren’t there to look seemless or attractive, they’re there simply to provide side plates for USB, USB-C, Audio, HDMI, and related ports. No complaints but if you’re 100% focused on looks, this might not be an ideal option. Me… well I’m a funny looking fella and it’s probably noticeably I’m not staunchly focused on the appearance of anything. I’m all about function over form.

Further Tests – I’ll have more tests in the future, for now, that’s a wrap.

Summary

Other comments, observations, and notes on this laptop will be forthcoming. In a few months I’ll plan to write a follow up. In the meantime, if you’re looking for a relatively light (1.81 kg / 4 lbs), relatively powerful laptop (32 GB RAM, 4GB Video, 8th Gen i7 or i9 option) this is about as good as you’ll get for the price. If power isn’t your ultimate intent with a laptop purchase I highly suggest checking out the Matebook.

As for Windows 10. My verdict is still out on it, leaning toward “ugh, fuggidabout it I’m going back to MacOS/Linux land” but I’m going to give it a good go before I declare that. There’s definitely a few elements of the latest OS that I like and I also want to get a good feel for it’s Linux system before I write it off. Maybe I’ll stick with it after all? Until then, happy holiday hardware hacking!

Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

Part 3 of 3 – Coding Session in Go – Cobra + Viper CLI for Parsing Text Files, Retrieval of Twitter Data, and Exports to various file formats.

UPDATED PARTS:

  1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
  2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation
  3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval (this post)

Updated links to each part will be posted at bottom of  this post when I publish them. For code, written walk through, and the like scroll down below the video and timestamps.

0:54 The thrashing introduction.
3:40 Getting started, with a recap of the previous sessions but I’ve not got the sound on so ignore this until 5:20.
5:20 I notice, and turn on the volume. Now I manage to get the recap, talking about some of the issues with the Twitter API. I step through setup of the app and getting the appropriate ID’s and such for the Twitter API Keys and Secrets.
9:12 I open up the code base, and review where the previous sessions got us to. Using Cobra w/ Go, parsing and refactoring that was previously done.
10:30 Here I talk about configuration again and the specifics of getting it setup for running the application.
12:50 Talking about Go’s fatal panic I was getting. The dependency reference to Github for the application was different than what is in application and don’t show the code that is actually executing. I show a quick fix and move on.
17:12 Back to the Twitter API use by using the go-twitter library. Here I review the issue and what the fix was for another issue I was having previous session with getting the active token! Thought the library handled it but that wasn’t the case!
19:26 Now I step through creating a function to get the active oath bearer token to use.
28:30 After deleting much of the code that doesn’t work from the last session, I go about writing the code around handling the retrieval of Twitter results for various passed in Twitter Accounts.

The bulk of the next section is where I work through a number of functions, a little refactoring, and answering some questions from the audience/Twitch Chat (working on a way to get it into the video!), fighting with some dependency tree issues, and a whole slew of silliness. Once that wraps up I get some things committed into the Github repo and wrap up the core functionality of the Twitz Application.

58:00 Reviewing some of the other examples in the go-twitter library repo. I also do a quick review of the other function calls form the library that take action against the Twitter API.
59:40 One of the PR’s I submitted to the project itself I review and merge into the repo that adds documentation and a build badge for the README.md.
1:02:48 Here I add some more information about the configuration settings to the README.md file.

1:05:48 The Twitz page is now updated: https://adron.github.io/twitz/
1:06:48 Setup of the continuous integration for the project on Travis CI itself: https://travis-ci.org/Adron/twitz
1:08:58 Setup fo the actual travis.yml file for Go. After this I go through a few stages of troubleshooting getitng the build going, with some white space in the ole’ yaml file and such. Including also, the famous casing issue! Ugh!
1:26:20 Here I start a wrap up of what is accomplished in this session.

NOTE: Yes, I realize I spaced and forgot the feature where I export it out to Apache Cassandra. Yes, I will indeed have a future stream where I build out the part that exports the responses to Apache Cassandra! So subcribe, stay tuned, and I’ll get that one done ASAP!!!

1:31:10 Further CI troubleshooting as one build is green and one build is yellow. More CI troubleshooting! Learn about the travis yaml here.
1:34:32 Finished, just the bad ass outtro now!

The Codez

In the previous posts I outlined two specific functions that were built out:

  • Part 1 – The config function for the twitz config command.
  • Part 2 – The parse function for the twitz parse command.

In this post I focused on updating both of these and adding additional functions for the bearer token retrieval for auth and ident against the Twitter API and other functionality. Let’s take a look at what the functions looked like and read like after this last session wrap up.

The config command basically ended up being 5 lines of fmt.Printf functions to print out pertinent configuration values and environment variables that are needed for the CLI to be used.


var configCmd = &cobra.Command{
Use: "config",
Short: "A brief description of your command",
Long: `A longer description that spans multiple lines and likely contains examples
and usage of using your command. For the custom example:
Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
Run: func(cmd *cobra.Command, args []string) {
fmt.Printf("Twitterers File: %s\n", viper.GetString("file"))
fmt.Printf("Export File: %s\n", viper.GetString("fileExport"))
fmt.Printf("Export Format: %s\n", viper.GetString("fileFormat"))
fmt.Printf("Consumer API Key: %s\n", viper.GetString("consumer_api_key")[0:6])
fmt.Printf("Consumer API Secret: %s\n", viper.GetString("consumer_api_secret")[0:6])
},
}

view raw

config.go

hosted with ❤ by GitHub

The parse command was a small bit changed. A fair amount of the functionality I refactored out to the buildTwitterList() and exportFile, and rebuildForExport functions. The buildTwitterList() I put in the helper.go file, which I’ll cover a littler later. But in this file, which could still use some refactoring which I’ll get to, I have several pieces of functionality; the export to formats functions, and the if else if logic of the exportParsedTwitterList function.


var parseCmd = &cobra.Command{
Use: "parse",
Short: "This command will extract the Twitter Accounts form a text file.",
Long: `This command will extract the Twitter Accounts and clean up or disregard other characters
or text around the twitter accounts to create a simple, clean, Twitter Accounts only list.`,
Run: func(cmd *cobra.Command, args []string) {
completedTwittererList := buildTwitterList()
fmt.Println(completedTwittererList)
if viper.Get("fileExport") != nil {
exportParsedTwitterList(viper.GetString("fileExport"), viper.GetString("fileFormat"), completedTwittererList)
}
},
}
func exportParsedTwitterList(exportFilename string, exportFormat string, twittererList []string) {
if exportFormat == "txt" {
exportTxt(exportFilename, twittererList, exportFormat)
} else if exportFormat == "json" {
exportJson(exportFilename, twittererList, exportFormat)
} else if exportFormat == "xml" {
exportXml(exportFilename, twittererList, exportFormat)
} else if exportFormat == "csv" {
exportCsv(exportFilename, twittererList, exportFormat)
} else {
fmt.Println("Export type unsupported.")
}
}
func exportXml(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting xml export to %s.", exportFilename)
xmlContent, err := xml.Marshal(twittererList)
check(err)
header := xml.Header
collectedContent := header + string(xmlContent)
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportCsv(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting txt export to %s.", exportFilename)
collectedContent := rebuildForExport(twittererList, ",")
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportTxt(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting %s export to %s.", exportFormat, exportFilename)
collectedContent := rebuildForExport(twittererList, "\n")
exportFile(collectedContent, exportFilename+"."+exportFormat)
}
func exportJson(exportFilename string, twittererList []string, exportFormat string) {
fmt.Printf("Starting %s export to %s.", exportFormat, exportFilename)
collectedContent := collectContent(twittererList)
exportFile(string(collectedContent), exportFilename+"."+exportFormat)
}
func collectContent(twittererList []string) []byte {
collectedContent, err := json.Marshal(twittererList)
check(err)
return collectedContent
}
func rebuildForExport(twittererList []string, concat string) string {
var collectedContent string
for _, twitterAccount := range twittererList {
collectedContent = collectedContent + concat + twitterAccount
}
if concat == "," {
collectedContent = strings.TrimLeft(collectedContent, concat)
}
return collectedContent
}
func exportFile(collectedContent string, exportFile string) {
contentBytes := []byte(collectedContent)
err := ioutil.WriteFile(exportFile, contentBytes, 0644)
check(err)
}

view raw

parse.go

hosted with ❤ by GitHub

Next up after parse, it seems fitting to cover the helpers.go file code. First I have the check function, which simply wraps the routinely copied error handling code snippet. Check out the file directly for that. Then below that I have the buildTwitterList() function which gets the config setting for the file name to open to parse for Twitter accounts. Then the code reads the file, splits the results of the text file into fields, then steps through and parses out the Twitter accounts. This is done with a REGEX (I know I know now I have two problems, but hey, this is super simple!). It basically finds fields that start with an @ and then verifies the alphanumeric nature, combined with a possible underscore, that then remove unnecessary characters on those fields. Wrapping all that up by putting the fields into a string/slice array and returning that string array to the calling code.


func buildTwitterList() []string {
theFile := viper.GetString("file")
theTwitterers, err := ioutil.ReadFile(theFile)
check(err)
stringTwitterers := string(theTwitterers[:])
splitFields := strings.Fields(stringTwitterers)
var completedTwittererList []string
for _, aField := range splitFields {
if strings.HasPrefix(aField, "@") && aField != "@" {
reg, _ := regexp.Compile("[^a-zA-Z0-9_@]")
processedString := reg.ReplaceAllString(aField, "")
completedTwittererList = append(completedTwittererList, processedString)
}
}
return completedTwittererList
}

The next function in the Helpers.go file is the getBearerToken function. This was a tricky bit of code. This function takes in the consumer key and secret from the Twitter app (check out the video at 5:20 for where to set it up). It returns a string and error, empty string if there’s an error, as shown below.

The code starts out with establishing a POST request against the Twitter API, asking for a token and passing the client credentials. Catches an error if that doesn’t work out, but if it can the code then sets up the b64Token variable with the standard encoding functionality when it receives the token string byte array ( lines 9 and 10). After that the request then has the header built based on the needed authoriztaion and content-type properties (properties, values? I don’t recall what spec calls these), then the request is made with http.DefaultClient.Do(req). The response is returned, or error and empty response (or nil? I didn’t check the exact function signature logic). Next up is the defer to ensure the response is closed when everything is done.

Next up the JSON result is parsed (unmarshalled) into the v struct which I now realize as I write this I probably ought to rename to something that isn’t a single letter. But it works for now, and v has the pertinent AccessToken variable which is then returned.


func getBearerToken(consumerKey, consumerSecret string) (string, error) {
req, err := http.NewRequest("POST", "https://api.twitter.com/oauth2/token",
strings.NewReader("grant_type=client_credentials"))
if err != nil {
return "", fmt.Errorf("cannot create /token request: %+v", err)
}
b64Token := base64.StdEncoding.EncodeToString(
[]byte(fmt.Sprintf("%s:%s", consumerKey, consumerSecret)))
req.Header.Add("Authorization", "Basic "+b64Token)
req.Header.Add("Content-Type", "application/x-www-form-urlencoded;charset=UTF-8")
resp, err := http.DefaultClient.Do(req)
if err != nil {
return "", fmt.Errorf("/token request failed: %+v", err)
}
defer resp.Body.Close()
var v struct {
AccessToken string `json:"access_token"`
}
if err := json.NewDecoder(resp.Body).Decode(&v); err != nil {
return "", fmt.Errorf("error parsing json in /token response: %+v", err)
}
if v.AccessToken == "" {
return "", fmt.Errorf("/token response does not have access_token")
}
return v.AccessToken, nil
}

Wow, ok, that’s a fair bit of work. Up next, the findem.go file and related function for twitz. Here I start off with a few informative prints to the console just to know where the CLI has gotten to at certain points. The twitter list is put together, reusing that same function – yay code reuse right! Then the access token is retrieved. Next up the http client is built, the twitter client is passed that and initialized, and the user lookup request is sent. Finally the users are printed out and below that a count and print out of the count of users is printed.


var findemCmd = &cobra.Command{
Use: "findem",
Short: "A brief description of your command",
Long: `A longer description that spans multiple lines and likely contains examples
and usage of using your command. For example:
Cobra is a CLI library for Go that empowers applications.
This application is a tool to generate the needed files
to quickly create a Cobra application.`,
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("Starting Twitter Information Retrieval.")
completedTwitterList := buildTwitterList()
fmt.Printf("Getting Twitter details for: \n%s", completedTwitterList)
accessToken, err := getBearerToken(viper.GetString("consumer_api_key"), viper.GetString("consumer_api_secret"))
check(err)
config := &oauth2.Config{}
token := &oauth2.Token{AccessToken: accessToken}
// OAuth2 http.Client will automatically authorize Requests
httpClient := config.Client(context.Background(), token)
// Twitter client
client := twitter.NewClient(httpClient)
// users lookup
userLookupParams := &twitter.UserLookupParams{ScreenName: completedTwitterList}
users, _, _ := client.Users.Lookup(userLookupParams)
fmt.Printf("\n\nUsers:\n%+v\n", users)
howManyUsersFound := len(users)
fmt.Println(howManyUsersFound)
},
}

view raw

Findem.go

hosted with ❤ by GitHub

I realized, just as I wrapped this up I completely spaced on the Apache Cassandra export. I’ll have those post coming soon and will likely do another refactor to get the output into a more usable state before I call this one done. But the core functionality, setup of the systemic environment needed for the tool, the pertinent data and API access, and other elements are done. For now, that’s a wrap, if you’re curious about the final refactor and the Apache Cassandra export then subscribe to my Twitch @adronhall and/or my YouTube channel ThrashingCode.

UPDATED SERIES PARTS

    1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
    2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation
    3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval (this post)