Tag Archives: twitter

Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

Part 3 of 3 – Coding Session in Go – Cobra + Viper CLI for Parsing Text Files, Retrieval of Twitter Data, and Exports to various file formats.

UPDATED PARTS:

  1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
  2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation
  3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval (this post)

Updated links to each part will be posted at bottom of  this post when I publish them. For code, written walk through, and the like scroll down below the video and timestamps.

0:54 The thrashing introduction.
3:40 Getting started, with a recap of the previous sessions but I’ve not got the sound on so ignore this until 5:20.
5:20 I notice, and turn on the volume. Now I manage to get the recap, talking about some of the issues with the Twitter API. I step through setup of the app and getting the appropriate ID’s and such for the Twitter API Keys and Secrets.
9:12 I open up the code base, and review where the previous sessions got us to. Using Cobra w/ Go, parsing and refactoring that was previously done.
10:30 Here I talk about configuration again and the specifics of getting it setup for running the application.
12:50 Talking about Go’s fatal panic I was getting. The dependency reference to Github for the application was different than what is in application and don’t show the code that is actually executing. I show a quick fix and move on.
17:12 Back to the Twitter API use by using the go-twitter library. Here I review the issue and what the fix was for another issue I was having previous session with getting the active token! Thought the library handled it but that wasn’t the case!
19:26 Now I step through creating a function to get the active oath bearer token to use.
28:30 After deleting much of the code that doesn’t work from the last session, I go about writing the code around handling the retrieval of Twitter results for various passed in Twitter Accounts.

The bulk of the next section is where I work through a number of functions, a little refactoring, and answering some questions from the audience/Twitch Chat (working on a way to get it into the video!), fighting with some dependency tree issues, and a whole slew of silliness. Once that wraps up I get some things committed into the Github repo and wrap up the core functionality of the Twitz Application.

58:00 Reviewing some of the other examples in the go-twitter library repo. I also do a quick review of the other function calls form the library that take action against the Twitter API.
59:40 One of the PR’s I submitted to the project itself I review and merge into the repo that adds documentation and a build badge for the README.md.
1:02:48 Here I add some more information about the configuration settings to the README.md file.

1:05:48 The Twitz page is now updated: https://adron.github.io/twitz/
1:06:48 Setup of the continuous integration for the project on Travis CI itself: https://travis-ci.org/Adron/twitz
1:08:58 Setup fo the actual travis.yml file for Go. After this I go through a few stages of troubleshooting getitng the build going, with some white space in the ole’ yaml file and such. Including also, the famous casing issue! Ugh!
1:26:20 Here I start a wrap up of what is accomplished in this session.

NOTE: Yes, I realize I spaced and forgot the feature where I export it out to Apache Cassandra. Yes, I will indeed have a future stream where I build out the part that exports the responses to Apache Cassandra! So subcribe, stay tuned, and I’ll get that one done ASAP!!!

1:31:10 Further CI troubleshooting as one build is green and one build is yellow. More CI troubleshooting! Learn about the travis yaml here.
1:34:32 Finished, just the bad ass outtro now!

The Codez

In the previous posts I outlined two specific functions that were built out:

  • Part 1 – The config function for the twitz config command.
  • Part 2 – The parse function for the twitz parse command.

In this post I focused on updating both of these and adding additional functions for the bearer token retrieval for auth and ident against the Twitter API and other functionality. Let’s take a look at what the functions looked like and read like after this last session wrap up.

The config command basically ended up being 5 lines of fmt.Printf functions to print out pertinent configuration values and environment variables that are needed for the CLI to be used.

The parse command was a small bit changed. A fair amount of the functionality I refactored out to the buildTwitterList() and exportFile, and rebuildForExport functions. The buildTwitterList() I put in the helper.go file, which I’ll cover a littler later. But in this file, which could still use some refactoring which I’ll get to, I have several pieces of functionality; the export to formats functions, and the if else if logic of the exportParsedTwitterList function.

Next up after parse, it seems fitting to cover the helpers.go file code. First I have the check function, which simply wraps the routinely copied error handling code snippet. Check out the file directly for that. Then below that I have the buildTwitterList() function which gets the config setting for the file name to open to parse for Twitter accounts. Then the code reads the file, splits the results of the text file into fields, then steps through and parses out the Twitter accounts. This is done with a REGEX (I know I know now I have two problems, but hey, this is super simple!). It basically finds fields that start with an @ and then verifies the alphanumeric nature, combined with a possible underscore, that then remove unnecessary characters on those fields. Wrapping all that up by putting the fields into a string/slice array and returning that string array to the calling code.

The next function in the Helpers.go file is the getBearerToken function. This was a tricky bit of code. This function takes in the consumer key and secret from the Twitter app (check out the video at 5:20 for where to set it up). It returns a string and error, empty string if there’s an error, as shown below.

The code starts out with establishing a POST request against the Twitter API, asking for a token and passing the client credentials. Catches an error if that doesn’t work out, but if it can the code then sets up the b64Token variable with the standard encoding functionality when it receives the token string byte array ( lines 9 and 10). After that the request then has the header built based on the needed authoriztaion and content-type properties (properties, values? I don’t recall what spec calls these), then the request is made with http.DefaultClient.Do(req). The response is returned, or error and empty response (or nil? I didn’t check the exact function signature logic). Next up is the defer to ensure the response is closed when everything is done.

Next up the JSON result is parsed (unmarshalled) into the v struct which I now realize as I write this I probably ought to rename to something that isn’t a single letter. But it works for now, and v has the pertinent AccessToken variable which is then returned.

Wow, ok, that’s a fair bit of work. Up next, the findem.go file and related function for twitz. Here I start off with a few informative prints to the console just to know where the CLI has gotten to at certain points. The twitter list is put together, reusing that same function – yay code reuse right! Then the access token is retrieved. Next up the http client is built, the twitter client is passed that and initialized, and the user lookup request is sent. Finally the users are printed out and below that a count and print out of the count of users is printed.

I realized, just as I wrapped this up I completely spaced on the Apache Cassandra export. I’ll have those post coming soon and will likely do another refactor to get the output into a more usable state before I call this one done. But the core functionality, setup of the systemic environment needed for the tool, the pertinent data and API access, and other elements are done. For now, that’s a wrap, if you’re curious about the final refactor and the Apache Cassandra export then subscribe to my Twitch @adronhall and/or my YouTube channel ThrashingCode.

UPDATED SERIES PARTS

    1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
    2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation
    3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval (this post)

     

Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation

Part 2 of 3 – Coding Session in Go – Cobra + Viper CLI for Parsing Text Files, Retrieval of Twitter Data, Exports to various file formats, and export to Apache Cassandra.

UPDATED PARTS:

  1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
  2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation (this post)
  3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

Updated links to each part will be posted at bottom of  this post when I publish them. For code, written walk through, and the like scroll down below the video and timestamps.

Hacking Together a CLI Installing Cassandra, Setting Up the Twitter API, ENV Vars, etc.

0:04 Kick ass intro. Just the standard rocking tune.

3:40 A quick recap. Check out the previous write “Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files” of this series.

4:30 Beginning of completion of twitz parse command for exporting out to XML, JSON, and CSV (already did the text export previous session). This segment also includes a number of refactorings to clean up the functions, break out the control structures and make the code more readable.

In the end of refactoring twitz parse came out like this. The completed list is put together by calling the buildTwitterList() function which is actually in the helpers.go file. Then prints that list out as is, and checks to see if a file export should be done. If there is a configuration setting set for file export then that process starts with a call to exportParsedTwitterList(exportFilename string, exportFormat string, ... etc ... ). Then a simple single level control if then else structure to determine which format to export the data to, and a call to the respective export function to do the actual export of data and writing of the file to the underlying system. There’s some more refactoring that could be done, but for now, this is cleaned up pretty nicely considering the splattering of code I started with at first.

50:00 I walk through a quick install of an Apache Cassandra single node that I’ll use for development use later. I also show quickly how to start and stop post-installation.

Reference: Apache Cassandra, Download Page, and Installation Instructions.

53:50 Choosing the go-twitter API library for Go. I look at a few real quickly just to insure that is the library I want to use.

Reference: go-twitter library

56:35 At this point I go through how I set a Twitter App within the API interface. This is a key part of the series where I take a look at the consumer keys and access token and access token secrets and where they’re at in the Twitter interface and how one needs to reset them if they just showed the keys on a stream (like I just did, shockers!)

57:55 Here I discuss and show where to setup the environment variables inside of Goland IDE to building and execution of the CLI. Once these are setup they’ll be the main mechanism I use in the IDE to test the CLI as I go through building out further features.

1:00:18 Updating the twitz config command to show the keys that we just added as environment variables. I set these up also with some string parsing and cutting off the end of the secrets so that the whole variable value isn’t shown but just enough to confirm that it is indeed a set configuration or environment variable.

1:16:53 At this point I work through some additional refactoring of functions to clean up some of the code mess that exists. Using Goland’s extract method feature and other tooling I work through several refactoring efforts that clean up the code.

1:23:17 Copying a build configuration in Goland. A handy little thing to know you can do when you have a bunch of build configuration options.

1:37:32 At this part of the video I look at the app-auth example in the code library, but I gotta add the caveat, I run into problems using the exact example. But I work through it and get to the first error messages that anybody would get to pending they’re using the same examples. I get them fixed however in the next session, this segment of the video however provides a basis for my pending PR’s and related work I’ll submit to the repo.

The remainder of the video is trying to figure out what is or isn’t exactly happening with the error.

I’ll include the working findem code in the next post on this series. Until then, watch the wrap up and enjoy!

1:59:20 Wrap up of video and upcoming stream schedule on Twitch.

UPDATED SERIES PARTS

    1. Twitz Coding Session in Go – Cobra + Viper CLI for Parsing Text Files
    2. Twitz Coding Session in Go – Cobra + Viper CLI with Initial Twitter + Cassandra Installation (this post)
    3. Twitz Coding Session in Go – Cobra + Viper CLI Wrap Up + Twitter Data Retrieval

 

Twitter for Developers, Cutting the Bullshit, Quelling the Trash Tire Fire

It’s been over a decade that Twitter has been an active part of the developer community. It’s grown in popularity from day one, and now holds the uneasy crown as the place for hot takes, trash from politicians, and the general tire fire that is the news. In many ways, that’s what they’ve aimed for. But then there’s us developers, people who make software, who make Twitter, who build all of this technology internet stuff right? We’re here using Twitter still, even amid the backstabbing and Twitter UI’s API’s being yanked from under us. They’ve of course in the past also banned UI’s and somehow here we are still using the service. However, I digress, Twitter’s wrongs against developers are numerous after we effectively built the service. In spite of all this we developers are a large contingent of people on Twitter. It’s still an amazingly useful medium for software developers, and especially new software developers, to get involved with. It’s a very effective tool to strengthen our careers and continue conversations within the developer communities themselves. One just has to avoid the cruft, and that’s what I intent to tackle some of in this article.

This list I’ve put together is of things that I personally have learned, often by stumbling through and discovering myself. These activities on Twitter do have a net positive effect on your career and ability to communicate with the world and local developer communities. First I’ll cover positive use cases of Twitter that are immensely useful as a software developer. These are even compounded if you’re an advocate of open source, cool technologies and libraries, and other miscellaneous things.

1. Twitter as a Communication Tool

First and foremost, Twitter has been and does – mostly – continue to be a communication tool. I make use of Twitter to connect with people for conference organizing, code projects, open source work, to have geek lunch, nerd brunch, and many other things that come up. It can and ought to be one of your primary communication mediums in that it connects many of the key active people within our overall communities. More so than email and other mediums by a large percent. If you intend to have a long term net effect and grow your presence and activities (conferences, meetups, coding groups, etc) you want to foster Twitter has become the de facto medium to be active on.

2. Twitter as a Collector of People

Twitter, even though it does seem to attract some of the most villainous scum (literally, not a figure of speech or hyperbole) and have some pretty horrifying problems (people calling in SWAT’s on people (extremely illegal), death threats, harassment) the net benefit within the community to bring people together has far outclassed pretty much any other system out there. Hacker News doesn’t, Facebook doesn’t, Google+ is cancelled, and about every other social media platform has failed to bring together the develop community in an effective and useful way.

3. Twitter for Answers

Even though I don’t often go to Twitter to find answers, sometimes I do. Often it is a last resort. After all, Twitter is most efficient at providing a place for links, quick blurbs, bumbling and babbling threads from people, and of course cat pictures and hot takes.

The combination powers of Twitter with other services however exponentially increases the ability of Twitter to help with answers. For example, write up a solidly written question on Stackoverflow or one of the branched out services and then post the question on Twitter, maybe inquiring for some retweets and boom, doubling, tripling, and greater multiplier of people looking at the question that can provide a prospective answer!

4. Twitter, Firestarter

One of the things I’ve also found Twitter good for is an outlet for pushing and often straightening out bad behavior in the community. Ever done something racist? Ever known someone to pull some misogynistic action? Yeah, unfortunately I know of these things happening too, and Twitter forces apologies and better behavior among people. But it also is a place people can wreck themselves and be just as destructive as they can learn to better themselves, especially those humans of us that have poor behavior and disrespectful tendencies.

But just as much as individual behaviors among us, Twitter has been used to straighten out some pretty trash behaviors from corporations. Sure, they’re not really people, but the conflagrations of this notion – true or not – make for pressure to be applied to corporations through other means besides the products and services they sell us individual humans, which to often are things we have to buy regardless, and this medium provides us an avenue to induce better behaviors in spite of purchases.

There of course is the positive and negative of this forced societal behavior and in many ways, improving corporate behavior throughout the world, but it’s here. Pressure of the people, often organized and started through Twitter, including against Twitter itself sometimes, is heavily rooted in activity right there on ole’ Twitter itself.

GSD Tactical Twitter

Alright, now to the meat of things. Twitter is great at all these things but how does one make the best use of it without it turning into an outright tire fire trash dump of distraction and stress? Well, it’s moderately easy, but one has to be careful.

1. Find Good and Entertaining People

My personal advice when starting on Twitter is to skip the companies. Don’t follow any of them. Same goes for organizations or any group account of sorts. The key to find good content, good common ground, and useful links, news, and related communities is to follow individuals that are involved in those things you want to be involved in already. The following are some specific examples, and for me, great people to follow.

2. Lift Up Others, Tweet to Others, Get Involved

When on Twitter, one can just lurk. It’s a completely valid thing to do. However lurking isn’t super high value. You just won’t get that much out of it. Instead, get involved. Find a link with something interesting, write up a tweet and post it. See something interesting someone else just tweeted, respond! See something that isn’t right, maybe tweet why it isn’t.

Always a good idea, regardless of the trash that is often on Twitter to still stay courteous, kind, and friendly. Remember, not everyone is from the mold you’ve come from, or seen things the way you have, so tread lightly and friendly and things mostly work out real well. Overall, people are attuned to helping those that help themselves and helping those that we run in social circles with.

All in all, get involved, tweet at, with, and all around your fellow Twitterers. Your return will improve and in the process you’ll add more value for others too.

3. Follow & Prune the Firehose of Tweets

Alright, I’ve written to follow and lift up others. That’s groovy, but also you gotta bring the hammer down sometimes. When that firehose of tweets just gets a little overwhelming check out what tweets are helpful, rate them to yourself, and unfollow some people if it’s not the direction or the tweets you’re getting value from.

Even though it’s difficult when just starting to use Twitter, the ratio will be more followed than followers for you. But as time goes forward and you get past 50 followers, 100, 500, 1000 you’ll need to make sure to keep the list of people you’ve followed just equal to or less than how many people follow you. It’ll help keep your feed manageable and also help you to keep interactions beneficial for you, followers, and followed.

4. The Down-Low on Conferences

If you’re looking to attend a conference, Twitter via hashtags is a great way to get information on conferences. Dig in, dig deep. Talk to people about the conference in particular. If necessary get into direct messages and invoke the whisper net if need be. Sometimes conferences can be exponentially useful and sometimes they end up bothersome cash burning wastes of time. Figure out what you want from a prospective conference and dig in via Twitter, you’ll prevent wasting time and burning cash, and exponentially increase the positives you can get out of a conference.

5. Filter the Trash Fire

Ok, let’s get super serious. One way Twitter has become a trash fire for many or most people these days is because of the political trash dumped in. Much of Twitter for the general public is bot armies from Russia, crazies like the nutty Wohl kid, and other junk nut accounts. One way to notch this down to a minimal trash fire is to throw some filters (i.e. mute certain words) on your Twitter account. For example here’s my list:

filters-muted-words

Now as you’ve read that, remember that my goal has been to focus the stream on tech content with a little heavy metal, a few cats, and other entertainment here and there. For example I’m fine with sports events like baseball and football but really don’t want to get distracted by it in on my Twitter stream. On game day those events just overwhelm the tweets and things that are useful get drowned out.

Now a lot of the other stuff in the list is the horrifying reality of the United States today, reflected on Twitter, and part of something that I don’t want distracting me either. Overall this has made Twitter dramatically more useful for me again.

Putting Together Medium Lists of Mediums

Recently I tweeted this, mostly in jest of the plethora and confusion of mediums in which to publish content, but also as a kind of reminder to get things straight for myself. Then, by order of that get something written so I could tell others where I put what, and what is what that I publish. Here’s the details.

Medium — i.e. this blog, is a place where I’m going to journal about whatever and I do mean whatever. I might write about cycling, tech, coding, city planning, music, or all sorts of things. No topic is going to be off limits but the one overriding theme here is this will be off the cuff, spontaneous, quickly written and probably completely unedited content that I post. More simply, this is going to be where I write out my stream of consciousness. I have no idea, especially since this is a new effort on my behalf, how frequent it may be. I am guessing I may post 1–4 times per week.

Composite Code — This is my technical blog, with a touch of metal monday, and other things related to my technical interests and my technical work. This is mostly going to include things like conferences, meetups, and where I’ll be speaking, technical docs, write ups on how-tos with languages, tech stack deployments, site reliability engineering, and related material. I post a blog entry here about every ~10–15 days I believe. However I am working on making a much more regular post to this blog to the frequency of about 1–3 times per week.

Twitter— This is my twitter feed, it is indeed merely a twitter feed. It includes a host of tech related things, but also I engage, use it as a primary communication medium with the tech community, and sometimes ramble on about a few other random thoughts or post some solid, brutally awesome, heavy as heavy is, heavy metal. Twitter tends to be my highest volume medium in which I post things to, at about 25–30 tweets per day.

Thrashing Code News — This is my newsletter. I use this to get subscribers first details on upcoming conferences I may be involved in organizing, or conferences that I find that are must attend events. I also post some minor summaries of blog entries for the month (sometimes) and also post about upcoming meetups I’ll be speaking at or other travels out and about where we may be able to meetup, hack some code, enjoy a coffee, have a round of beer, eat, or otherwise get together and nerd on tech, code, and related things. I generally publish a new newsletter every 40–60 days. It’s very low volume!

LinkedIn — I don’t really do much here besides receive emails from tons of random recruiters who 94.6% of the time never actually read my profile, but send me stuff for jobs like “C# Code Janitor” and “Trash Fire Putter Outterer”. My real use for LinkedIn tends to boil down to two specific things: one is a place to put work and resume descriptions and such, and two I use it as an way to manage some auth and interactions. I check this about once every 4–20 days.

Andy Piper & Troy Howard, Now Twitter is up to Something!

Twitter is up to something. I’m betting it’s something good.

In the last 2 weeks I’ve found out two fellow coders are rolling into the Twitter family. These two people are top tier talent, so I’m just assuming Twitter had their act together when they went after these two new recruits. So who are these two individuals? Andy Piper and Troy Howard, two people everybody keeps track of. Wait, you do keep track of these guys right? Hmmm, if you don’t it might be high time you need to get in gear and follow them! Here are their deets, so you’re in the loop.

Andy Piper

Andy Piper

Andy Piper @andypiper, heading over to become Developer Advocate in London. Andy has been a great advocate over at Cloud Foundry. I only assume, as many who have used the Cloud Foundry Platform, he’ll continue to be an advocate for it. I’m super excited to see the efforts Andy leads forward with in this new role with Twitter. I’ll be keeping an eye out and hopefully this year landing in London to visit for a few lines of code and a brew or two.

Troy Howard

Troy Howard

Troy Howard @thoward37 is heading over to become the Technical Documentation Super Genius (my label) to which he humbly refers to as Documentarian. He’s helped lead projects like Node PDX Conf (which he and I stumbled ourselves into 2+ years ago) and he’s since knocked out work with organizing Write the Docs,

hujs

hujs

Hujs (check out Glenn Block’s write up) and others! Besides being a mad awesome conference organizer he’s all over the Portland tech community, code space & devops world.

For other trend setters and coders that get shit done and make waves, check out my Awesome Coders category. I’ve introduced more than a few top tier amazing people over the years that I’m totally stoked to have worked along side, hacked with, coded with or otherwise been involved with in the software & hardware industry!

Summary => References =>

So begs the question, “what’s Twitter up to eh?

Twitter Peoplez

I always see these #ff or #followfriday lists on Twitter.  I figured I’d put together a list, then I ended up creating another list, and another.  It appeared I was past my 140 character limit so here are those lists of Twitterers to follow.

Amazon Web Services Twitterers

Windows Azure Twitterers

Silverlight Twitterers

.NETters (No particular groupings…)

With that list I ponder – anyone have any additional suggestions for follows in these or other software development related areas?