Golang Execution on OS-X vs. Linux

Recently I was dorking about with some Go code on OS-X. Just working with it via iTerm 2 and using some basic go run whateverthefile.go commands and displaying results out the the shell. All was well, and then I dived into some code that used so Go Routines. The code looked something like this.

package main

import (
    "time"
    "fmt"
)

func say(s string) {
    for i := 0; i<5; i++ {
        time.Sleep(2 * time.Second)
        fmt.Println(s)
    }
}

func main() {
    go say("1. First thread is running.")
    go say("  2. Second thread is running.")
    say("    3. Thread is running in main.")
    fmt.Println("Ended.")
}

That was fine. I ran the code and the results were as expected.

OSX$   2. Second thread is running.
    3. Thread is running in main.
1. First thread is running.
1. First thread is running.
  2. Second thread is running.
    3. Thread is running in main.
    3. Thread is running in main.
1. First thread is running.
  2. Second thread is running.
  2. Second thread is running.
    3. Thread is running in main.
1. First thread is running.
1. First thread is running.
  2. Second thread is running.
    3. Thread is running in main.
Ended.

Then I ran it again. Also these results were expected.

OSX$     3. Thread is running in main.
  2. Second thread is running.
1. First thread is running.
    3. Thread is running in main.
1. First thread is running.
  2. Second thread is running.
  2. Second thread is running.
    3. Thread is running in main.
1. First thread is running.
1. First thread is running.
    3. Thread is running in main.
  2. Second thread is running.
    3. Thread is running in main.
Ended.
1. First thread is running.

You’ll notice the slightly out of order response as each thread completes. With the first set of results the last response is “Ended.” and with the second set of results “Ended.” is second to last. This of course is expected as the threads are likely to be executing while the last line of code is executed. But, there’s a catch to this depending where you run this code.

On the MacOS/OS-X platforms, when you run the shell it runs certain things with an assumed “&” after the command. Now, I’m not sure the specifics of what is happening (I’m looking it up and will happily post a super detailed reference link if you’ve got one, lemme know @Adron.) The “&” is however used on Linux/Unix systems to designate to the shell, run this command on the background so that the user remains in control of the shell. Specifically, from the man pages.

If a command is terminated by the control operator &, the shell executes the command in the background in a subshell. The shell does not wait for the command to finish, and the return status is 0.

The reason I bring this up, is if I run the same thing and those same results come back on Linux or Unix, you would not see anything after the “Ended.” runs, because the application terminates at that point and none of the background threads are actively connected to the shell. In other words the result where “Ended.” occurs second to last would look like this.

LINUX-SHELL$     3. Thread is running in main.
  2. Second thread is running.
1. First thread is running.
    3. Thread is running in main.
1. First thread is running.
  2. Second thread is running.
  2. Second thread is running.
    3. Thread is running in main.
1. First thread is running.
1. First thread is running.
    3. Thread is running in main.
  2. Second thread is running.
    3. Thread is running in main.
Ended.

At first glance, it might seem like the application has run successfully, but look closely and count the number of responses from each thread. In this example, the shell has lost priority when the “Ended.” message appears and there is only four messages that read “1. First thread is running.”. However, after the “Ended.” message there should have been another one, like the second example above that I ran on OS-X. On Linux, the solution to executing this example is actually to add the ampersand at the end of the command.

go run whateverthefile.go & and then you’ll see the execution of the program with the correct results.

Well that’s it for this often overlooked and confusing behavior on Linux versus OS-X. Happy hacking, cheers!

Addition/Correction April 3rd, 2017: It was brought to my attention that this is very specifically the shell, not the operating system. This is an important thing to note, and sometimes is easily confused or mistaken outright. The effect can even be different behavior between shell use on the same operating system, regardless of OS-X, Linux, UNIX, or whatever.

Industry Introspection

farnsworth-in-chair.jpgEvery few days I like to sit down like an old academic professor and just read and ponder what’s going on in the computer hardware and software industries. As of late it’s been interesting to dive into whatever Elon Musk is working on also; electric cars, solar energy, rockets, or even boring machines. Another thing I’ve had a curiosity in and continue to find interesting is machine learning, or as some call it in the mainstream movie media artificial intelligence.

I’ll tackle some of these technologies real quick so we all have a current situational report. Each of these topics seems to be a area of work that could use some extensive clarification. I’ll provide some of my own thoughts along with a few links of reference to dig in deeper, so that one doesn’t fall into the trap of uniformed oblivious media consumer.

Rockets (i.e. Space)

My History: I haven’t always paid attention to this space because I never had intention of working in the area, however who isn’t interested in rockets in some way. Well, considering my desire for other work, I was extremely fortunate where I grew up to be involved in rocketry. I grew up in Picayune, Mississippi which is a mere ~15-20 miles away from John C. Stennis Space Center.

I had the joy of experiencing rocket tests at John C. Stennis and seeing research into rockets as a child, and on some of my first paid computer related gigs. As one might suspect, they use computers to do rocket research and help with launches. Shocker right!

My Thoughts: The space we’re in right now is impressive. The market is actually getting involved in launching it’s own rockets, which means we’re likely only years or maybe a decade or two of having an economically sustainable rocket program. Hopefully NASA can work on more focused deep space missions now while Musk’s Space-X and others refine and perfect orbital rockets for satellites and all that mess.

SITREP:

Machine Learning

farnsworth-future-of-ai-mlMy History: I’ve toyed with machine learning on and off again, working on pathing algorithms for objects to decide travel patterns to supervised learning algorithms. In the end I’ve generally ended up working on other things in my day to day work but I know this will be changing in the near future (next year or three). It’s an extremely interesting space of work and research.

My Thoughts: First, getting AI & ML (That’s artificial integlligence and machine learning) conflated, especially in the media, is starting to reflect the popularity of said space in the software industry. However, for the most part these two things are effectively the same thing. It’s just different words describing the industry space where we’re trying to make machines make decisions we deem intelligent based on available data.

That actually leads to many other discussions. What do we as humans deem intelligent and what happens when available data isn’t enough? But more words on that for another day. I know the questions are burning in the mind of every chief executive of something that wants this mythical AI they keep hearing about and paying voluminous amounts of money to their big data bad ass data science ninja architects to implement but rarely have answers for all of it.

The SITREP:

Overall there’s a ton of material ending up on the web related to AI/ML, and my top suggestion is to start googling so you can pick and choose which aspects you want to read about in this space. One could dive in via the super technical aspect of how systems work that are being used for processing, how the algorithms work, or even working with data to model good decision results from data sets (i.e. diving into training). But it’s really a space that is awash in resources. Dive in!

End Words

That’s it for my industry introspections for now. If you’re interested in reading about this, programming material, and related topics subscribe, follow, or RSS feed read the ole’ blog here.

Thrashing Code Metal Monday for Week of February 13th, 2017

I’ve been getting a fair dose of New Orleans cajun grub lately so I went back to my roots and picked out a few bands from NOLA (That’s New Orleans, Louisiana FYI). Be prepared, NOLA Metal is a special level of sludgy heaviness and growl combined with some slightly psychedelic melodies thrown in.

dungeon2

With that, enjoy your coding heavy Monday metal!

Top 3 for Spotify

https://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A3xtIpqzIOfQUxKce8BU4Ka&size=detail&theme=dark https://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A1m0B9ak05G0jqDY4ACLhQu&size=detail&theme=darkhttps://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A5kuYamMO00pHPdRQcAXWTl&size=detail&theme=dark

Top 3 Videos for Youtube

Thrashing Code Metal Monday for Week of February 6th, 2017

This is the first of many of a series that is starting today. Thrashing Code Metal Monday’s picks for heavy tunes to code to. Ya know, if you can handle it.

With that introduction, may the thrashing code begin!

Top 3 for Spotify

https://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A5ctFffJBdJe8PZL7W7NeML&size=detail&theme=dark https://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A67ZMMtA88DDO0gTuRrzGjn&size=detail&theme=darkhttps://embed.spotify.com/follow/1/?uri=spotify%3Aartist%3A3dnH7fdVm2X07MK6Fkbhbt&size=detail&theme=dark

Top 3 Videos for Youtube

Data Diluvium Design Ideas

This post includes a collection of my thoughts on design and architecture of a data generation service project I’ve started called Data Diluvium. I’m very open to changes, new ideas, or completely different paradigms around these plans altogether. You can jump into the conversation thread. What kind of data do you often need? What systems do you want it inserted into?

Breakdown of Article Ideas:

  • Collected Systems API – This API service idea revolves around a request that accepts a schema type for a particular database type, an export source for inserting the data into, and generating an amount of data per the requested amount. The response then initiates that data generation, while responding with a received message and confirmation based on what it has received.
  • Individual Request API – This API service idea (thanks to Dave Curylo for this one, posted in the thread) revolves around the generation of data, requested at end points for a particular type of random data generation.

Alright, time to dive deeper into each of these.

Collected Systems APIs

  • https://datadiluvium.com/schema/generate – This API end point would take a schema with the various properties needed. For any that aren’t set, a default would be set. The generation process would then randomize, generate, and insert this data into any destination source specified. Here are some prospective examples I’ve created:

    A very basic sample JSON schema

    [
      {
        "schema": "relational",
        "database": "text"
      }
    ]
    

    In this particular example, I’ve created the simplist schema that could be sent into the service. For this particular situation I’d have (currently not decided yet) defaults that would randomly create a table, with a single column, and generate one element of data in that table. Other properties could be set, which would give control over the structure created in which to insert the data into. An example would be the following.

    [
      {
        "schema": "relational",
        "database": "postgresql",
        "structure": [
          {
            "table": "Users",
            "columns": [
              {"name": "id", "type": "uuid"},
              {"name": "firstname", "type": "firstname"},
              {"name": "lastname", "type": "lastname"},
              {"name": "email_address", "type": "email"}
            ]
          },
          {
            "table": "Addresses",
            "columns": [
              {"name": "id", "type": "uuid"},
              {"name": "street", "type": "address"},
              {"name": "city", "type": "city"},
              {"name": "state", "type": "state"},
              {"name": "postalcode", "type": "zip"}
            ]
          },
          {
            "table": "Transactions",
            "columns": [
              { "name": "id", "type": "uuid" },
              { "name": "transaction", "type": "money" },
              { "name": "stamp", "type": "date" }
            ]
          }
        ]
      }
    ]
    

    In this example, the properties included are three tables; UsersAddresses, and Transactions. In the first table, Users, the columsn would be; idfirstnamelastname, and email_address. Each of these then have a type property which sets the type of data to be generated for the columns. The same type of set of properties is then included for the Addressesand Transactions tables and their respective columns.

    Some additional questions remain, such as if the tables exist in the database, would the insertion build SQL to create the tables? Should it be assumed that the tables exist already and have the appropriate settings set to insert the data into the tables? Again, a great thing to discuss on the thread here.

  • https://datadiluvium.com/schema/validate – This could be used to validate a schema request body. Simply submit a schema and a validation response would be returned with “Valid” or “Invalid”. In the case of an invalid response, a list of prospective and known errors would be returned.

These two API end points focus around building out large data to test systemic environments and the respective construction of those environments. The actual generation of the data is assumed for this API service and the individual generation of data is discussed below in the individual request APIs.

Individual Request APIs

The following API calls could be implemented with fairly straight forward random data generation. A number can easily be randomized and returned, a word can be chosen from a dictionary, and a city returned from a list of cities. The following are prospective API calls to return data of this type.

The next level of complexity for data generation would be the slightly structured data generation. Instead of having an arbitrary list of addresses, we could or would prospectively generate one. But on the other hand, maybe we should just randomly create actual addresses that can be validated against an actual real address? That seems to have the possiblity of issues in the real world in spite of the fact all the addresses out there in the world are basically publicly accessible data. But questioning how or what the data would or could actually represent will be a great thing to discuss in the thread.

The next level of data generation complexity would be to generate sentences and other related data. This could be done a number of ways. If we wanted to have it generate intelligent sentences that made sense, it would take a little bit more work then for example generating lorum ipsum.

TLDR;

This blog entry just details the starting point of features for the Data Diluvium Project. If you’d like to jump into the project too, let me know. I’m generally working on the project during the weekends and a little during the week. There’s already a building project base that I’m starting with. If you’re interested in writing some F#, check out the work Dave Curylo has done here. I’ve been pondering breaking it out to another project and sticking to the idea of microservice but with F# for the work he put in. Anyway if you’ve got ideas on how to generate data, how you’d like to use it in your applications, or other related ideas please dive into the conversation on the Github Thread here.