Author Archives: Adron

About Adron

See: https://compositecode.blog/about

ML Spends A Year In Burgundy with Jon Oropeza at ML4ALL

We’re building up to ML4ALL 2019, and in the meanwhile I want to re-introduce some of the past speakers and show you their talks. This first, of the many, is Jon Oropeza.

The Talk

A Summary of ML Spends a Year in Burgundy

pic1

Jon starts his presentation journey with a childhood story. It involves computers and things, his childhood girlfriend, thumb wars, and other fascinating things. Give it a watch, get enthralled with a first step into machine learning for a professional web application developer, Jon.

pic2.pngJon introduces us via this story, to some of his first experiments with machine learning. He had a great adventure a while back with Burgundy and wine, and found lot’s of data on the wine of this and other sorts. He went to work finding out the best way to find how machines could determine a place and year through a liquid medium; wine. Getting into the wine varietals, the grapes, and other related criteria, where they’re born, and additional data Jon made great progress.

Going further he found ratings and other peripheral data and that he wanted even more, needed even more data, but found some interesting answers along the way. To see, hear, and learn more give his talk a viewing.

This is one of many talks from ML4ALL 2018, I’ll post more here with more reviews, with this being a taste of our aim with ML4ALL 2019. Join us, submit a proposal, or pick up a ticket today! It’s gonna be another create conference.

Distributed Database Things to Know: Gossip

Some of the names used can seem to conflate the actual purpose of a feature’s functionality in distributed databases. However gossip is pretty spot on. Within a group of people gossiping the purpose is to find out each other’s business. What’s going on with Frank, who’s he seeing, and Sally started a business, say what! In the end, all gossippers get into the business and understand what Frank, Sally, and the whole crew are up to. This is a good analogy for what gossip does in a distributed database, or distributed systems in general.

The way gossip works in node, is on a peer-to-peer basis. It’s a communication protocol with the purpose of minding the other nodes business so the singular node gossiping can go about its business. The process runs every second and exchanges state messages between the nodes, which then can update their respective state and keep all nodes informed.

Preventing over-communication and mixed messages, the list is derived from seed nodes for all nodes in the cluster. When a node boots up it initiates its gossip from this seed node, which we usually have a few of, and then continues with that gossip list. Note, that seed nodes aren’t a single point of failure, as other nodes in the cluster will take their place if need be, they’re just kind of designated as the lead to initiate a gossip list from.

It is important in Apache Cassandra to also designate a single seed node per replication group (i.e. datacenter) for the seed list. This is recommended for fault tolerance, else gossip has to communicate across higher latency to hit each datacenter, which can eat at response time and performance of the gossip. Think of sending a snail mail USPS letter to a friend to get gossip news! That would take months just to find out what’s going on, kind of the same version of that for computer nodes going across datacenters to talk to the seed node.

An Inspiring VLOG of @noopkat’s trip to Portland for ML4ALL

Last year amid the various blogs, conversations, writing, and other things I was doing attempting to put together a cool video of my ML4ALL experience my friend Suz (AKA @noopkat) posted a VLOG that summarized it better than I could put together, so I went about doing other things I could be productive at – like watching this VLOG she did.

 

Re-installation of Linux, The Fuckery of Windows 10 & Microsoft Licensing

Notice: TLDR is at the very bottom.

Notice: This post may damage sensitive ears. At points it may be vulgar.

Notice: If you want to get to just the Linux parts, scroll down to the “Patience and Linux Saves The Day” headline.

The Beginning of Horror

First a few positives for Windows 10. It runs a Bash system thingy on it now, that’s pretty cool. It loads much faster than Windows 7 (and Vista, LULz). It still plays and supports basically all the games first. It has active tiles, but I guess that’s an arguable positive.

Windows 10 isn’t that horrifyingly bad, it seems, at first glance. But it requires a bunch of unnecessary work to re-map in your brain how it works in relation to the web, to bash, to Linux and the whole *nix POSIX world. You might say, but why does that even matter? Well, it matters because we’re in a web world, which is at it’s core a POSIX world of Linux and at worst MacOS, but it sure as hell isn’t a Windows world anymore. That’s just what consumers get that are powerless to do anything else about the fact they’re generally always going to be stuck with a purchase between Windows Device and Windows Device.

Now combine that with the fact in the last ~8-10 years almost every major change in the industry through the advent of a new technology, new platform, or otherwise has been released for several years on Linux or MacOS before it even attempts to come to Windows. Docker is a prime example, and it’s arguable that it still doesn’t really work on Windows. That’s the starting point of the horror for me.

The First Mistake of Mine and Microsoft’s

The real horror however started with these next few facts. I was pondering keeping Windows 10 on the machine to play games, so I got a key to update Windows 10 from whatever it is to “Professional” so that I could use Docker and Hyper-V. Because Docker doesn’t really run natively on Windows it need Hyper-V to be able to do anything. I “purchased” an upgrade from the Windows Store to Professional so that I could work with Docker and Hyper-V. It went through a process and all seemed well.

The Second Mistake

With Windows 10 upgraded, I also thought it’d be great to have a completely clean install. Windows 10 on this machine had all sorts of cruft on it. Trash games and other junk, I didn’t even realize at this time it was supposed to be the clean version from the Microsoft store. But being that it had so much cruft and other unknown settings, drivers, and configuration mess setup I figured I’d format the drive and reload. To do this I went out and downloaded a Windows 10 image. I assumed wrongly that it would give me the version pertinent to whatever the machine has on it, in this case Professional. It gave me “Enterprise” for whatever reason. This was the second mistake.

I then formatted the drive and reloaded Windows 10. I commenced to think everything was fine and dandy, you see, in the docs it says the machine should have the key bound to the hardware. Which I now know is utter bullshit, or simply software that just doesn’t work. I should have known better than to trust this stuff. But I made a mistake, I did, and now I was going to pay for it.

After re-installing Creative Cloud from Adobe, all my JetBrains software, and tons of other things I noticed a whole host of annoyances. Like the fact that the software cruft was still everywhere. I didn’t want this garbage shit, what gives. This was supposed to be a clean install. I kept digging about and installing things. I got OBS configured again specifically for Windows 10, because ya see, the folders still disagree on “\” vs “/” and Windows attaches every storage medium to an anchor letter. So all the sources were broken, and the list goes on. I spent hours upon hours getting Windows 10 setup under the notion I’d be able to use it.

Then it happened. Windows reports that it is not activated. So I began the process of activated, I click on updates and such and clicked try again. I went through the loops of help documentation. Nothing was happening. After trying every loop in the help docs that Microsoft provides I was getting frustrated. Then I noticed what the fatal issue was. I somehow now had “Enterprise” loaded and a key for “Professional”. I didn’t know, being what I suspect is a mere “consumer” now, how to actually choose which version of Windows to download. I had no MSDN, didn’t really need or want one, I thought. I also didn’t want to call in any favors because I wouldn’t want others to have to do that. Whenever I show or teach, or record things, I want to be able to tell people exactly what I’m using and why. Not that I “got the hookup” from inside Microsoft.

I continued, at this point, over 24 hours into the loading, setup, and configuration of this machine and I realize that I don’t even actually have a key, the fucking Microsoft site didn’t save the damned key, and I’m fighting with licensing to recover the OS and make sure that I can use it ongoing. This vain hope of using Windows, of catering and shaping much of my workflow around Windows just so that I could have it for games came from the initially reported (from Dell support itself) that Linux just simply wouldn’t work on the XPS 15 after I had spent 2 hours flailing around trying to install Ubuntu Linux on the machine. For reference, here’s the basic process I went through to get Linux on, but it appeared to fail by freezing up, throwing a bunch of kernel panics, and other miscellaneous nonsense. Either way I had given up and thought Windows 10 would be easier and had invested the time in it to get going.

I even recorded this effort, which is moderately useful and has some good bits of information. But mostly just the installation failure and freeze ups.  😦

It was now the end of the weekend, I’d sunk all my time in this, and Windows 10 was now in the late evening of Sunday telling me I didn’t have a legit copy of Windows. I was furious, enraged, and thought I now had a $2k+ dollar brick. I couldn’t get a legit copy of Windows without going some secret back door deal, shelling out even more money for a full copy of Windows, or so I thought. I felt screwed, shorted, and simply fucked over by Microsoft and their misleading notions of how their own software works around Windows. Honestly though, it wasn’t the first time, and I hadn’t felt this enraged with computers and software since the last time – almost a decade ago – when I used Windows and wrote primarily in Visual Studio (not Visual Studio Code).

End State Condition 1: Windows 10 loaded and not activated.

giphy

End State Condition 2: The wrong version is installed, Enterprise, instead of Professional. Based on the download from the automated consumer download option from Microsoft.

giphy (3)

End State Condition 3: I was now stuck in a support loop with the docs. No option led to a resolution.

giphy (1)

End State Condition 4: There was no contact information or way to submit a ticket without paying more money. Consumer support failure.

giphy (2)

Conclusion: I wasn’t even supposed to be here today.

wasntevensupposedtobe

I was done, even if the laptop became a very expensive paper weight I was done, screw it.

The Light At the End of the Tunnel Shines!

So I had a beer (huge shout out to North Coast Brewing, Old Rasputin ROCKS!) and settled myself down into a stupor. Kind of a technology depression I’d fallen into for this Windows garbage again. I knew better I told myself over and over again, “I knew better“! At some point about halfway through this wonderful beer, I had another thought, “I do know better and there’s obviously a way to get Linux installed on this machine!” Seriously, I’ve fought harder installations than this! I’d tried installing Red Hat 3.2 back in the day and was successful! Ok, I’m going to calmly pick up this Mac here and start researching. I started finding lots of material about how the 9750 XPS 15 basically is plagued with poor driver support for Linux, but then with a few tweaks of keywords and Google-fu coming into play, I hit a few winners! That’s when I started to climb out of this pit of despair and tweeted this.

 

 

 

I thought, even if it takes until 4am no better way for success after this complete debacle with Windows 10 then to just say I’m going to live tweet my efforts via all the hacks and tweaks on websites I’d just found right! Right. But alas, I did, and into the installation I went!

Patience and Linux Saves The Day

With the USB Stick loaded (same as I had done before in the video posted above in which the installation failed) I began. First step forward was to reboot and get into the BIOS (v1.7.0) and turn AHCI mode on under SATA options. I’m honestly not even sure why RAID On was set since there isn’t multiple drives, what even is this black magic in modern BIOS’s I do not know.

2

I booted using the F12 key for the boot options menu of the BIOS.

1

This process, because of some oddity in the Dell keyboard it’s easier to just press a bunch instead of holding down F12. I show an example in this short video in the tweet below, then selecting the right boot option once at the boot screen.

 

 

That gets us to the boot and install Linux screen. Did I mention Ubuntu looks crazy nice on this 4K screen? No? Well it does. Real glossy!

3

This is where things are kind of tricky. I need to “Try Ubuntu without installing” screen and it flashes by too fast. So a lot of fast button pushing ensues. Click, click, click, click, click, click. Like this, then hit “shift” real fast, then hit “e”! It’s like those console game secret codes practically!! 😬  Shown in the tweets below, the button pressing maneuver and the subsequent screen where the next setting needs to be made.

 

Set nomodeset after the words quiet splash.

4

Next up save that and continue. Then press F10 and the installation screen will appear. Start the installation by clicking that icon up there in the left hand corner that reads “Install Ubuntu 18.04.1 LTS”! Continue through the steps choosing the standard things you’d want. Connect to your WiFi. Then on Updates and Other Software do NOT choose the 3rd part option.

5

Then boot into partially busted Ubuntu. A few more steps tho. sudo gedit /etc/default/grub and set nouveau.modeset=0 and then sudo shutdown -r now. Then sudo apt install mokutil.

6

Then it’s the tricky Nvidia Drivers. Gotta use the “Software & Updates” option. Don’t use the ppa, so goes the word. Select the “Using NVIDIA driver…” and Apply Changes. Reboot after. 😑

 

 

 

 

 

 

 

Then this will probably happen. 👩🏻‍🔧 So make a password and all, then reboot. Note, it’ll show on this screen in plain text. This isn’t 100% sure to happen at this point, based on some things I read, but it likely should happen. I’d like to hear from you if you’re going through this and it doesn’t popup at this point.

9

Then after you enter a password. Do enter a password. Don’t forget it either, even though you’ll rarely enter this one. Then this on reboot will happen, continue.

11

Then on reboot check the driver. Check the driver via command line with nvidea-smi. If it failed, do another full sudo apt-get update.

12

Then go and unselect the driver, apply, and select the driver again and apply.

13

Then I did some battery tune ups too, even though I’ve also read they’re all there already in the latest Ubuntu. Thus, it doesn’t hurt to run these to make sure.

sudo apt update<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
sudo apt install tlp tlp-rdw powertop
sudo tlp start
sudo powertop --auto-tune sudo reboot

That was it. I rebooted again, just to make sure all the services were restarted, no kernel panics, no machine freak outs and all that jazz. I then immediately went into installing all the things I wanted on the machine. But more on that later, for now, success had been achieved and the horrors of a $2k+ dollar paperweight had been avoided!

TLDR;

If you’re a coder, use Linux for the sake of us all. Throw Windows 10 in the trash bin, Microsoft already has too, note that everything is going CLI first, container based, and cloud focused Azure WHICH IS AWESOME, KEEP DOING IT MICROSOFT!! Using Windows 10 and attempting to get/keep/verify it upon an error or two is an insult to consumers of the system. It really does seem like the OS is kept around out of pity or something. Final word, Dell and all y’all gotta get that hardware support for Linux going, otherwise what are you even doing as a hardware manufacturer. Push that stuff out there, make it a priority. Dell was one of the preeminent machine options and a king among those that survived the 90’s, a company with amazing support, and a solid coder’s platform. The company is still largely there, but Linux should be an even higher priority. Take another 10-20 people and unleash and enable them on that problem space! Then we all win.

TLDR; Notice:

It also seems, just in case, always have at least one absolutely wonderful, premium, and spectacularly tasty beer handy to come out of the horrors of technology depression and calm down. It helps. I suppose one could alternate the “beer” for you’re preferred calming beverage like tea or such though.

I wish all vastly better luck in all their installations and interactions with their devices than I had, but rest assured, this XPS 15 is now a beast and beautifully runs Linux (Ubuntu so far)!

On that note, happy thrashing code all, and best of luck and skill in your endeavors!

Virtual Box Issue “Kernel driver not installed (rc=-1908)”

Alright, nothing like multi-OS environments so you can prepare for the maximum number of issues that might come up. My recent issue is re-opening some virtual machines, originally created on Windows 10, and now with intent to use them on Linux. When trying to open these existing virtual machines I’ve gotten the following error:

Kernel driver not installed (rc=-1908)

The VirtualBox Linux kernel driver (vboxdrv) is either not loaded or there is a permission problem with /dev/vboxdrv. Please reinstall the kernel module by executing

‘modprobe vboxdrv’

as root.

where: sublibOsInit what: 3 VERR_VM_DRIVER_NOT_INSTALLED (-1908) – The support driver is not installed. On linux, open returned ENOENT.

vbox-error1908

Error Screenshot on Ubuntu Linux

Upon researching I found this could occur on multiple operating systems really, and have included the fixes for the various platforms. First however, let’s talk about the Linux fix.

Linux

For this particular situation I tried the command given in the dialog modprobe vboxdrv to see if that would fix it. But no go. I got some additional errors that seemed nonsensical.

arg

At this point I dug into the search engine. The first post I found seemed to make sense. I tried the commands.

sudo apt-get install build-essential module-assistant
sudo m-a prepare

step1.png

That went well, so on to the next step of building the virtualbox kernel driver.

sudo /etc/init.d/vboxdrv setup

That gave me “sudo: /etc/init.d/vboxdrv: command not found“. So I did a find for the file.

sudo find / -name "vboxdrv"

The results pointed out that my install of Virtual Box, done via the Ubuntu App Store, was version 5.2.8. This is clearly not going to work out well because the latest version is 6.0.4 right now. Next I uninstalled this old version via the Ubuntu App Store Application.

step2.png

Next I navigated back out the Virtual Box Download page. There I downloaded the latest version and installed it via the downloadable Ubuntu *.deb file.

This installs Virtual Box through the Ubuntu App Store application but sets up the more up to date repository to get the installation from.

step3.png

Ok, so all this at this point I ran VirtualBox after the installation, it installed according to the standard directories this time but still didn’t work. What the hell, I’ve no idea. I did another search and found Øyvind Stegard’s blog entry on fixing the “VirtualBox + Secure Boot = Fail“. He states at the start of the blog entry,

“The problem is the requirement that all kernel modules must be signed by a key trusted by the UEFI system, otherwise loading will fail. Ubuntu does not sign the third party vbox* kernel modules, but rather gives the user the option to disable Secure Boot upon installation of the virtualbox package.”

Stegard then adds two very important links with pertinent information about this issue, I’ve reposted here too:

After going through Øyvind‘s steps, I finally got the previously specified command from above.

sudo /etc/init.d/vboxdrv setup

Now VirtualBox started up just fine. Finally. Ugh! But hey, in the process I now understand a little bit more about secure boot, current BIOS’s, and related hardware concerns. Not that I had set out or wanted to deal with this right now, but boom, it’s done!

Alright, but if you fall into this with MacOS, it’s a slightly different story. Here’s some fixes around that.

MacOS

Ok, I started writing up and searching as I wrote, and instead of continuing to write it looks like this blog entry popped up with the exact error I’d gotten in the past (the reason I wanted to included MacOS for future ref). If you’ve gotten this error, check out Xilin Sun‘s post Solving VirtualBox “Solving VirtualBox “kernel driver not installed (rc=-1908)” Error on macOS“.

Bunches of Databases in Bunches of Weeks – PostgreSQL Day 1

May the database deluge begin, it’s time for “Bunches of Databases in Bunches of Weeks”. We’ll get into looking at databases similar to how they’re approached in “7 Databases in 7 Weeks“. In this session I got into a hard look at PostgreSQL or as some refer to it just Postgres. This is the first of a few sessions on PostgreSQL in which I get the database installed locally on Ubuntu. Which is transferable to any other operating system really, PostgreSQL is awesome like that. Then after installing and getting pgAdmin 4, the user interface for PostgreSQL working against that, I go the Docker route. Again, pointing pgAdmin 4 at that and creating a database and an initial table.

Below the video here I’ve added the timeline and other details, links, and other pertinent information about this series.

0:00 – The intro image splice and metal intro with tunes..
3:34 – Start of the video database content.
4:34 – Beginning the local installation of Postgres/PostgreSQL on the local machine.
20:30 – Getting pgAdmin 4 installed on local machine.
24:20 – Taking a look at pgAdmin 4, a stroll through setting up a table, getting some basic SQL from and executing with pgAdmin 4.
1:00:05 – Installing Docker and getting PostgreSQL setup as a container!
1:00:36 – Added the link to the stellar post at Digital Ocean’s Blog.
1:00:55 – My declaration that if Digital Ocean just provided documentation I’d happily pay for it, their blog entries, tutorials, and docs are hands down some of the best on the web!
1:01:10 – Installing Postgesql on Ubuntu 18.04.
1:06:44 – Signing in to Docker hub and finding the official Postgresql Docker Image.
1:09:28 – Starting the container with Docker.
1:10:24 – Connecting to the Docker Postgresql Container with pgadmin4.
1:13:00 – Creating a database and working with SQL, tables, and other resources with pgAdmin4 against the Docker container.
1:16:03 – The hacker escape outtro. Happy thrashing code!

For each of these sessions for the “Bunches of Databases in Bunches of Weeks” series I’ll follow this following sequence. I’ll go through each database in this list of my top 7 databases for day 1 (see below), then will go through each database and work through the day 2, and so on. Accumulating additional days similarly to the “7 Databases in 7 Weeks

Day 1” of the Database, I’ll work toward building a development installation of the particular database. For example, in this session I setup PostgreSQL by installing it to the local machine and also pulled a Docker image to run PostgreSQL.

Day 2” of the respective database, I’ll get into working against the database with CQL, SQL, or whatever that one would use to work specifically with the database directly. At this point I’ll also get more deeply into the types, inserting, and storing data in the respective database.

Day 3” of the respective database, I’ll get into connecting an application with C#, Node.js, and Go. Implementing a simple connection, prospectively a test of the connection, and do a simple insert, update, and delete of some sort against the respective database built on the previous day 2 of the same database.

Day 4” and onward I’ll determine the path and layout of the topic later, so subscribe on YouTube and Twitch, and tune in. The events are scheduled, with the option to be notified when a particular episode is coming on that you’d like to watch here on Twitch.

Next Events for “Bunches of Databases in Bunches of Days

It’s Official, ML4ALL 2019, Machine Learning Conference 4 All v2!

It’s official, we’ve got dates and tickets are open for ML4ALL 2019! Our CFP will be open in a number of hours, not days, and I’ll do another update the second that we have that live.

What is ML4ALL?

ML4ALL stands for “Machine Learning for All“. Last year I enjoyed working with Alena Hall, Troy Howard, Glenn Block, Byron Gerlach, and Ben Acker on getting a great conference put together, and I’m looking forward to rounding up a team and doing a great job putting together another great conference for the community again this year!

Last year @lenadroid put together this great video of the event and some short interviews with speakers and attendees. It’s a solid watch, take a few minutes and check it out for a good idea of what the conference will be like.

Want to Attend? Help!

Tickets are on sale, but there’s a lot of other ways to get involved now. First, the super easy way to keep track of updates is to follow the Twitter account: @ml4all. The second way is a little bit more involved, but can be a much higher return on investment for you, by joining the ML4ALL Slack Group! There we discuss conference updates, talk about machine learning, introduce ourselves, and a range of other discussions.

If you work for a company in the machine learning domain, plying the wave of artificial intelligence and related business, you may want to get involved by sponsoring the conference. We’ve got a prospectus we can send you for the varying levels, just send an email to ml4allconf@gmail.com with the subject “Plz Prospectus”. We’ll send you the prospectus and we can start a conversation on which level works best for your company!

The TLDR;

ML4ALL is a conference that will cover from beginner to advanced machine learning presentations, conversations, and community discussions. It’s a top conference choice to put on your schedule for April 28-30th, pick up tickets for, and submit a proposal to the CFP!