May the database deluge begin, it’s time for “Bunches of Databases in Bunches of Weeks”. We’ll get into looking at databases similar to how they’re approached in “7 Databases in 7 Weeks“. In this session I got into a hard look at PostgreSQL or as some refer to it just Postgres. This is the first of a few sessions on PostgreSQL in which I get the database installed locally on Ubuntu. Which is transferable to any other operating system really, PostgreSQL is awesome like that. Then after installing and getting pgAdmin 4, the user interface for PostgreSQL working against that, I go the Docker route. Again, pointing pgAdmin 4 at that and creating a database and an initial table.
Below the video here I’ve added the timeline and other details, links, and other pertinent information about this series.
0:00 – The intro image splice and metal intro with tunes.. 3:34 – Start of the video database content. 4:34 – Beginning the local installation of Postgres/PostgreSQL on the local machine. 20:30 – Getting pgAdmin 4 installed on local machine. 24:20 – Taking a look at pgAdmin 4, a stroll through setting up a table, getting some basic SQL from and executing with pgAdmin 4. 1:00:05 – Installing Docker and getting PostgreSQL setup as a container! 1:00:36 – Added the link to the stellar post at Digital Ocean’s Blog. 1:00:55 – My declaration that if Digital Ocean just provided documentation I’d happily pay for it, their blog entries, tutorials, and docs are hands down some of the best on the web! 1:01:10 – Installing Postgesql on Ubuntu 18.04. 1:06:44 – Signing in to Docker hub and finding the official Postgresql Docker Image. 1:09:28 – Starting the container with Docker. 1:10:24 – Connecting to the Docker Postgresql Container with pgadmin4. 1:13:00 – Creating a database and working with SQL, tables, and other resources with pgAdmin4 against the Docker container. 1:16:03 – The hacker escape outtro. Happy thrashing code!
“Day 1” of the Database, I’ll work toward building a development installation of the particular database. For example, in this session I setup PostgreSQL by installing it to the local machine and also pulled a Docker image to run PostgreSQL.
“Day 2” of the respective database, I’ll get into working against the database with CQL, SQL, or whatever that one would use to work specifically with the database directly. At this point I’ll also get more deeply into the types, inserting, and storing data in the respective database.
“Day 3” of the respective database, I’ll get into connecting an application with C#, Node.js, and Go. Implementing a simple connection, prospectively a test of the connection, and do a simple insert, update, and delete of some sort against the respective database built on the previous day 2 of the same database.
“Day 4” and onward I’ll determine the path and layout of the topic later, so subscribe on YouTube and Twitch, and tune in. The events are scheduled, with the option to be notified when a particular episode is coming on that you’d like to watch here on Twitch.
Next Events for “Bunches of Databases in Bunches of Days“
I’ve been putting together a list of conferences that I want to aim to attend this coming year. I made it, then thought, “somebody else could use this list probably” so here it is. If you think of any other specific conferences I ought to add and attend please leave a comment. Enjoy!
For to long lock-in has been referred to with an almost entirely negative connotation even though it can be inferred in positive and negative situations. The fact is that there’s a much more nuanced and balanced range to benefits and disadvantages of lock-in. Often this may even be referred to as this or that dependency, but either way a dependency often is just another form of lock in. Weighing those and finding the right balance for your projects can actually lead to lock-in being a positive game changer or something that simply provides one a basis in which to work and operate. Sometimes lock-in actually will provide a way to remove lock-in by providing more choices to other things, that in turn may provide another variance of lock-in.
Concrete Lock-in Examples
The JavaScript Lock-In
Take the language we choose to build an application in. JavaScript is a great example. It has become the singular language of the web, at least on the client side. This was long ago, a form of lock-in that browser makers (and standards bodies) chose that dictated how and in which direction the web – at least web pages – would progress.
JavaScript has now become a prominent language on the server side now too thanks to Node.js. It has even moved in as a first class language in serverless technology like AWS’s Lambda. JavaScript is a perfect example of a language, initially being a source of specific lock-in, but required for the client, that eventually expanded to allow programming in a number of other environments – reducing JavaScript’s lock in – but displacing lock in through abstractions to other spaces such as the server side and and serverless functions.
The .NET Windows SQL Server Lock In
JavaScript is merely one example, and a relatively positive one that expands one’s options in more ways than limits one’s efforts. But let’s say the decision is made to build a high speed trading platform and choose SQL Server, .NET C#, and Windows Server. Immediately this is a technology combination that has notoriously illuminated in the past * how lock-in can be extremely dangerous.
This application, say it was built out with this set of technology platforms and used stored procedures in SQL Server, locking the application into the specific database, used proprietary Windows specific libraries in .NET with the C# code, and on Windows used IIS specific advances to make the application faster. When it was first built it seemed plenty fast and scaled just right according to the demand at the time.
Fast forward to today. The application now has a sharded database when it hit a mere 8 Terabytes, loaded on two super pumped up – at least for today – servers that have many cores, many CPUs, GPUs, and all that jazz. They came in around $240k each! The application is tightly coupled to a middle tier, that is then sort of tightly coupled to those famous stored procedures, and the application of course has a turbo capability per those IIS Servers.
But today it’s slow. Looking at benchmarks and query times the database is having a hard time dealing with things as is, and the application has outages on a routine basis for a whole variation of reasons. Sometimes tracing and debugging solves the problems quickly, other times the servers just oversubscribe resources and sit thrashing.
Where does this application go? How does one resolve the database loading issues? They’ve already sunk a half million on servers, they’re pegged out already, horizontally scaling isn’t an option, they’re tightly coupled to Window Servers running IIS removing the possibility of effectively scaling out the application servers via container technologies, and other issues. Without recourse, this is the type of lock in that will kill the company if something is changed in a massive way very soon.
To add, this is the description of an actual company that is now defunct. I phrased it as existing today only to make the point. The hard reality is the company went under, almost entirely because of the costs of maintaining and unsustainable architecture that caused an exorbitant lock in to very specific tools – largely because the company drank the cool aid to use the tools as suggested. They developed the product into a corner. That mistake was so expensive that it decimated the finances of the company. Not a good scenario, not a happy outcome, and something to be avoided in every way! This is truly the epitomy of negative lock in.
Of course there’s this distinctive lock in we have to steer clear from, but there’s the lock in associated with languages and other technology capabilities that will help your company move forward faster, easier, and with increasing capabilities. Those are the choices, the ties to technology and capabilities that decision makers can really leverage with fewer negative consequences.
The “Lock In” That Enables
One common statement is, “the right tool for the job”. This is of course for the ideal world where ideal decisions can be made all the time. This doesn’t exist and we have to strive for balance between decisions that will wreck the ship or decisions that will give us clear waters ahead.
For databases we need to choose the right databases for where we want to go versus where we are today. Not to gold plate the solution, but to have intent and a clear focus on what we want our future technology to hold for us. If we intend to expand our data and want to maintain the ability to effectively query – let’s take the massive SQL Server for example – what could we have done to prevent it from becoming a debilitating decision?
A solution that could have effectively come into play would have been not to shard the relational database, but instead to either export or split the data in a more horizontal way and put it into a distributed database store. Start building the application so that this system could be used instead of being limited by the relational database. As the queries are built out and the tight coupling to SQL Server removed, the new distributed database could easily add nodes to compensate for the ever growing size of the data stored. The options are numerous, that all are a form of lock-in, but not the kind that eventually killed this company that had limited and detrimentally locked itself into use of a relational database.
At the application tier, another solution could have been made to remove the ties to IIS and start figuring out a way to containerize the application. One way years ago would have been to move away from .NET, but let’s say that wasn’t really an option for other reasons. The idea to mimic containerization could have been done through shifting to a self-contained web server on Windows that would allow the .NET application to run under a singular service and then have those services spin off the application as needed. This would decouple from IIS, and enable spreading the load more quickly across a set number of machines and eventually when .NET Core was released offer the ability to actually containerize and shift entirely off of Windows Server to a more cost efficient solution under Linux.
These are just some ideas. The solutions of course would vary and obviously provide different results. Above all there are pathways away from negative lock in and a direction toward positive lock in that enables. Realize there’s the balance, and find those that leverage lock in positively.
Nuanced Pedantic Notes:
Note I didn’t say all examples, but just that this combo has left more than a few companies out on a limb over the years. There are of course other technologies that have put companies (people actually) in awkward situations too. I’m just using this combo here as an example. For instance, probably some of the most notorious lock in comes from the legal ramifications of using Oracle products and being tied into their sales agreements. On the opposite end of the spectrum, Stack Overflow is a great example of how choosing .NET and scaling with it, SQL Server, and related technologies can work just fine.
Hello everybody and especially the upcoming ML4ALL attendees! For those that haven’t registered, here’s the link for that, and for those that have welcome aboard! Here’s two of the excellent speakers that will be presenting at ML4ALL Paige (@DynamicWebPaige) and Suz (@Noopkat)!
Years ago I got into technology, specifically software development and programming. I also had worked within the infosec, networking, and operations side of the industry for a while. But I’d always kind of focused on and emphasized in my career the work I did, and do within software development.
One of the things I’ve realized recently is a massive change in the industry. The technology sector, especially what is so often associated with the startup scene, has become old industry. It is on the verge of being just like other heavily regulated, unionized, slower moving, and by necessity, less innovative industries (do NOT get me wrong, all industries, even the most regulated continue to try to progress, it just becomes exponentially more difficult). The key thing to note, is the oft agile, quickly moving, intuitively inventive, progressive and out of the box thinking of software development has become dramatically more similar to something like building architecture; regulated, limited, and demanded upon by society.
These changes have been coming for a long while, since more and more of the world has been taken over by software and the computers that run the software. We’re on the verge of having so many automated things that most of humanity will know no more about the software automated nature of the things (re: automated cars, etc) then they do about internal combustion engines (they’ve been around for 160 years)
This is a time where the technology future isn’t unknown, the paths have started to be more identified and clear, and dictated to in the extreme. The Government is more involved; regulators, legislators, lawyers of every ilk, patent trolls, and more. You name it, the people who make capitalism and independent action in a market a nasty business are all at the table these days. I mean, even the nastiest of them all tried to have a tech council, but that even fell apart on dearest leader Trump.
So… ya know…
Good Riddance to Doom & Gloom
I just wrote this and realized I’d painted a pretty droll and dreary future for us developers and technologists. I’m inherently a positive person about the future, but I know the national leadership down to the rank and file citizens of this country have left me in a droll and dreary present. Thus the old pyre of depression sinks in and all of a sudden the future I can see is droll and dreary. But really, there’s hope, and so often humanity actually thrives, succeeds, and perseveres when the cards are stacked against us. But being realistic, there are some pretty staunch hands stacked against humanity these days. We’ve got our work cut out for us.
Striving to Prepare for Tomorrow
So where do we — the tech sector — as an industry go? Where do I go? I myself am not sure, I’m in the process of lining up 2018 as I write this (more on this really soon). I’m studying algorithms, looking at multitudes of industries, and trying to figure out what the future holds. What industry is actually going to take us forward into tomorrow without decimating and wrecking that very future?
The current national leadership doesn’t have a clear plan and has tried to thrash the economy into a near convulsed state to again become more reliant on coal, more oil, and other arcane forms of energy we still have a noose around our necks with. They apparently disregard the tech sector, nor seem to understand it or the rise of the clean energy sector. Eventually they’ll be gone, which hopefully our republic won’t be, and we’ll be able to again lead forward into the future. Fight to strive for industries and efforts that are the innovative, inventive, creative, builders of a better, hopeful, exciting, and clean world for ourselves.
How can I, as just one of many players in the technology sector have affect and effect in a way that prepares us for tomorrow more effectively? How can I find the traits of the industry again that I loved so much? That innovative, inventive, creative, builder structured mentality of thinking, learning, and pushing forward? The current industry seems to be as much of a roadblock to itself as the future, so I’m trying to think of ways in which I myself, and any of us can lead to better lives but still harness and enjoy those things that made us love and want to work within this industry.
…and yeah, I have a few thoughts on the matter. Which I’ve ordered by what I’m confident is the priority we need to resolve them in order to truly advance society to a more sustainable, equitable, and positive future. Not that all of these things shouldn’t be worked on right now, because they all should absolutely be!
Energy & Economic Challenges
Before we as a species can really continue thinking about the future, at least in any honest and truly long term (like say greater than 25 years even) we really need to get a handle on how we run economic systems and especially how we provide power and logistics for the planet. We’ve set ourselves up for some pretty immediate and deadly energy problems now and in the coming years. The best news is there are legitimate solutions on the table, but we’ve got to get them built!
One of the solutions, which isn’t talked about very often, is that we simply can and should use less energy. Look at much of what we’ve done in the past 50 years. Gone from 8 mpg in a car to almost 22 in a giant SUV while many cars can sustain 40 or more mpg. Even better we are moving steadily, albeit to slowly, toward electric energy for cars and road transportation. In buildings we’ve gone from houses that can barely maintain a steady temperature without extensive use of electricity (which is often in the US, created by coal). But new houses can maintain a steady temperature with a minimal amount of energy used. All of these are pretty significant advances, but there are more coming sooner in some cases then we might realize.
Tesla, Ford, and GM are all working toward advances in these realms. In railroading, well, they are and have been there for many decades. But the future may hold even more clean options for our logistical needs in this country and around the world. Hopefully it keeps moving forward as it is, and reaps the huge potential benefits that can be found there.
I keep working on and providing, when possible, options and assistance in this industry. I’m by no means a full time logistics or planner person, but my interest and fields of study often take me to work side by side with individuals in these industries. As technology moves further into freight, passengers logistics (re: cars, trains, planes, etc) I’m likely to be involved and happy to be involved in any way I can.
Housing, Technology, and Markets
The notion, and especially the nonsense, behind Government controlled markets, housing, and capitalism is under heavy attack these days. From ideological means to actual assault in the legislatures of the nation and states of this country and unfortunately, more than a few others (re: Brexit, etc). Much of the arguments form around equity, many others around the now decimated middle class since labor work is starting to be such a small part of the overall world economy. We have the proverbially defined sharing economy taking off, but it is in many ways a kind of race to the bottom for many of those working that economy as a way of life.
At the same time we have a greedy, glutenous, and excessive use of housing space and construction focused around single family homes. Meanwhile (read “Color of Law” for more on these following comments) we know with extensive historical understanding that neighborhood divisions, people segregation, and auto dependency is almost all pervasively enforced by zoning that limits most modern construction to trashy apartments and cheap, ticky tacky single family home housing. This has left many cities, where the majority of people continue to move to for entertainment, life, and work, with a severe and expensive housing shortage.
The advances in building construction, technology, and an almost complete tear down of societal neighborhood associations and Governmental zoning is required to fix these extensive housing shortages. We have to build, and regardless of economic mode (communism, socialism, fascism, capitalism, or whatever) we need more housing in cities, and the advancement of technologies along with changes to our societal organization of zoning must occur. I write must, but really it will, as it’s being forced regardless of how much opposition some push up against it. Simply we must make way for the future. What we do dictates if that future will be a good one, or one that is crumbling in our very hands as we try to build it.
I’m involved, here in Seattle (and still some when I can in Portland) to move this issue forward. The old ideas around forced segregation through zoning must die, but we also must repair our thinking and infrastructure around housing, zoning, and how and in which way we can live effectively. Cities like Seattle, Amsterdam, Copenhagen, Berlin, Vancouver BC, Portland, and others like them hold the answers and will push those forward.
Misogyny, Sexism, Racism, Bigotry…
I have lots of friends. I recently was curious and checked the overall demographics. I must do well to make friends across barriers, as the demographic range of friendships I enjoy are as diverse as the very population of the United States. I am thankful for and extremely happy that I can call so many of you friends, that we’ve all broken down, destroyed, and insured none of these societal constraints prevent our friendships.
But that doesn’t mean my network of friends, family, and loved ones throughout the world are representative of what everyone gets to experience. More barriers need destroyed, more equity needs to be found, more opportunity needs to be made available, and more lives need to be freed from petty constraints in order to help all of society move forward to a better future. The technology sector, even in it’s increasingly matured state, still has a massive and dramatic hand to play in this card game of success.
The issues, in tech and of course elsewhere, and unfortunately etched into the fibers of our entire society still need a lot of work. Let’s say, it’s easily the nastiest and most difficult refactoring of society ever. Today it needs to be intelligent, but there’s still the fights that will bring about the change, the laws that will and need to be broken, and in general, there is pain to be had. But for all of us people, this is a fight, this is pain that will endure, but intelligently we must make our world a better place.
I ask myself as everyone should; what are my biases, transgressions, and how can I insure to be a better soul today, tomorrow, and onward. How can I help others not plod forward and make ignorant, damaging, and hurtful biases and transgressions? How can I use my abilities, my privilege, and my status in various fields — not just in my tech domain — but in the city I live, the local community, the neighboring communities, and the are throughout that I influence and have some effect on?
I read and learn steadily about these topics, read from other people and persons perspectives about the world, history and how and what really happened (first step, toss that trash show they teach in public schools! To much of it is blatantly wrong and misguided!). Knowing what my fellow citizens’ actions hurt others, and also what accomplishments they did make, and how it all shaped where we are today is extremely important in bringing together people. I’ll likely write a lot more on this topic in the coming year, as my study on the topic has become more intense these days. There is indeed, a lot we all have accomplished (from jazz and blues of the delta feeding the soul and heart of America to the spikes we laid for the railroads, to the rockets we launched, the math it took, and the way we reshape our cities to make the better for all people today).
Tomorrow
Right now I’m wrapping up 2017 and these ideas were running through my mind as I wrote some code. I found it funny that I was being productive writing the code, getting the problems resolved, but all these rather complex topics were rolling around in my mind on background threads. In an upcoming post I’ll be delving into those parts of the tech industry I love so much, that I hope won’t be squelched out in the coming years. I’ll aim to structure and formalize some of the passions, projects, and efforts coming up in 2018. Until then, hope all goes well with you and yours, cheers!
Take the language we choose to build an application in. JavaScript is a great example. It has become the singular language of the web, at least on the client side. This was long ago, a form of lock-in that browser makers (and standards bodies) chose that dictated how and in which direction the web – at least web pages – would progress.
JavaScript is merely one example, and a relatively positive one that expands one’s options in more ways than limits one’s efforts. But let’s say the decision is made to build a high speed trading platform and choose SQL Server, .NET C#, and Windows Server. Immediately this is a technology combination that has notoriously illuminated in the past * how lock-in can be extremely dangerous.
One common statement is, “the right tool for the job”. This is of course for the ideal world where ideal decisions can be made all the time. This doesn’t exist and we have to strive for balance between decisions that will wreck the ship or decisions that will give us clear waters ahead.
You must be logged in to post a comment.