Tag Archives: deployment

Setting Up Nodes, Firewall, & Instances in Google Cloud Platform

Here’s the run down of what I covered in the latest Thrashing Code Session (go subscribe here to the channel for future sessions or on Twitch). The core focus of this session was getting some further progress on my Terraform Project around getting a basic Cassandra and DataStax Enterprise Apache Cassandra Cluster up and running in Google Cloud Platform.

The code and configuration from the work is available on Github at terraform-fields and a summary of code changes and other actions taken during the session are further along in this blog entry.

Streaming Session Video

In this session I worked toward completing a few key tasks for the Terraform project around creating a Cassandra Cluster. Here’s a run down of the time points where I tackle specific topics.

  • 3:03 – Welcome & objectives list: Working toward DataStax Enterprise Apache Cassandra Cluster and standard Apache Cassandra Cluster.
  • 3:40 – Review of what infrastructure exists from where we left off in the previous episode.
  • 5:00 – Found music to play that is copyright safe! \m/
  • 5:50 – Recap on where the project lives on Github in the terraformed-fields repo.
  • 8:52 – Adding a google_compute_address for use with the instances. Leads into determining static public and private google_compute_address resources. The idea being to know the IP for our cluster to make joining them together easier.
  • 11:44 – Working to get the access_config and related properties set on the instance to assign the google_compute_address resources that I’ve created. I run into a few issues but work through those.
  • 22:28 – Bastion server is set with the IP.
  • 37:05 – I setup some files, following a kind of “bad process” as I note. Which I’ll refactor and clean up in a subsequent episode. But the bad process also limits the amount of resources I have in one file, so it’s a little easier to follow along.
  • 54:27 – Starting to look at provisioners to execute script files and commands before or after the instance creation. Super helpful, with the aim to use this feature to download and install the DataStax Enterprise Apache Cassandra or standard Apache Cassandra software.
  • 1:16:18 – Ah, a need for firewall rule for ssh & port 22. I work through adding those and then end up with an issue that we’ll be resolving next episode!

Session Content

Starting Point: I started this episode from where I left off last session.

Work Done: In this session I added a number of resources to the project and worked through a number of troubleshooting scenarios, as one does.

Added firewall resources to open up port 22 and icmp (ping, etc).

resource "google_compute_firewall" "bastion-ssh" {
  name    = "gimme-bastion-ssh"
  network = "${google_compute_network.dev-network.name}"

  allow {
    protocol = "tcp"
    ports    = ["22"]
  }
}

resource "google_compute_firewall" "bastion-icmp" {
  name    = "gimme-bastion-icmp"
  network = "${google_compute_network.dev-network.name}"

  allow {
    protocol = "icmp"
  }
}

I also broke out the files so that each instances has its own IP addresses with it in the file specific to that instance. Later I’ll add context for why I gave the project file bloat, by refactoring to use modules.

terraform-files.png

Added each node resource as follows. I just increased each specific node count by one for each subsequent node, such as making this node1_internal IP google_compute_address increment to node2_internal. Everything also statically defined, adding to my file and configuration bloat.

resource "google_compute_address" "node1_internal" {
  name         = "node-1-internal"
  subnetwork   = "${google_compute_subnetwork.dev-sub-west1.self_link}"
  address_type = "INTERNAL"
  address      = "10.1.0.5"
}

resource "google_compute_instance" "node_frank" {
  name         = "frank"
  machine_type = "n1-standard-1"
  zone         = "us-west1-a"

  boot_disk {
    initialize_params {
      image = "ubuntu-minimal-1804-bionic-v20180814"
    }
  }

  network_interface {
    subnetwork = "${google_compute_subnetwork.dev-sub-west1.self_link}"
    address    = "${google_compute_address.node1_internal.address}"
  }

  service_account {
    scopes = [
      "userinfo-email",
      "compute-ro",
      "storage-ro",
    ]
  }
}

I also setup the bastion server so it looks like this. Specifically designating a public IP so that I can connect via SSH.

resource "google_compute_address" "bastion_a" {
  name = "bastion-a"
}

resource "google_compute_instance" "a" {
  name         = "a"
  machine_type = "n1-standard-1"
  zone         = "us-west1-a"

  provisioner "file" {
  source      = "install-c.sh"
  destination = "install-c.sh"

  connection {
    type     = "ssh"
    user     = "root"
    password = "${var.root_password}"
    }
  }

  boot_disk {
    initialize_params {
      image = "ubuntu-minimal-1804-bionic-v20180814"
    }
  }

  network_interface {
    subnetwork = "${google_compute_subnetwork.dev-sub-west1.self_link}"
    access_config {
      nat_ip = "${google_compute_address.bastion_a.address}"
    }
  }

  service_account {
    scopes = [
      "userinfo-email",
      "compute-ro",
      "storage-ro",
    ]
  }
}

Plans for next session include getting the nodes setup so that the bastion server can work with and deploy or execute commands against them without the nodes being exposed publicly to the internet. We’ll talk more about that then. For now, happy thrashing code!

Troubleshooting Node.js Deploys on Beanstalk – The Express v4 node ./bin/www Switch Up

I’ve gotten a ton of 502 errors and related issues that crop up when deploying the Beanstalk. One of the issues that cropped up a few times recently, until I stumbled into a working solution was the 502 NGINX error. I went digging around and ended up just trying to deploy a default, fresh from the ‘express newAppNameHere’ creation and still got the error.

I went digging through the Beanstalk configuration for the app and found this little tidbit.

Node Command (Click for full size image)

Node Command (Click for full size image)

I’ve pointed out the section where I’ve added the command.

node ./bin/www

Based on the commands that are executed normally, it seems `npm start` would work work to get the application started. But I have surmised the issue is that the commands are executed sequentially;

node server.js
node app.js
npm start

When these are executed in order, errors crop up and the command that should work `npm start` begins with a corrupted and error laden beginning. Leaving the application not running. However by adding the `node ./bin/www` to the text box all the others are skipped and this command is issued, resulting in a running application.

The other thing is to follow the now standard approach of just issue `npm start`, but being sure to replace what I put in the text box above (`node ./bin/www`) with `npm start` so that beanstalk only runs npm start instead of the ordered execution.

Deploycon, PaaS & the pending data tier gravity fallout…

For a quick recap of last years Deploycon & related talks, check out my “Day #3 => DeployCon && Enterprise && Data Gravity” entry from last year.

PaaS Systems aren’t always effectively distributed. Heroku has fallen over every time east-1 has gone down at AWS. Not that I’m saying they’ve done bad, just pointing that out. With Cloud Foundry, there’s several key SPOFs (Single Points of Failure), and with all PaaS Systems the data tier is often the neglected pairing of the system. I’ve been wanting to write about this for a few months now and Deploycon has lit a fire for me to do just that.

Deploycon – “Platform Services and Developer Expectations” **

I’m on a panel at Deploycon titled “Platform Services and Developer Expectations” and this leads right back around to that. This SPOF issue is concerning to me as PaaS Providers talk up the offerings more and more with little light actually shone on this issue. In some ways each is moving away form their respective SPOFs, but overall they’re all pretty prevalent throughout. For security, each has a non-distributed database, which technically needs backed up still – no clear replication or other mechanisms setup to ensure data integrity in a failure situation. Of course, the huge saving grace with a PaaS, is that if the overall system goes down or a SPOF blows up, all the existing deployed applications will generally continue to run. Unless of course the routing and networking are also SPOF. This is the largest glaring concern with PaaS Systems that I see today.

One of the other things about PaaS that has always led to a ton of questions is “what about my PostGresql/mysql/Riak/mongodb/database thing and how do I do X, Y, Z with it to ensure scalability in my PaaS.” In almost every case it ends with a simple and unfortunate answer, “…when it comes to data, a PaaS doesn’t really do a damn thing for ya…” This is obviously not very helpful. The entire reason to put a PaaS into place is to simplify life, the sad fact that it barely does a thing for the data tier isn’t very helpful.

Now, hold on a second before you start screaming at me about “but a PaaS does X, Y and Z and isn’t even supposed to touch that aspect of things…” let me elaborate a bit more. The panel at Deploycon states “…Developer Expectations” and when things are getting simplified in the way a PaaS does, developers assume that if it does all this fancy magic for an application it ought to simplify the data side of things too! Right? Well no, and it isn’t going to for the foreseeable future. But no matter what, it doesn’t change the fact that developers often have that expectation.

Now, I could write at length about all the reasons that PaaS doesn’t really do anything for the data tier. I could wax poetic about how a distributed database (re: Riak, Cassandra, etc) just doesn’t lend itself to a cookie cutter approach to deployment under a PaaS or an RDBMS has umpteen different configurations for stability, scaling, hot swappable services, and other such complexities around the data tier. But instead I’m going to skip all, maybe cover some of those things another day, and jump right into some of the things that are actually moving forward to fill this gap.

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

The most obvious reason there isn’t a simple turn key solution to the data side of things with a PaaS ecosystem is that data is complex and extremely diverse. There’s distributed key/value stores (Riak, Cassandra), there’s sort of kind of distributed databases (Mongo), graph databases (Neo4j), the age old RDBMS (DB2, SQL Server, Oracle’s Stuff, etc) and the million solutions around that, there’s key/value in memory styled databases that are insanely fast, like Redis. Expanding just slightly you have software that works around these systems such as Hadoop & Riak CS & the list goes on. All of it focused on the data tier and maintaining one, two or some form of the three points around CAP Theorem (http://en.wikipedia.org/wiki/CAP_theorem), atomicity and other key capbilities.

All of the PaaS Systems, including public and private often have some sort of plug-in style architectures for data. Whether it is Apprenda which is closed to community and closed source or an ongoing open to community PaaS like OpenShift or Cloud Foundry, things still fall almost entirely to the developers or database team to build an architecture around the data. When looking at solutions to simplify data in PaaS Systems the closed source solutions we have no idea what they’re up to in this regard. The one’s that are open source or in large part public and involved in the community PaaSes, like EngineYard, Heroku, Cloudbees and others we can really see the directions and efforts around creating real PaaS style solutions to the data tier problem.

BOSH, Vagrant, etc…  One of the best solutions I’ve seen so far is the ability of Bosh, which was created by the Cloud Foundry team while at VMware, to spool up an environment that includes such things as a Riak Cluster (or other cluster). Currently Brian McClain & Dr Nic have worked to put together such Bosh + Vagrant scripts & get things rolling. I myself will be spending some considerable time on just that. But beyond that this is a good start in enabling data tier back ends.

How to close the gap, between absurdly simple application deployment and still arduous and difficult data tier deployment? For the next several years I think we’ll have cumbersome deployment practices around the data tier. There won’t be anything as elegantly simple as Cloud Foundry’s single line deployment or AppFog’s one click deployment of a web application. The best we can do at this time, is to streamline around pieces and architectures, and at least get them into a kind of simple 3 step deployment.

Please drop a comment or two on how you think we might simplify the data side of the PaaS toolchain. Also drop a few tweets in the twitterverse too, I’m sure that’ll be exploding as usual. I’m @adron, ping me.

Cheers, happy data architecting.

** the Deployconpanel will be at 4:30pm in Santa Clara on April 2nd. Come check it out.

Deploy a Framework Friday #2 with ASP.NET MVC 4

First let’s build a standard ASP.NET MVC Application (yes, you can do the same without the MVC Project, but I STRONGLY recommend never creating a standard ASP.NET application again, EVER). This quick run through assumes you’ve already setup a Cloud Foundry enabled PaaS w/ your already installed Iron Foundry components for .NET development. Get an Iron Foundry enabled account here, it’s 100% free, so no reason not to give it a go.

There are a few ways to do this. One, the click on the start page new project option.

Click on the start page new project option...

Click on the start page new project option…

Two, the menu based selection option.

Menu based new project option...

Menu based new project option…

Three, the command button option.

Click on the command button new project option

Click on the command button new project option

So pick your poison and then select the ASP.NET MVC 4 Project Template. On the next dialog, you’ll enter the information of where the project will go and choose the ASP.NET MVC 4 Web Application Project Template.

Selecting the ASP.NET MVC 4 Web Application (Click for full size image)

Selecting the ASP.NET MVC 4 Web Application (Click for full size image)

The next dialog select the Basic template, leave Razor selected, and I always leave the unit tests project there also. Once this is done we’re now ready to give our project a little taste of deployment.

Since we’re working in the land of GUIs and .NET, I’ll keep this to a completely GUI based deployment. However, don’t forget that the vmc-IronFoundry is available for pushing also. So don’t fear the CLI, but we’ll skip it for now (tip: It’s practically the same as the other deploys).

On the project in Visual Studio right click and select Push to Cloud Foundry.

Publish option on the context menu (click for full size image)

Publish option on the context menu (click for full size image)

A dialog will appear for the basic parameters needed for the deployment. Enter information that you need for your application and click ok.

Setting the parameters (click for full size image)

Setting the parameters (click for full size image)

Now when you navigate to your cloud in the Cloud Explorer UI (Start -> All Programs -> Iron Foundry -> Cloud Explorer) you’ll see the application running under that cloud environment.

As always, it is indeed THAT easy when using a Platform as a Service to deploy to!

Deploy a Framework Friday #1 with Ruby and Sinatra

Alright, just for fun I’m kicking off a new blog series. I’m going to publish a new “Deploy a Framework Friday” each week for about the next, well, bunch of weeks. There are a TON of frameworks that are available on PaaS Technologies.

This first entry I’m going to implement a simple Sinatra app with Ruby. Nothing fancy, simply a hello world and the respective deployment to a Cloud Foundry PaaS.

First, let’s whip out the super complex code (right, this isn’t complex, I just like sarcasm). The hello.rb file I created.

require 'sinatra'

get '/' do
  "Hello World!"
end

get '/route' do
  "Hello from a route URI!"
end

Next add a Gemfile & respective Gemfile.lock as such.

Gemfile

source "http://rubygems.org"
gem 'sinatra'

Gemfile.lock

GEM
  remote: http://rubygems.org/
  specs:
    rack (1.4.1)
    rack-protection (1.2.0)
      rack
    sinatra (1.3.2)
      rack (~> 1.3, >= 1.3.6)
      rack-protection (~> 1.2)
      tilt (~> 1.3, >= 1.3.3)
    tilt (1.3.3)

PLATFORMS
  ruby

DEPENDENCIES
  sinatra

Then deploy using the Cloud Foundry VMC.

vmc push

If you’ve forgotten, be sure to target and login first.

vmc target api.ironfoundry.me
vmc login

That does it. Yeah, not a whole lot to get started working on a Sinatra Project. For more information on Sinatra check out the main web presence here http://www.sinatrarb.com/.

For more information on Cloud Foundry or Iron Foundry click on the respective link.

For the code sample, check out the working “paasIt” code repo on Github.

Next week I’ll do a baseline ASP.NET MVC 4 Application and get it deployed.

IaaS vs. PaaS or Infrastructure vs. Platform and I Want Beer NOW!

A friend and now coworker of mine, Richard Seroter (@rseroter & Blog) decided to do a comparo. I took the infrastructure based deployment, ala IaaS and he took the platform based deployment, ala PaaS. What we’ve done is taken a somewhat standard ASP.NET MVC with Entity Framework, a SQL Server Database, a UX & UI design and got it running locally. From there we then deployed the same application the two respective ways to deploy the web application to a live environment. He took the Tier 3 PaaS (Iron Foundry + Cloud Foundry for the win) and I took the tried and true method of deploying via Windows 2008 Server instances via the Tier 3 Infrastructure.

Here are the steps I went through and for his steps check out this blog article on the PaaS deployment.

Part #1 – Get Some Servers Setup

First things first, I need two instances. If you’re following along, you can basically use whatever instances or server you want. AWS, Rackspace, or Windows Azure. Based on that there may be a few steps here or there you may need to alter, add or subtract from the process. One for the ASP.NET MVC Application and one for the SQL Server Database. The web app server doesn’t need a ton of resources, so I built it and scaled back RAM and cores to a single core.

ASP.NET MVC Web Server

ASP.NET MVC Web Server (Click for full size image)

In the next step here I’ve selected additional software to be installed on the instance. I’ll need .NET 4.0 so I’ve added this as shown.

Selecting .NET 4.0 for Addition to the Instance

Selecting .NET 4.0 for Addition to the Instance (Click for full size image)

After setting up the web server I also setup a database server. For the database server I made sure to allocate some decent resource, setting up 2 cores and 8 GB RAM. I also added the SQL Server installation based on Tier 3’s software packages so it would install automatically when the image is created.

All My Instances Running

All My Instances Building & Running (Click for full size image)

When I setup the SQL Server instance, I used a blue print feature that allows the SQL Server to be installed directly on the image. This of course saved me a lot of time. But it does add to the deployment time of the instance in the cloud.

Part #2 – Setting up Windows Server 2008

The first thing we’ll need to do is log into these machines and configure them, standard infrastructure stuff. Open up the Server Manager (which launches automatically on instances) and verify that we have IIS installed on the web server.

Database Server

Server Manager

Server Manager (Click for full size image)

Next log into the database server and verify that the SQL Server is up, running and create the initial database.

Thusly…

Using SQL Server Management Studio checking that the SQL Server Exists

Using SQL Server Management Studio checking that the SQL Server Exists

Once I had both of the servers up and running I got the application ready to deploy. First a little schema generation to use to deploy the database.

Don't Use "Script Database as..." option, use the "Tasks" option...

Don’t Use “Script Database as…” option, use the “Tasks” option…

Once the script is generated then transfer it and execute it against the database on the database server.

Execute the SQL Schema Create Script

Execute the SQL Schema Create Script (Click image for full size)

Always a good thing, even if all green lights are seen on the SQL execution, go in and make sure the tables are all there.

Web Server

Publish

Publish (Click Full Image)

Publish Application (click for full size image)For the web server, as long as IIS is already installed, the setup is fairly easy. First snag the compiled bits that need deployed. We’ll do a direct drop onto the server and get it running.

To get the compiled bits, right click on the Visual Studio Project and select publish. Add a deployment scenario, which I did and set it up to just spit the bits out to a directory. There of course a multiple options at this point to use FTP, WebDav or whatever your choice is. I’m not a particular fan of any of those in particular, they’re all fairly easy.

Deployment Publication Options

Deployment Publication Options (Click for full size image)

Interuption!!!

At this point I actually got hit with the “.NET 4.0 isn’t installed…” which it should have been. I opened up windows update and realized that it had not successfully executed nor had the .NET 4.0 install. This happens with all sorts of instances, regardless of provider, so make sure that the bits we need are installed. Also, with Windows, it’s a really good idea to get windows update turned on.

Back to Deployment

Now that we have the built bits just copy them onto the web application server into the inetpub wwwroot directory. Once you have that copied over you would be able to navigate to the IP of the machine this is setup on. At this time you may also want to setup a cname or a-record to point to the IP, so you can use a friendlier URI.

Retrospective

Now think about what has just gone on for a moment. We had to literally build out machines, add software and more. There were a lot of steps. This takes anywhere from 30 minutes to a few hours of actual work. In a larger business or an enterprise environment it could get extended out even further. Because of the extra complexity it could also end up broken, requiring extra troubleshooting and coding. There could even be a host of odd one off configuration issues with the hosting software itself.

Imagine you wanted to host an ASP.NET, PHP, Ruby on Rails and a Node.js App on the Server. That would be almost impossible. Consider how much extra configuration knowledge an ops person would need to troubleshoot each one of those frameworks. Just sit back and contemplate the complexities involved for a moment. All the complexity goes away with something like Cloud Foundry or Open Shift. With someone managing that system for you, such as us here at Tier 3 with our Web Fabric PaaS, AppFog, Cloud Foundry, or one of the other providers even more of the complexities just disappear.

Time for Summary & Beer

With all the steps and individual tasks needed to get something running in an IaaS Environment, go check out how slick getting something up and running with a PaaS style environment. The juxtaposition between what Richard had to go through versus what I had to go through is pretty significant. Simply put, for the vast majority of all application development can be done against a PaaS Environment and likely should. Digging deeper into the infrastruture elements is rarely needed except in rare scaling circumstances, such as the volume that Facebook, LinkedIn or Netflix deal with. Even then, as has been stated by these companies, they have a PaaS of their own they often build software to. So why not have this ability where you build software?

One of my key metrics, and I’ll be elaborating on this metric more in the future, is when I get to head out of the office for the day, relax, have a beer, and think about what I’ll get to create next. I call this my “Beer Enabler Measure“. PaaS technologies make it much easier for me to get to the relaxing part of my day a lot faster than IaaS technologies, and both of these make sure that I’m not pulling an all nighter without a beer like traditional hosting environments often do.

In the end, sure, infrastructure can be important and can help in transitioning legacy applications into an easier to manage environment. Today though, if you’re doing web application dev of any type, it should be deployed against a PaaS Environment either private or public.