A Look Inside the Deploy Process at Justworks

Justworks being a Ruby shop, we needed a way to deploy our code to our various servers in a way that integrates easily with the different tools we use in our codebase. We needed the ability to gracefully restart web servers like Thin and Unicorn, tools like Sidekiq and Gearman, managed by Bluepill, and more.

Naturally, we looked into Capistrano, a tool for managing remote servers and used for code deployment. While Capistrano worked for our purposes, as the engineering team started to grow, there were lots of features we wanted to have. The main one was some sort of UI with a queuing system for engineers.

Unlike other places I’ve worked where the DevOps team or a specific person was responsible for deploying code for other engineers, at Justworks, we wanted the flexibility of allowing every engineer to test, merge, and deploy their code independently.

Thus, on September 23rd 2013, Skylab was born!

Skylab takes its name from the 1970s NASA space station.

What is Skylab?

Skylab is a tool built in house that gives developers the extra features they need to deploy code on their own, without having to know much about Capistrano, as it abstracts that for them.

Some of the features of Skylab are:

• Multi environment deployments

• Code deployment as well as migration deployments

• Rollbacks

• Integration with Jenkins to show test status

• Integration with Slack for notifications

• Deployment history

• Deploy logs

• Deploy approvals

Skylab's Slack integration

It also includes a queuing system for engineers, which goes like this:

• Engineers queue themselves up for a deploy

• When it’s their turn, they merge their PR, which results in a SHA change in Skylab

• They can then trigger a build in Skylab of a deploy branch from master now that their changes are merged in

• After the build is complete, they request approval and start deploying to each environment

• Multiple engineers can deploy together by joining the same queue

Related Article: My Summer as a Software Engineering Intern at Justworks

The Deploy Process Flow

Let’s take a look at what a deploy process flow looks like. Here's a typical deploy flow:

• Engineer creates a PR in Github

• Jenkins starts a test suite against the pull request

• PR gets reviewed and approved

• Test suite passes

• Engineers queue themselves up in Skylab

• Engineer merges the PR

• Skylab’s SHA gets updated

• Jenkins master suite tests against master branch kicks off

• Engineer triggers a build of the deploy branch from the master branch in Skylab

• Once the build is done, an approval is required from a manager, PM or the Infrastructure team

• Once approval is given (also via Skylab), the engineer deploys to staging, then Sandbox and then Production, testing their changes on each environment

Skylab’s SHA gets updated

DB Migrations

The process for running DB migrations against each environment is the same as a regular code deploy. Migrations are usually run off-hours if they involve a schema change that may result in a table lock (adding an index, new column, etc.).

Reverting

If we ever need to rollback a code push/migration for any reason, the process is very similar:

• Engineer either reverts their PR they previously merged or creates a new PR with the revert

• PR gets approved

• Test suite runs

• Proceed with the normal code deploy

Compliance

From a compliance perspective, any code push should get explicit approval from someone other than the engineer deploying the code. In our case, we have a group of deploy approvers.

When an engineer merges their code and is ready for the “build” step, someone from the list of approvers has to hit the “Approve” button to allow the process to continue. Logs showing details of deployer and approver are collected for every deployment and are made available for audit review as needed.

The Pros and Cons of Skylab

The pros:

• Customizations, optimized for our specific use case

• Slack integration

• Notifications

• The ability to hold deploys (during payroll, for instance)

• Deploy approval system

• Ability to tie in graphs and logs

• Ability to run migrations

The cons:

• Takes effort to develop and maintain

• Not a standard, supported solution

• Only used for our main application, not a general solution; in order to use it for other things, it has to be modified

• Specific to Rails

• UI could use some love

Final Thoughts

I’ve seen many different deployment processes and tools in my career. While some worked better than others, success also depends on the requirement of the various engineering teams. Some companies put all the responsibility of merging, testing, and deploying code on one person, a “release manager”, and other organizations prefer to give more autonomy to the engineers.

Related Article: Justworks’ VP of Engineering: Why I Joined Justworks

I have been a fan of the latter since working at Justworks. I think it’s important for engineers to understand what is involved in shipping code and how tests are run, even if DevOps abstracts a lot of that for them.

Skylab has been super useful for us at Justworks, and I’m looking forward to seeing how we can leverage it as we work towards Dockerized deployments for our application.

Interested in becoming a part of the Justworks engineering team? Take a look at our job openings.