Buildkite, an extremely quick overview

According to the guys @ Buildkite

Buildkite is a CI and build automation tool that combines the power of your own build infrastructure with the convenience of a managed, centralised web UI. The open source agent is installable on any machine or network and provides a simple, repeatable process for running builds using the same tools you use for development and production. Buildkite’s centralised web service handles job distribution, agent monitoring, integration with GitHub and Slack, and provides everyone on your team with an elegant interface for monitoring builds, tailing logs and downloading artefacts.

You can imagine it like a tool which allows you to do CI, automate deployments and updates by utilising your own scripts and build steps on your own infrastructure.

So, with that really quick intro out of the way, let me explain why I like Buildkite.

Script and Automate All the Things

Build steps are usually defined in a buildkite.yml file in the repository for which a pipeline was created. Usually the very first step is configured directly using the Buildkite web-UI - that would be the buildkite-agent pipeline upload step which uploads the buildkite.yml file.

That would look something like this:

steps:
  - label: 'Build stuff'
    command: 'docker build -t my_app:$BUILDKITE_COMMIT .'

  - wait

  - label 'Run CI stuff'
    command: 'docker run my_app:$BUILDKITE_COMMIT random_test_script.sh'

  - label 'Run even more CI stuff'
    command: 'docker run my_app:$BUILDKITE_COMMIT random_test_script2.sh'

  - wait

  - label 'Push'
    command: 'docker push my_app:$BUILDKITE_COMMIT'

  - block: "Deploy"

  - name: "Deploy"
    command: 'docker run --rm=true my_app:$BUILDKITE_COMMIT random_deploy_script.sh'

In this example we assume that we’re building a Docker container for our app, then we run our test scripts, we push it and we run or deploy scripts. Now, the thing is, this is pretty static. And yes, we have already automated a lot - testing, deploying, etc, however maybe we don’t want to run the same scripts in every scenario. We may want to run some extra CI stuff or even change our deploy script.

What we’re already doing is already really useful, but we can optimise it even more.

For example, we can dynamically generate our build steps every time we run a build. Instead of our first step being buildkite-agent pipeline upload we can make a script that generates our buildkite.yml and then have our first step be something like this: ruby generate_script.rb | buildkite-agent pipeline upload. Buildkite already provides us with a lot of useful ENV variables, such as the commit SHA, build ID, etc, so we can make our build script compare between commits, see if there is something extra we need to do this time (for example a database migration before a deploy) and automatically generate our build steps accordingly. Imagine integrating such a system with a Slack bot - you could deploy with a simple command from your #devops channel. :smiley:

Nodes for Buildkite

Buildkite agents run on your own infrastructure, this means that you can easily automate it’s scaling in the cloud. For example if you use bigger nodes and your run the Buildkite agent in Docker you can automatically start more Docker containers with the Buildkite agent, which will allow you to run more builds at the same time. In AWS you can configure an autoscaling group which can scale in and out when required. This means that you don’t need to run any Buildkite agents if you don’t have anything to build. Furthermore, if you provision your infrastructure with Terraform you can easily update anything (for example the Buildkite agent) by adjusting the launch configuration of your autoscaling group.

Conclusion

As a whole you can see what you can do with Buildkite. I use it both for personal projects, such as this blog, and at work. It integrates really well with your workflows and is highly customisable. It is definitely for anyone who wants a good CI tool with some cool functionality.