Counting Clouds

Let's Talk Immutable

Fri, 15 Oct 2021 09:12:00 +0200

Immutable OSs

Matthew Waanders recently brought to my attention a new desktop OS by Fedora: Kinoite. It is based on Fedora Silverblue, but with a KDE desktop environment. What is really fascinating about it is the idea behind both OSs. They’re both immutable desktop operating systems built around rpm-ostree and that is exactly what I’d like to focus on in this post.

Usually operating systems use package installers like apt, dnf, yum on Linux and even homebrew on macOS. These focus essentially on downloading packages and installing them in a local directory where they can be accessed by the user or other apps. This approach, although fast and quite flexible does have a drawback: its live updating of packages without easy stateful rollback. Sometimes packages can use different libraries across OSs and maybe for your specific one that you are installing them on, they’re quite simply not tested enough. So what can be done to improve this?

Enter images

We know images from the container world. We package an application and all of its dependencies in an image which can then be replicated exactly on other machines. What if we take this concept and apply it to OSs? What if we make a repo for images of apps ready to use with the correct state and have users pull them to their local OSs from there?

This is exactly what rpm-ostree is doing. The packages to be installed are all on a remote server that OSs can synchronise from. If a package exists on the remote server, then it should be synced to the OSs as well. As the rpm-ostree project self describes it is almost like git where people push to and pull from remote.

The main benefits of this are consistency and scalability. If you have a big fleet of EC2 instances to manage for example, you want to make sure that your package management system can handle that scale and do it consistently. We want the guarantee that once deployed a package will run the exact same way on one instance and every other around it. Of course, I’m an advocate for fully immutable infrastructures where instead of updating running EC2 instances you should just replace them with a newer AMI with all required updates, but that is a topic for a different post.

Let’s go a bit further in the architecture of such an OS. Let’s take the following filesystem for example. These are the default mount points as show in the Fedora CoreOS docs:

TARGET        SOURCE                                                   FSTYPE  OPTIONS
/             /dev/vda4[/ostree/deploy/fedora-coreos/deploy/$hash]     xfs     rw
|-/sysroot    /dev/vda4                                                xfs     ro
|-/etc        /dev/vda4[/ostree/deploy/fedora-coreos/deploy/$hash/etc] xfs     rw
|-/usr        /dev/vda4[/ostree/deploy/fedora-coreos/deploy/$hash/usr] xfs     ro
|-/var        /dev/vda4[/ostree/deploy/fedora-coreos/deploy/var]       xfs     rw
`-/boot       /dev/vda3                                                ext4    ro  

Every other mount point here, with the exception of /etc and /var, are immutable. Only rpm-ostree, and in the case of Fedora CoreOS bootupd, can make changes to the rest of the filesystem.

This way of managing filesystems is secure and efficient for servers, but now projects like Kinoite and Silverblue have brought it to the desktop as well. I am definitely excited to see those in action and play around with them and I encourage you to do the same.

The Complexity of Kube

Thu, 29 Jul 2021 17:40:00 +0200

Kubernetes and its features

I recently read an article in the Dutch Eos Psyche & Brein magazine titled “Waarom je de makkelijke oplossing over het hoofd ziet”. It goes about explaining why is it that when faced with a challenge people tend to pick an additive solution in contrast to a subtractive one. The author cites the article “People systematically overlook subtractive changes” from Nature where the authors describe the results of their research about why people tend to chose solutions that add stuff to solve a problem instead of the, in a lot of cases, simpler way of removing stuff instead.

This got me thinking about the current state of the container orchestration platforms and the way that they are designed and built. Bear with me for a moment.

From the inception of Borg (“Large-scale cluster management at Google with Borg”) the architecture of container management systems has been, just like the applications they were made to run, highly distributed, and that makes sense. Why would we make one single application that has to manage critical workloads across thousands of machines? That would be the biggest weakest link in the chain if there ever was one. In no way do I want to suggest that distributed, decoupled admin planes are a bad idea, on the contrary. However, one of the doors that such a design approach opens is the possibility of adding new functionality.

Let’s look at the Kubernetes API for example. Over the years new APIs and resources have been added that can manage practically anything in your application’s lifecycle: Ingresses, Jobs, Persistent Volumes. Kube has become a one-stop-shop for everything needed to run distributed applications at scale. But then you start to wonder: does it need that? Do we want to have everything being managed and stored by one singular system on a comparatively small amount of storage (etcd)? At wat point is Kube just another layer of AWS-like functionality on top of AWS?

If we zoom in to one specific resource, the ConfigMap, we can already see one of those extra functionalities that you wonder if we really need. Applications use a lot of environment variables for configuration on runtime. Kube provides us with a handy way of storing and giving the applications those environment variables, or even mounting whole files to them, whenever needed. All of that data get stored in the central control plane in etcd. If you use a managed Kubernetes solution you could say that that data is safe, however if you are managing it yourself, you have a lot of weight on your shoulders. Why not use something already existing with a much higher uptime and reliability guarantees? Something like the AWS SSM Parameter Store for example. Applications can get everything they need from the Parameter Store in a safe way without ever having to rely on plain-text environment variables in their runtimes.

I can give more examples: why rely on volumes or EFS to write files to if you can just write them to S3?

In any case, this is what I am trying to say: do we really need to add more complexity to what a container orchestrator or can we simplify it and make it easier for people not only to manage it, but to contribute to it?

Terraform 0.12's Template Syntax

Fri, 24 Aug 2018 18:38:00 +0200

Terraform is not a programming language

And this is exactly why in some cases you end up coming up with maybe weird and unnecessarily complex code to achieve something that you can easily fix if you had the conveniences of a programming language (like if else statements, for loops, etc.). There have been plenty of cases where you would want to generate something in your cloud init on the fly or just add something to some other configuration file, but you couldn’t extract the exact data you need because you want to loop over something or only add stuff that meets a certain condition.

Template Syntax in Terraform 0.12

Among a lot of other absolutely great features, which I’ll talk about in other blogposts, we finally get to play with if else statements and for loops. HashiCorp have a blogpost about it and some of the things you’d be able to do in the latest release. I’m super excited to get my hands on this and play with it when it releases. I’m sure a lot of you would find it absolutely amazing as well.

Have fun Terraforming. 😃

CoreOS and RedHat

Sat, 21 Jul 2018 20:28:00 +0200

CoreOS joins RedHat

As you may have heard in recent blogposts from CoreOS and RedHat - CoreOS and all of their projects are joining RedHat. This is extremely interesting, especially for me since I really like Container Linux as an OS. There are, of course, going to be changes to the rest of the CoreOS projects, like:

Tectonic is going to be integrated into OpenShift
The Quay registry is going to become RedHat Quay
Projects such as etcd, Ignition, Clair, etc. will be moved to RedHat as well

However, in this post, I’d like to talk about two things:

Container Linux and the way it is going to be integrated in the RedHat product lineup
The Operator framework

Fedora CoreOS

So, Container Linux is becoming an official Fedora project under the name Fedora CoreOS. The official post on the Fedora blog touched the subject of how this is going to look like.

From what we can see the new CoreOS is going to be build from the ground up based on Fedora technologies. This means that the old Container Linux is something that we’re not going to see in its original state. It is known that some technologies, such as Ignition, will be kept in the new OS, however the old update engine will probably be updated to something else.

We also know that technologies from the current Fedora Atomic Host, like SELinux hardening, will be added to Fedora CoreOS as well.

I’m still not exactly sure what I think of that, but I remain positive and hope to be able to play with the new Fedora CoreOS soon to see what it can do. Its definitely going to be exciting. 😃

The Operator Framework

CoreOS’ operators are amazing tools which I have been using more and more as of late. What exactly they are how they work is a topic for another blogpost. 😊

What RedHat has already announced is that they’re planning on developing a way of testing and delivering Operators on OpenShift. How that exactly is going to be implemented - we’ll just have to wait and see.

Conclusion

As a whole I find this topic extremely interesting, mainly because of my interest in the CoreOS technologies. I think that RedHat is definitely going to bring innovation and support for these and I’m really excited to see what they’re going to do. In any case I think the current community will be happy with the new technologies we’ll get to play with. 😃

Buildkite Agent V3

Wed, 11 Apr 2018 13:21:00 +0200

Hello Buildkite

I’ve written before about Buildkite and how awesome it is, but recently they have made it even more awesome than it was before. They have release v3 of their agent, which adds some really sweet functionality. Make sure to check their post about it as well. 😃

Welcome v3

You can read about all new features, such as agent annotation and environment variable substitution in the post I mentioned above, but what I really want to talk about are the plugins.

Plugins are essentially an extension to the native functionality of the agent itself. We currently run a lot in docker containers in which we install extra software that we need or we add extra steps with scripts that provide some extra functionality, logging in to AWS ECR for example.

Well, now we can just simply use a plugin. There is already a library with open-source plugins which you can directly use, or you can write a plugin yourself.

Let’s take the ecr-buildkite-plugin for example. You can use it to automatically log in to ECR without having to add an extra step with a custom script.

steps:
  - command: ruby build.rb
    plugins:
      ecr#v1.1.3:
        login: true

Awesome, right? 😃 There are much more plugins which I encourage you to go and play around with.

Go an try it out yourself

The Buildkite documentation offers extensive information on using the plugins and the rest of the new features, so definitely check it out together with the rest of the awesome new additions to the agent as well.

Quickstart - Building Alexa Bots

Mon, 12 Mar 2018 12:01:00 +0100

Introduction

Some time ago I got a Echo Dot to experiment with building bots and serverless computing. I have to say, the Echo Dot is the perfect device for starting with that since the development for it is pretty intuitive, well documented and, best of all, you can easily run your app on AWS.

Needless to say, make sure you have an Amazon account with which you can register for Amazon Developers and AWS.

So, let’s have a look how to get started with building bots quickly.

Skill Types, User Interaction and Components

Choosing the Proper Skill Type

Before starting to code anything you should have a clear idea what your skill is going to do and how your user is going to interact with it. Since this is a simple introduction you don’t have to make anything super complicated. Make a simple trivia game or a weather bot for example.

There are multiple skill types you can choose from:

Custom Skill
Smart Home Skill
Video Skill
Flash Briefing Skill

For a more in-depth look of the different types of skills you can check the Alexa Skills Kit documentation.

For these kind of bots you’d have to go for a “custom skill”, since what we’re trying to do is not related to any of the other supported uses.

User Interaction and Components

When going for a custom skill you have to define the way the user interacts completely by yourself. This means taking care of things like:

Intents
Slots
Sample utterances
Interaction model

So, what’s what?

Component	Function
Intents	What a user can do with a skill. For example - get weather for a specific city, set default city, etc. You would usually name them the same way in your code - “GetWeatherForCity” or “SetDefaultCity”, etc.
Slots	Essentially the variable in your intent. For example the “GetWeatherForCity” and “SetDefaultCity” intents will require a city to be defined. In this case we will have a “City” slot.
Sample utterances	A list of phrases a user might use when communicating with the skill. For example a sample utterance might be “weather in Amsterdam”.
Interaction model	The mapping of intents and sample utterances. For example the utterance “weather in my city” will be mapped to the “GetWeatherForCity” intent.

Before you start building any code think about how the user is going to interact with your bot. Start by:

Making intents based on the different actions users can take.
Defining the variables in the intents and making slots for them.
Thinking of sample utterances.
Mapping your intents to the sample utterances.

The Alexa Skills Console

This is where we’re going to do all of the necessary configurations of the skill and build our interaction model. You can access it through the Amazon Developers portal.

The process is pretty simple and straightforward:

Create a new skill by clicking on the “Create Skill” button.
Give it a name and a locale.
Choose a model - on our case “custom”.
Click on “Create Skill”.

After the skill has been created you will be presented with a console where you can build, test and publish your skill. In the beginning page there is a handy checklist which I recommend you to follow to get started easily. It will guide you through:

Configuring an invocation name.
Configuring your intents, sample utterances and slots.
Building the interaction model.
Configuring the endpoint for your code.

The code

When you’re ready with designing the next step is to put everything in our code.

I recommend having a look at some of the sample code which is available on the Alexa organisation in GitHub. Depending on your preferences have a look at:

I personally work mainly with Ruby and Python, and since Ruby is not an option for AWS Lambda I’m going to be using Python.

What I usually do is split the helpers, intents and main functions in separate files. This makes it more readable in my personal opinion.

Have a look through the sample code and tinker around with it. You can familiarise yourself with the request and response JSON interface on the Alexa developers documentation. The documentation details the request format (what is going to be sent to your code) and the response format (what is going to be sent back to your Alexa bot), and that is, at the end of the day, what is the most important part. The rest of the code is up to your preferences.

AWS Lambda

You can run your code on AWS Lambda which makes it extremely simple. You can connect it easily to your Alexa bot - just provide the ARN of your Lambda function and tada, you’re done. 🎉

Make sure to package your code and any extra modules you may have used to a zip file and upload it directly to Lambda or S3. You can automate it pretty easily with a Makefile or something of the sorts. The AWS documentation has a page about creating NodeJS and Python deployment packages for AWS Lambda which are pretty useful.

Testing

Its pretty easy to test your bot. Use the Alexa developers console to test your both using voice, text or JSON input. You can also connect it to an Echo device you own and test it there as well.

Conclusion

So, this is how to get started building Alexa bot in a nutshell. Have fun creating awesome bots and coming up with new ideas. 😃

Why I like Buildkite

Thu, 14 Sep 2017 17:41:00 +0200

Buildkite, an extremely quick overview

According to the guys @ Buildkite

Buildkite is a CI and build automation tool that combines the power of your own build infrastructure with the convenience of a managed, centralised web UI. The open source agent is installable on any machine or network and provides a simple, repeatable process for running builds using the same tools you use for development and production. Buildkite’s centralised web service handles job distribution, agent monitoring, integration with GitHub and Slack, and provides everyone on your team with an elegant interface for monitoring builds, tailing logs and downloading artefacts.

You can imagine it like a tool which allows you to do CI, automate deployments and updates by utilising your own scripts and build steps on your own infrastructure.

So, with that really quick intro out of the way, let me explain why I like Buildkite.

Script and Automate All the Things

Build steps are usually defined in a buildkite.yml file in the repository for which a pipeline was created. Usually the very first step is configured directly using the Buildkite web-UI - that would be the buildkite-agent pipeline upload step which uploads the buildkite.yml file.

That would look something like this:

steps:
  - label: 'Build stuff'
    command: 'docker build -t my_app:$BUILDKITE_COMMIT .'

  - wait

  - label 'Run CI stuff'
    command: 'docker run my_app:$BUILDKITE_COMMIT random_test_script.sh'

  - label 'Run even more CI stuff'
    command: 'docker run my_app:$BUILDKITE_COMMIT random_test_script2.sh'

  - wait

  - label 'Push'
    command: 'docker push my_app:$BUILDKITE_COMMIT'

  - block: "Deploy"

  - name: "Deploy"
    command: 'docker run --rm=true my_app:$BUILDKITE_COMMIT random_deploy_script.sh'

In this example we assume that we’re building a Docker container for our app, then we run our test scripts, we push it and we run or deploy scripts. Now, the thing is, this is pretty static. And yes, we have already automated a lot - testing, deploying, etc, however maybe we don’t want to run the same scripts in every scenario. We may want to run some extra CI stuff or even change our deploy script.

What we’re already doing is already really useful, but we can optimise it even more.

For example, we can dynamically generate our build steps every time we run a build. Instead of our first step being buildkite-agent pipeline upload we can make a script that generates our buildkite.yml and then have our first step be something like this: ruby generate_script.rb | buildkite-agent pipeline upload. Buildkite already provides us with a lot of useful ENV variables, such as the commit SHA, build ID, etc, so we can make our build script compare between commits, see if there is something extra we need to do this time (for example a database migration before a deploy) and automatically generate our build steps accordingly. Imagine integrating such a system with a Slack bot - you could deploy with a simple command from your #devops channel. :smiley:

Nodes for Buildkite

Buildkite agents run on your own infrastructure, this means that you can easily automate it’s scaling in the cloud. For example if you use bigger nodes and your run the Buildkite agent in Docker you can automatically start more Docker containers with the Buildkite agent, which will allow you to run more builds at the same time. In AWS you can configure an autoscaling group which can scale in and out when required. This means that you don’t need to run any Buildkite agents if you don’t have anything to build. Furthermore, if you provision your infrastructure with Terraform you can easily update anything (for example the Buildkite agent) by adjusting the launch configuration of your autoscaling group.

Conclusion

As a whole you can see what you can do with Buildkite. I use it both for personal projects, such as this blog, and at work. It integrates really well with your workflows and is highly customisable. It is definitely for anyone who wants a good CI tool with some cool functionality.

Orchestrating the Orchestration

Fri, 25 Aug 2017 21:37:00 +0200

Introduction

Docker with docker-compose is an amazing combination for local development and testing, but it’s not something you would simple use on it’s own when running containers in production, and especially in distributed environments. We want to automate as much as possible so that everything can be done with one simple command in Slack. (And yes, chatops is quite the efficient way of working with your environments.) When we’re working with (really) big environments it’s no longer enough to have deployment scripts that SSH to each node, or tools like “capistrano”.

So, what can we do about inefficient deployments, and as a whole, production environments?

Enter Kubernetes

After poking around a couple of other solutions, such as smarm mode in Docker, Kubernetes felt like the more mature, optimisable and scalable option. And although it has it’s upsides, it comes with a lot of quirks.

In order to get Kubernetes to qualify for production you need to do a couple of things yourself.

Plan it’s deployment (Terraform, scripts, etc.)
Configure monitoring tools (Heapster, InfluxDB, Grafana)
Configure logging tools (Fluentd, Elasticsearch, Kibana)
Configure RBAC access control
Set up autoscaling
Figure out updating

If you only deploy Kubernetes by itself you basically get a really fancy environment which can do a lot for you, but it will be insecure, prone to failure and tedious to manage.

Now, the question is - weren’t orchestration tools supposed to make it easy for us to work with distributed production environments? To be honest, that’s pretty idealistic. In the current state Kubernetes requires manual maintenance and although it does not necessarily mean that it is complicated, but why can’t we automate it?

Personal experience

In the company I currently work for we ended up crating a step-by-step plan for upgrading Kuberentes. It basically consists of

SSH-ing to every node and updating the running kubelet
Updating the cloud-config of the worker autoscaling group in AWS and replacing the master nodes one by one with the new cloud-config

As you can see this can be a tedious process. It requires SSH-ing to nodes which work production loads and updating a lot of configurations. This is already quite the risk. The last thing we want to do is SSH to nodes and manually update them.

Tectonic by CoreOS

Tectonic is actually a pretty nifty solution for managing Kubernetes. You can use it to deploy a Kubernetes cluster with Terraform (yes, Tectonic uses Terraform), update it, manage the nodes and also work with Kubernetes itself - manage deployments, services, etc.

The software comes with a CLI and GUI installers for deploying Kubernetes and a handy dashboard for further management of the working cluster.

What I personally think is really nice is the built-in Prometheus monitoring for your cluster, so you don’t have to worry with the whole Heapster and InfluxDB setup. If anyone has had to deal with setting up RBAC authorisation in Kubernetes, you’de be happy to hear that you can manage that via Tectonic too.

Orchestrating the orchestration

Now, my question is - why isn’t Kubernetes shipped with Tectonic by default? A whole extra layer of orchestration has been added for the purpose of orchestrating our orchestration tool. It is kinda funny for me to see that so many systems exist on top of each other when it can all be boiled down to one. I think it would be amazing if Kubernetes just shipped with Tectonic by default and we would not have to worry about all of the management.

For anyone that would like to start using Kubernetes in production - take a look at Tectonic. 😃

Terraform and IAM

Sat, 05 Aug 2017 17:53:00 +0200

Introduction

In my previous post I talked about deploying a Jekyll blog on AWS and I mentioned the possibility to manage the IAM configuration (which I then explained how it can be done via the AWS console) with Terraform. So, let’s have a look.

There is a solid reason why people use the infrastructure-as-code approach. Not only can we keep everything documented and version-controlled, but we can also do all of our management from the comfort of the CLI.

Of course, there are other infrastructure-as-code implementations, such as Puppet, but I personally use Terraform, mainly because of it’s simplicity and capabilities.

If you are new to Terraform I advise you the check it’s documentation and play around with it. It’s definitely worth it.

Note: In this post I will not be talking about setting up and configuring Terraform.

Making the IAM policy

If you want to easily generate an IAM policy you can use the Policy Generator, located in the “Create Policy” section of your IAM dashboard.

We want our deployer to have the following permissions:

Create and manage CloudFront distributions
Create and manage S3 buckets

These options are quite permissive, but if we want to limit them we have to do some parts of the deployment process manually, like:

Creating an S3 bucket
Creating a CloudFront distribution

I’m fine with the deployer having the permissions to create CloudFront distributions and S3 buckets, so for now I’ll allow it to have all permissions for S3 and CloudFront.

You can play around with the policy generator and when you’re ready review your policy. Mine looks like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "<GENERATED_SID>",
            "Effect": "Allow",
            "Action": [
                "cloudfront:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "<GENERATED_SID>",
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Refining the policy for Terraform usage

Terraform will take care for generating IDs, etc. so we can remove the Sid fields from our JSON.

You’ll end up having something like this:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "cloudfront:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:CreateBucket"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

The Terraform part of everything

Now we can make our .tf files, which will define what Terraform should do.

What we’re going to need is:

An account
A user policy
An IAM access key

User

That’s pretty simple, I have the following configuration:

resource "aws_iam_user" "blog_deployer" {
  name = "blog_deployer"
}

IAM Access Key

This would look something like this:

resource "aws_iam_access_key" "blog_deployer_access_key" {
  user = "${aws_iam_user.blog_deployer.name}"
  pgp_key = <base-64 encoded PGP public key>
}

By adding your PGP public key you can then get the IAM user’s secret key from Terraform.

IAM User Policy

This is what your user policy should look like in Terraform:

resource "aws_iam_user_policy" "blog_deployer_user_policy" {
  name = "blog_deployer_user_policy"
  user = "${aws_iam_user.blog_deployer.name}"

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "cloudfront:*"
          ],
          "Resource": [
              "*"
          ]
      },
      {
          "Effect": "Allow",
          "Action": [
              "s3:CreateBucket"
          ],
          "Resource": [
              "*"
          ]
      }
  ]
}
EOF
}

Output

There are a couple of ways to get the IAM user’s access ID and secret key, the easiest of all being to just output it to the console, which I would not recommend since it’s totally not secure.

You can output the user’s ID by including this in your .tf file:

output "id" {
  value = "${aws_iam_access_key.blog_deployer_access_key.id}"
}

For outputting your secret key add the following:

output "secret" {
  value = "${aws_iam_access_key.blog_deployer_access_key.encrypted_secret}"
}

To get the secret key in a secure way run terraform output secret and then you can decrypt it.

Applying the Terraform configuration

If you’re working in a new directory remember to run terraform init to download the Terraform AWS plugin.

When you’re ready you can run terraform plan and you should see something like this:

Refreshing Terraform state in-memory prior to plan...

Plan: 3 to add, 0 to change, 0 to destroy.

Check to make sure that we’re deploying the correct configuration and then run terraform apply.

You’ll see the following when Terraform has applied everything:

Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Outputs:

access key = <ACCESS_KEY>

Conclusion

That’s how you can deploy an IAM user with an attached user policy with Terraform. You can use the provisioned user with s3_website to deploy to S3 and CloudFront.

Jekyll, S3 & CloudFront

Thu, 03 Aug 2017 19:30:00 +0200

Introduction

I’ve been poking around with Jekyll in my free time and I have to say I really enjoy working with it. It, unlike other platforms, such as Ghost, can not only be deployed easily, but you can version control it with Git and automate a lot of the work.

I am planning, in future posts, to have a look at automating Jekyll deployment. A script triggered after a new push is something that I find really cool.

From what I’ve noticed a decent chunk of people host their blogs on GitHub Pages, which, although a decent service, still lacks SSL on own domains. I, myself, have the urge to always SSL web traffic, so I decided to have a look for a different hosting method.

Enter AWS

I work with AWS quite a lot, both for my work and school, and it is the platform I am most familiar with. Keeping in mind how cheap and scalable S3 is, especially for hosting static websites, I think it’s a pretty decent option for hosting a Jekyll blog.

So, this is basically how I deployed my blog:

Prerequisites

Have a jekyll blog
Have an AWS account

Provision a deployer account in IAM

It is a good idea to make an IAM account for s3_website with only the needed permissions. I would personally not like it if it had full permissions to my AWS account.

I will be talking about managing AWS with Terraform in future posts, so for now we’ll just use the AWS console.

Creating an IAM policy

So, let’s go ahead and create an IAM policy for our deployer.

In your AWS console hit the “Create Policy” button in IAM -> Policies and select the Policy Generator option. The Policy Generator is a super easy tool for making IAM policies by just selection the resources and actions that should be allowed to be used.

I did not refine the policies that much, and yes, I really should, but just for starters I came up with the following policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "<GENERATED_SID>",
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "arn:aws:s3:::vasko.io"
            ]
        },
        {
            "Sid": "<GENERATED_SID>",
            "Effect": "Allow",
            "Action": [
                "cloudfront:*"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

So, when you’re done, review your policy, test it out, give it a name and hit “Create Policy”.

Creating an IAM account

So, after we’re done with the policy we can proceed with creating the account. In IAM go to “Users” and hit “Add User”. Give it a recognisable name, such as s3_website_deployer or something of the sorts.

Please make sure to tick the “Programmatic access” checkbox at “Access type”.

Next give it your permission by selecting “Attach existing polices directly” and searching for your policy.

Now you can review and create your account. In the final screen you will be presented with the ID and secret key, which you need to copy to the s3_website.yml config file.

Creating a certificate in “Certificate Manager”

NOTE: Make sure you’re in the N. Virginia region for this.

We can provision a certificate for your CloudFront distribution for free from the AWS Certificate Manager.

Go to Certificate Manager from your dashboard and hit “Request a certificate”, type in your domain, review and request. AWS will send you a mail for approval.

After you’ve approved it you can go back to your preferred region.

Configuring `s3_website`

First have a look at the official repo on GitHub. You can find most of the information you need there, but just to sum up the deployment process, here is what you need to do.

Install the gem by running gem install s3_website
cd to the directory where your blog is
Run s3_website cfg create to generate the configuration file
Put your AWS credentials by either:
- Copy-pasting your access credentials
- Using ENV variables
Put your S3 bucket name
Configure which files should be excluded from download by listing them under exclude_from_upload
Configure CloudFront

Configuring Credentials

I ended up with the following as my configuration:

s3_id: <%= ENV['AWS_ID'] %>
s3_secret: <%= ENV['AWS_SECRET'] %>

Configuring S3

For S3 make sure to put your

S3 bucket name (if it does not exist yet it will make it for out)
S3 endpoint(if you wish to use something else than the default one)

You should end up having something like this:

s3_bucket: my.bucket.name
s3_endpoint: eu-west-1

Configuring files to be excluded from upload

You probably don’t want anything extra to be uploaded than just your blog, so we can just remove some files from the upload list. For example:

exclude_from_upload:
  - Rakefile

Configuring CloudFront

You do not have to add a CloudFront distribution ID yet, since s3_website will make a new one for you. You should, however, configure the CloudFront default cache behaviour, aliases and invalidations. I have the following configuration:

cloudfront_distribution_config:
  default_cache_behavior:
    min_ttl: <%= 60 * 60 * 24 %>
  aliases:
    quantity: 1
    items:
      - vasko.io

cloudfront_invalidate_root: true
cloudfront_wildcard_invalidation: true

After applying the configuration, s3_website will present you with a prompt asking whether you want to use CloudFront or not. You can type in Y and you will be given your CloudFront distribution ID. Then you can add it to the config file.

CloudFront SSL

When all of that is done go to CloudFront in your AWS dashboard and locate your distribution. From there you can hit the “Edit” button and configure SSL by selecting “Custom SSL Certificate” and then picking the correct certificate for your domain.

Restrict S3 access

Also, you can block access to the S3 bucket from the outside, so that people can only access your blog via CloudFront. You can do that by going to your CloudFront distribution in your AWS console -> “Origins”. Select the default origin and edit it. You will see that you get the option to “Restrict Bucket Access”, so go ahead and choose “yes”. Then create a new identity and choose “Yes, Update Bucket Policy”. CloudFront will automatically update the S3 bucket’s policy.

If the option to restrict bucket access is not available click on “Origin Domain Name” and re-select the S3 bucket that you’re using to host the blog.

Now, after CloudFront has updated the S3 policy, you will find out that you can still browse to it. That’s because CloudFront added the restricting policy, but it never removed the allowing policy. To fix that go to your S3 bucket in the AWS dashboard -> Permissions -> Bucket Policy. Remove the block that allows public access and leave only the CloudFront Origin Access Identity and save it.

Your end file should look something like this:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "2",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity <CLOUDFRONT_ID>"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<BUCKET>/*"
        }
    ]
}

Deploying your blog

Now, after all of the configuring is done we can go ahead and deploy our blog.

We need to do the following:

Generate our blog by running jekyll build or bundle exec jekyll build.
Push the website to S3 by running s3_website push.

Give CloudFront some time to run the invalidation and then browse to your blog. 🎉

So, is it ready yet?

Yes, yes it is. 😃 Your blog should now be accessible via your own domain. Any time you add more content just follow the deployment steps from above and your blog will be updated.

In future posts I will also look into automating the process with deployment scripts and Terraform.

So, how do you deploy your blog?

Counting Clouds

Let's Talk Immutable

Enter images

The Complexity of Kube

Terraform 0.12's Template Syntax

Terraform is not a programming language

Template Syntax in Terraform 0.12

CoreOS and RedHat

CoreOS joins RedHat

Fedora CoreOS

The Operator Framework

Conclusion

Buildkite Agent V3

Hello Buildkite

Welcome v3

Go an try it out yourself

Quickstart - Building Alexa Bots

Introduction

Skill Types, User Interaction and Components

Choosing the Proper Skill Type

User Interaction and Components

The Alexa Skills Console

The code

AWS Lambda

Testing

Conclusion

Why I like Buildkite

Buildkite, an extremely quick overview

Script and Automate All the Things

Nodes for Buildkite

Conclusion

Orchestrating the Orchestration

Introduction

Enter Kubernetes

Personal experience

Tectonic by CoreOS

Orchestrating the orchestration

Terraform and IAM

Introduction

Making the IAM policy

Refining the policy for Terraform usage

The Terraform part of everything

User

IAM Access Key

IAM User Policy

Output

Applying the Terraform configuration

Conclusion

Jekyll, S3 & CloudFront

Introduction

Enter AWS

Prerequisites

Provision a deployer account in IAM

Creating an IAM policy

Creating an IAM account

Creating a certificate in “Certificate Manager”

Configuring s3_website

Configuring Credentials

Configuring S3

Configuring files to be excluded from upload

Configuring CloudFront

CloudFront SSL

Restrict S3 access

Deploying your blog

So, is it ready yet?

Configuring `s3_website`