All the buzzwords and AWS abbreviations! Another post where I just want to note down something I thought was cool. Actually I have a lot more terraform related I'd like to write down, hopefully I will do more of it in the future.

Yesterday night I was doing some maintenance updating our loadbalancer. While I was at it I found a way to get rid of some previously hardcoded IAM credentials that were being used to upload backups of our HAProxy config via aws-cli to an S3 bucket.

Instead of creating a new IAM user with a policy to interact with a specific bucket, you can just give that policy to the EC2 instance's IAM role. Then inside the EC2 instance AWS cli is already installed with credentials for that role. At least on AWS Linux, may be different on other OSes. After giving the policy to the role it automagically just worked to interact with bucket without adding any extra credentials.

To add the policies to the role via terraform was super easy, it just needed the following:

resource "aws_iam_role_policy" "s3-gateway-buckets" {
  name   = "s3-gateway-buckets"
  role   = "${aws_iam_role.gateway.id}"
  policy = "${data.aws_iam_policy_document.s3-gateway-buckets.json}"
}

data "aws_iam_policy_document" "s3-gateway-buckets" {
  statement {
    effect = "Allow"

    actions = [
      "s3:*",
    ]

    resources = [
      "arn:aws:s3:::some-backup",
      "arn:aws:s3:::some-backup/*"
    ]
  }
}

It's almost 2019, and what better time to reminisce a bit and remember what I've learned the past year. I've been wanting to write this down for a while. Different things that I've learned while working at a startup in Berlin. These will be mostly work related, not specific things like how to write components in React. Rather professional working life type things. This post may come across as quite scattered. I'm writing down the things as I think of them. I also am writing this mostly for myself to reflect.

Some background: Since June 2017 I worked at a small Berlin startup Wellnow GmbH. The tl;dr; of Wellnow is essentially that it sells massages by coordinating Therapists with prospective clients. I joined the company as a Software Engineer, as part of the very small "tech" team of around three people. I've essentially been working both as a fullstack developer, and as a DevOps. At this point I've touched most of the platform behind the company, as well as the infrastructure that runs it. Initially the business was focused on the b2c market, i.e. selling to private customers, but we've just recently switched the focus completely to only focus on b2b, i.e. selling to other companies.

Learnings

In no particular order.

Keep a notebook. Quite often there's just too much to keep track of. It feels pretty natural then to write it down. One of the good things we have in our office is a ready supply of free little notebooks. I use mine daily to jot down ideas as well as to help divide up tasks into subtasks. At the end of the day I often also write down where I should continue the next day. I find this gives me good peace of mind when I leave work as I don't have to worry about forgetting anything.

Document to save time and thought. I've started to write down cheatsheets of commands to run in certain scenarios, e.g. when I need to rotate our Nomad cluster. Since we also run a few microservices in different languages and frameworks, many of which are not touched for months, I try to always ensure now that I keep a clear and concise README detailing how to run, test, lint, and debug the service. I've found documentation to be the most useful when managing our infrastructure. Keeping a diagram of the infrastructure helps refresh on how everything works when planning new changes. Having a playbook of commands for the servers helps in those oh-shit moments when something goes wrong.

Take charge of your own projects. Even if a project is assigned to you by a manager, it may be up to you to push for it to finish. Especially at a startup, where there is always so much going on. In some sense you have to be your own project manager for this.

From Autumn 2017 to Summer 2018 I worked on a project to rewrite most of the therapist location data to a new postgis geolocation backend. Even though most of the backend work was finished by the end of 2017, it took a few months more until we even started working on the frontend. Then the requirements changed back and forth until eventually the project was deployed.

Personally I could have stepped up more to push through this project. I could have worked on the frontend implementation myself, or I could have urged more in meetings and planning to focus on the project instead of continually postponing it.

Better to work focused. Perhaps more of an opionion, but I believe it is much better to be able to work focused on a single project, rather than work on multiple at once. Last year I had the problem that I was working on too many projects simultaneously, and I believe it lost me a ton of time as I had to constantly switch my focus between the tickets of these different projects. In the end, we also ended up scrapping a lot of these projects before they were even shipped. Had we instead focused on single projects at a time, we would have at least shipped something. Kind of frustrating. Like the previous point I know now that I need to step up a bit to voice my opinion on this.

Evaluate how much time you should spend on something. Not every project needs the quality of e.g. full test coverage and documentation. I find it useful to weight the amount of effort that should be put into the quality of a software project based on how much it will be used. Is it just a one-off script to migrate data, or is it a new backend API that will handle authentication? The former can likely be written as a quick python script without any formal testing and fanfare. The latter you probably want to give more care and attention to as it will stick around for a long, long time.

Before I joined the startup, I tended to go pretty heavy on the quality of a project. Even if it was just a trivial script that needed to be made. Now I try to keep this distinction in mind when starting any new project, especially in regards to testing.

Preparations for recruiting. Obvious in hindsight and not really a specific learning, but for me it was the first role where I had to be the interviewer as opposed to the person applying for the job. I think we had something that sort of worked. What helped me the most was to during the onsite interview have a clear script to follow or fall back on. I definitely have taken away some things which I'll be able to use in the future, and feel quite comfortable interviewing other engineers now.

Feel free to take initiative. At our startup it felt like there were always more fires to put out than people available, as such your coworkers will (probably) be glad if you show initiative and take responisiblity for a certain topic. Just be careful that you don't overextend yourself or you'll get burned. Of course, this might just be how it was at this startup. I imagine at larger companies there are likely already people invested the different topics and you may need to step more carefully. This is a positive experience which I think I will not find as easily outside of the startup culture.

Pick your battles. As a tech oriented person it's easy to butt in with your opinion everywhere. However at such a busy place as a startup, you don't need to be involved in every project. I found it better to not get involved in every tech discussion so that I could focus on my own tasks. I should trust that my fellow programmers can also make the right decisions without me, and if they need my opinion they can ask for it.

Try not to be too grumpy and give positive feedback. I think I can be quite pessimistic sometimes, especially as I spend a lot of my time fixing other people's code and with the stress that comes from being the inhouse sysadmin. At the same time, I may be affected by my native Swedish culture a bit too, where we can be a bit sparse on positive feedback.

I originally wrote this document for my work, where we were deep into a microservices architecture. The goal was to create a set of guidelines to be used when developing a new service. This way we could ensure some consistent quality for the code, as well as have it slot nicely into our then infrastructure.

We're now moving away from that architecture, but I don't want this document to go to waste. So I'm posting it here. Maybe someone else will find it useful as a template.

Minimum Requirements / Microservice Guidelines

This document details some requirements we would like to enforce for all of our microservices to ensure some level of quality. Use this document as a checklist to determine if your service is production ready. These requirements are not necessary set in stone, more a guideline.

In addition to these guidelines we strongly recommend following the 12 Factor App methodology.

Dependencies and Versioning

Dependencies locked to the correct versions and maintainable. A build must be reproducible in the future. Never lock to a third party's version control branch, e.g. don't use some github repo's 'dev-master’ in composer.

Semantic versioning should be used to categorize our builds.

If possible use a service like Evergreen or Greenkeeper to automatically keep your dependencies up to date.

For most projects, we should use Git-flow to organize our branches and keep a separate develop and master branch. It also helps us manage our releases. On github, develop should be the default branch.

Environment Variables and Secrets

Passwords, tokens, URLS to backing services and other variable parameters to your application must be passed as environment variables.

Each microservice must store it's environment variables in Hashicorp Vault and never ever in git.

Secrets must never be stored in a git repository. This includes SSL certificates.

Testing and Code Quality

There should be regression tests for the important parts for the service. This helps to ensure that when we make code changes in the future the original functionality still works as intended, or that we are at least alerted to the fact that we broke existing functionality.

There should also be a way to measure the test coverage. This helps find parts of the code which could be better tested.

Use linting tools for your programming language and framework, this helps you follow standardized code conventions and hopefully keeps your code more concise and clean.

Documentation

Dev/Contributor documentation must exist that tell the next developer how to maintain the project. E.g. clear instructions on how to install dependencies, build, run, and test the project. It should be possible to pick up the project in a few months time. A good place to put these is in the README, which is also something that every project must have.

Logging

Make sure that your docker image logs to stdout. This makes troubleshooting one step easier as then we won't have to shell directly into the container to view the logs and they will be picked up by filebeat.

Have different levels of logging depending on development/staging/production. Use a library that can handle different levels of logging, i.e.TRACE, DEBUG, INFO, WARN, ERROR are pretty typical log levels.

Slotting into our Infrastructure

Every microservice must have a Dockerfile. Likely it will also need a Docker repository to push builds to. For this we use AWS ECR. Typically this docker push step is done during our CircleCI build.

Each microservice must also have one or more Nomad job definition files per runtime environment. e.g. staging.json for staging, and production.json for production.

For continous integration we use CircleCI, as such the project must have a CircleCI configuration.

For error tracking we use Sentry, as such the project should send exceptions, errors and warnings to Sentry.

Each HTTP microservice should also have a simple healthcheck route, typically under http://service-name/health. This is then input to Statuscake and Consul, which periodically polls the service.

... but most importantly

The service must also work.

Checklist template

You can use the following markdown file to create a checklist via gist or github issue. This can be used for review before onboarding a new service.

### Dependencies and Versioning
- [ ] Dependencies locked and sane
- [ ] Semantic Versioning used to label builds.
- [ ] Automatically updated dependencies via Greenkeeper (or similar)
- [ ] Git flow configured with `develop` as the default branch on github.

### Environment Variables
- [ ] Passwords, tokens and other configuration are passed via environment variables.
- [ ] Environment variables stored in Vault.
- [ ] No secrets stored in the git repository.

### Testing and Code Quality
- [ ] Tests exist, run and pass
- [ ] Test coverage is measured
- [ ] Linter is implemented and configured

### Documentation
- [ ] README exists
- [ ] Instructions for how to install, run, build and test the project are documented.

### Logging
- [ ] Docker image captures logs to stdout/stderr.
- [ ] A logging library with different log levels is used instead of normal prints.

### Slotting into our Infrastructure
- [ ] Has a Dockerfile and an AWS ECR docker repository to push to.
- [ ] Has Nomad job definition files.
- [ ] Has a CircleCI configuration with working builds.
- [ ] Has Sentry set up for Error Logging
- [ ] Has a `/health` healthcheck route.
- [ ] Has Statuscake healthcheck set up.
- [ ] Has Consul healthcheck set up.

### Does it work?
- [ ] Yes :)

I originally wrote this rough cheatsheet for documentation at work, where I had to deal with different linux flavours' ways of configuring services / daemons.

These commands may be helpful when you are managing linux servers. Different versions of Ubuntu etc use different types of daemon/service managers.

Upstart

Service configurations are located in /etc/init/.

Enable a service on startup. There are two ways depending on OS.

chkconfig --add <service>

or

update-rc.d <service> default

You can then interact with the service using either initctl or service, again depending on OS. Sometimes both work, sometimes both work but show different services. Fun.

Initctl cheatsheet

Show services:

initctl list

Stop a service:

initctl stop <service>

Logs can sometimes be found in /var/log/upstart/<service>

Systemd

Simply put, service configurations are located in /etc/systemd/system/.

Enable a service on startup:

systemctl enable <service>

Interact with a given service.

systemctl start|status|restart|stop <service>

Although service probably also works.

Logs can usually be access via journalctl

journalctl -u <service>

I'm developing a small app in my free time using nodejs and express, while trying out Auth0 as a way to outsource the authentication layer nicely. One problem I face though is as I want to test out my API, which I now typically do via Insomnia (great tool btw), I couldn't get a long lasting token to test the with.

So I figured out a quick dirty trick to fix this. Jumping through the calls through different node packages for JWT (express-jwt -> jsonwebtoken); I found a clockTimeStamp option.

clockTimestamp: the time in seconds that should be used as the current time for all necessary comparisons.

Simply set the clockTimeStamp value to some small non-zero integer. This will fake the time for the expiry check to think that it's sometime at the beginning of probably unix time or something.

Some example code / barebones app:

const express = require("express");
const jwt = require("express-jwt");
const jwks = require("jwks-rsa");
require("dotenv").config();

const app = express();

const jwtOptions = {
  secret: jwks.expressJwtSecret({
    cache: true,
    rateLimit: true,
    jwksRequestsPerMinute: 5,
    jwksUri: "https://some-domain.auth0.com/.well-known/jwks.json"
  }),
  audience: "http://localhost:3000/api/",
  issuer: "https://some-domain.auth0.com/",
  algorithms: ["RS256"]
};

// This effectively disables the expiry check
if (process.env["DISABLE_JWT_EXPIRY"]) {
  console.log("WARNING: set clockTimeStamp to 0");
  jwtOptions.clockTimestamp = 1;
}

let jwtCheck = jwt(jwtOptions);
app.use(express.json());
app.use(jwtCheck);

app.get("/userinfo", (req, res) => {
  res.json(req.user);
});

console.log("Starting on port 4000");
app.listen(4000);

Running it locally now with the DISABLE_JWT_EXPIRY environment variable set I can skip having to get a fresh token. Another alternative would ofc be to just use a different JWT setup for local development, but eh... too lazy :)

Of course, don't use this anywhere outside local development.

Here is the Dockerfile for a small golang webservice I wrote, which I managed to make quite small. Saving it for myself and I guess I may as well share it since I think it will otherwise be thrown away.

Note I've replaced the more project-specific stuff. Like the project name and package. Probably there is some smarter way to compile it with the correct path.

# The file has two steps, first the builder then the actual running image.
# This part identifies it as the builder.
FROM golang:1.10 as builder

# Create appuser to avoid running as root later
RUN adduser --system appuser

# Import the project onto where it would go on the gopath
WORKDIR /go/src/github.com/<username>/<project>
COPY . .

# Fetch dependencies
RUN go get -d -v ./...
# Compile the binary. Mind the flags because it has to work in the next image.
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo .

# Here we start the actual image. Scratch is super barebones.
FROM scratch

WORKDIR /

# Copy over the files we need.
# The ssl certs are needed for doing any kind of ssl connection
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
# The actual binary.
COPY --from=builder /go/src/github.com/<username>/<project>/<project> /<project>
COPY --from=builder /etc/passwd /etc/passwd

# Use the non-root user.
USER appuser

# The tcp port to expose, in my case 1323 which is the default of the framework I use.
EXPOSE 1323

# Start the binary as the entrypoint.
ENTRYPOINT ["/<project>"]

When building this Dockerfile for my project, I get an image that is only 13.1MB big. Pretty good, compared to other services in e.g. NodeJS or Python that I have made before that usually end up 300MB+.

I wrote some tools to help with migrating an existing imap account from one server to another, so that you can keep your emails when you switch your domain name to a new provider. I think normally your email client might do this for you automatically just by you changing the server settings, but I have some users only using a webmail client, meaning they don't have a local copy to sync to the new server. Plus it's always good to have a backup.

To solve this I wrote some scripts that let me download an entire mailbox and then reupload it to a new server. You can find them here: https://github.com/Tethik/imap-tools

I initially intended to use these for migrating a domain name and the email to a new provider. Unfortunately I never got to carry out the migration, so I only got to try this out with some test accounts. The way I imagine the whole process of moving email servers with a domain name looks something like this:

1. Download all accounts and their emails locally using my script.
2. Do the domain transfer.
3. Set up the new imap accounts at the new provider.
4. Use the upload script for each account to upload all old emails to the new accounts.

At work we use git flow to organize our git repositories. master is the production branch, develop is the staging branch, and longer features get their own feature/* branch. On these branches, everything is built into docker images that are uploaded to a registry to later be deployed. git flow also gives us some basic version tagging that we want to use to track our releases.

I wanted to tag these docker images so that we could easier refer to them in our deployment. Where there would be a latest tag that would always be the latest version, and maybe a tag for every version.

I came up with a python script that wraps around docker and git to automatically generate these tags for me, with some basic customization. It's pretty rough, but you can find it here: https://github.com/Tethik/lame-cli-programs/tree/master/docker-branch-tagging

Usage

docker-branch-tagging init generates a default .docker-branch-tagging file that looks something like the following.

{
    "develop": ["latest","develop-{CIRCLE_BUILD_NUM}","{git_branch}"],
    "feature/(.+)": ["{git_branch}"],
    "master": ["master","master-{CIRCLE_BUILD_NUM}","{git_latest_version_tag}"]
}

The keys are regex, and the values are python format strings. The values get passed the current environment variables as well as two special case variables git_branch and git_latest_version_tag. The script will simply look for any keys matching the current git branch and perform the templating on the values to generate the different tags.

Doing a docker-branch-tagging build aws-blahabhla.com/example on the master branch would then result in something like the following.

docker build -t aws-blahabhla.com/example:master -t aws-blahabhla.com/example:master-123 -t aws-blahabhla.com/example:0.2.1 .

docker-branch-tagging push would then perform the docker push. Ungefär like so:

docker push aws-blahabhla.com/example:master 
docker push aws-blahabhla.com/example:master-123
docker push aws-blahabhla.com/example:0.2.1

On CircleCI, which is where we do our continous integration, the step for building and pushing the docker containers generally looks something like this now.

  build:
    docker:
      - image: circleci/python3
    steps:
      - run: sudo pip install awscli
      - run: sudo pip install "git+https://github.com/Tethik/lame-cli-programs#egg=docker_branch_tagging&subdirectory=docker-branch-tagging"
      - checkout
      - attach_workspace:
          at: .
      - setup_remote_docker
      - run: "docker login -u AWS -p $(aws ecr get-authorization-token --output text --query authorizationData[].authorizationToken | base64 --decode | cut -d: -f2) $DOCKER_REPOSITORY"
      - run: docker-branch-tagging build $DOCKER_REPOSITORY
      - run: docker-branch-tagging push $DOCKER_REPOSITORY

The attach_workspace is used to copy over whatever dependencies that may have been installed via e.g. npm or binaries/webpacks that may have been built in a previous workflow step. DOCKER_REPOSITORY is the environment variable I set in the project configuration to the AWS ECR uri.

Just for fun, I decided to try making a CI/CD pipeline for my CV. This post will be a bit rough and jumbled, sorry about that, probably would have been better to split this into several posts. Anyhow, here's how I did it.

Building the LateX file via Docker

Before I used to just update my CV via ShareLatex, so I didn't have any Latex packages installed on my system. Installing Latex is usually a confusing mess of packages, so I figured using docker might be a good fit. In addition I knew that CircleCI takes a docker image to launch it's jobs in, so I could reuse it later.

Luckily I found that someone else had made a docker image for xelatex. With some adjustments I made my own image that contained everything I needed for compiling my tex files.

Then I uploaded it to docker hub for free (since it's open source).

Setting up CircleCI

With the previously mentioned docker image I defined a CircleCI job step as follows.

  build_pdf:
    docker:
      - image: tethik/xelatex:latest
    steps:
      - checkout
      - run: make 
      - attach_workspace:
          at: .  
      - persist_to_workspace:
          root: .
          paths:
            - cv.pdf

The persist_to_workspace step saves the pdf so that it can be reused later in the deploy job using the attach_workspace step. The current attach_workspace step in the config above coming before the persist_to_workspace is to collect some scripted parts of the CV that I try to generate automatically. More on this later.

Deploying to Github Pages

For CD (Continuous Delivery) part of the process I needed somewhere to publish the document. I do have my own domains and servers, but I'd rather not have to give access to CircleCI to ssh or ftp into these. The hacky and cheap solution was to just reuse the same github repository and enable the Github pages feature. To do this I enabled Github Pages on the master branch, which I kept empty except for the final pdf output. Then I set up another step in the CircleCI config to commit and push the new pdf file to the master branch.

The CircleCI config step looks like this.

  deploy:
    docker:
      - image: circleci/node:8.9
    steps:
      - checkout
      - attach_workspace:
          at: .      
      - run: .circleci/deploy.sh

The deploy script itself looks like this.

#!/bin/sh

git config --global user.email "circleci@blacknode.se"
git config --global user.name "CircleCI Deployment"
mv cv.pdf ..
git fetch --all
git reset --hard origin/develop
git checkout master
mv ../cv.pdf .
git add cv.pdf
git commit -m "PDF build $CIRCLE_SHA1"
git push origin master

By default CircleCI generates a key on your github repo which only has READ access. To get around this I created a new ssh key and added it as a deploy key to the github repository. Then I removed the original key from the CircleCI project configuration, and added the new ssh key that I generated.

The final step was to add the redirect from my main homepage (this site) to the github pages link where the document would be hosted. Since the server is running Apache I could use the following config.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteRule ^(.)cv$ https://tethik.github.io/curriculum-vitae/cv.pdf
RewriteRule ^curriculum-vitae https://tethik.github.io/curriculum-vitae/cv.pdf
</IfModule>

It's a bit hacky, but it works :)

Automatically updating the document

The next fun step I wanted to do was add some parts to the document that would be automatically generated. Because writing CVs is boring and too manual. For now though I just coded something simple: A script that summarizes all my pull requests made on Github that are in some sense Open Source. i.e. not to repos that I own myself or repos that were created for e.g. schoolwork.

To organize the LateX files I set up the repo as follows. I wanted to keep the generated files separate from the main latex file that I copied from before.

partials/ ->  generated tex files.
src/github/ -> python script that generates into partials/
cv.tex

Inside the main cv.tex file I could then refer to the scripted content using the subfile package.

\subsection{Github Open Source Contributions}
\subfile{partials/pull_requests}

Inside the src/ folder I mean to keep scripts that generate the partials. The src/github/ folder contains a script that generates a pull_requests.tex into the partials/ folder. It talks the to Github GraphQL API and summarizes the info into a table.

I added the following CircleCI job config. This goes before the previously defined build_pdf step.

  build_partials:
    docker:
      - image: kennethreitz/pipenv
    steps:
      - checkout      
      - run: 
          command: pipenv install 
          working_directory: ~/project/src/github/
      - run: 
          command: pipenv run make
          working_directory: ~/project/src/github/
      - persist_to_workspace:
          root: .
          paths:
            - partials/

Again I persist_to_workspace to keep the resulting partial/*.tex files. Pipenv handles the python dependencies beautifully. The github api key and a blacklist of repos to ignore is passed through environment variables.

This is what the table looks like in the PDF.

In the future I'd like to add more content that's automatically generated. E.g. pypi and npm packages published, total github commit stats etc.

Final Result

You can find the repository here, the latest pdf built by the pipline here, and the CircleCI project here.

I still need to update my CV though.

Over the weekend I wrote a small tool that will automatically update the /etc/hosts file with your running docker container. You can find the script here: https://github.com/Spielstunde/docker-hosts-update

Example usage

First create a docker network. This will allow the containers to connect to each other and automatically resolve hostnames. This is most of the magic tbh.

docker network create evilcorp.internal

Now you can start new containers in the network:

docker run --network evilcorp.internal --rm -it nginx
docker run --network evilcorp.internal --name hello --rm -it nginx

After starting the above two container you can then start ...

sudo docker-hosts-update

... and you should see a section like this in your /etc/hosts file.

# ! docker-hosts-update start !
# This section was automatically generated by docker-hosts-update
# Don't edit this part manually :)
172.20.0.2   hello.evilcorp.internal
172.20.0.3   friendly_golick.evilcorp.internal
# ! docker-hosts-update end   !

Because these networks are on the same bridged network, docker will first of all ensure that their hostnames resolve to each other. E.g. in the first container it will be able to connect to the "hello" container, either via hello.evilcorp.internal or just hello.

What my script does is easily enable the host machine to resolve the hostnames too. This saves you the trouble of either getting the ip manually (because hostname, duh) but also removes the need for other approaches where you need to assign different ports to different containers, and then use some sort of service discovery tool and proxy to manage them. On both host and container the url http://hello.evilcorp.internal will resolve to the correct container.

Finally when you stop the containers using Ctrl+C or docker stop hello, you'll see the lines automatically removed from the hosts file, if docker-hosts-update is still running.