All Articles

A Microservices Guideline

I originally wrote this document for my work, where we were deep into a microservices architecture. The goal was to create a set of guidelines to be used when developing a new service. This way we could ensure some consistent quality for the code, as well as have it slot nicely into our then infrastructure.

We’re now moving away from that architecture, but I don’t want this document to go to waste. So I’m posting it here. Maybe someone else will find it useful as a template.

Minimum Requirements / Microservice Guidelines

This document details some requirements we would like to enforce for all of our microservices to ensure some level of quality. Use this document as a checklist to determine if your service is production ready. These requirements are not necessary set in stone, more a guideline.

In addition to these guidelines we strongly recommend following the 12 Factor App methodology.

Dependencies and Versioning

Dependencies locked to the correct versions and maintainable. A build must be reproducible in the future. Never lock to a third party’s version control branch, e.g. don’t use some github repo’s ‘dev-master’ in composer.

Semantic versioning should be used to categorize our builds.

If possible use a service like Evergreen or Greenkeeper to automatically keep your dependencies up to date.

For most projects, we should use Git-flow to organize our branches and keep a separate develop and master branch. It also helps us manage our releases. On github, develop should be the default branch.

Environment Variables and Secrets

Passwords, tokens, URLS to backing services and other variable parameters to your application must be passed as environment variables.

Each microservice must store it’s environment variables in Hashicorp Vault and never ever in git.

Secrets must never be stored in a git repository. This includes SSL certificates.

Testing and Code Quality

There should be regression tests for the important parts for the service. This helps to ensure that when we make code changes in the future the original functionality still works as intended, or that we are at least alerted to the fact that we broke existing functionality.

There should also be a way to measure the test coverage. This helps find parts of the code which could be better tested.

Use linting tools for your programming language and framework, this helps you follow standardized code conventions and hopefully keeps your code more concise and clean.

Documentation

Dev/Contributor documentation must exist that tell the next developer how to maintain the project. E.g. clear instructions on how to install dependencies, build, run, and test the project. It should be possible to pick up the project in a few months time. A good place to put these is in the README, which is also something that every project must have.

Logging

Make sure that your docker image logs to stdout. This makes troubleshooting one step easier as then we won’t have to shell directly into the container to view the logs and they will be picked up by filebeat.

Have different levels of logging depending on development/staging/production. Use a library that can handle different levels of logging, i.e.TRACE, DEBUG, INFO, WARN, ERROR are pretty typical log levels.

Slotting into our Infrastructure

Every microservice must have a Dockerfile. Likely it will also need a Docker repository to push builds to. For this we use AWS ECR. Typically this docker push step is done during our CircleCI build.

Each microservice must also have one or more Nomad job definition files per runtime environment. e.g. staging.json for staging, and production.json for production.

For continous integration we use CircleCI, as such the project must have a CircleCI configuration.

For error tracking we use Sentry, as such the project should send exceptions, errors and warnings to Sentry.

Each HTTP microservice should also have a simple healthcheck route, typically under http://service-name/health. This is then input to Statuscake and Consul, which periodically polls the service.

… but most importantly

The service must also work.

Checklist template

You can use the following markdown file to create a checklist via gist or github issue. This can be used for review before onboarding a new service.

### Dependencies and Versioning
- [ ] Dependencies locked and sane
- [ ] Semantic Versioning used to label builds.
- [ ] Automatically updated dependencies via Greenkeeper (or similar)
- [ ] Git flow configured with `develop` as the default branch on github.

### Environment Variables
- [ ] Passwords, tokens and other configuration are passed via environment variables.
- [ ] Environment variables stored in Vault.
- [ ] No secrets stored in the git repository.

### Testing and Code Quality
- [ ] Tests exist, run and pass
- [ ] Test coverage is measured
- [ ] Linter is implemented and configured

### Documentation
- [ ] README exists
- [ ] Instructions for how to install, run, build and test the project are documented.

### Logging
- [ ] Docker image captures logs to stdout/stderr.
- [ ] A logging library with different log levels is used instead of normal prints.

### Slotting into our Infrastructure
- [ ] Has a Dockerfile and an AWS ECR docker repository to push to.
- [ ] Has Nomad job definition files.
- [ ] Has a CircleCI configuration with working builds.
- [ ] Has Sentry set up for Error Logging
- [ ] Has a `/health` healthcheck route.
- [ ] Has Statuscake healthcheck set up.
- [ ] Has Consul healthcheck set up.

### Does it work?
- [ ] Yes :)

Published Nov 7, 2018

Security Engineer with a dash of software. Originally from Stockholm, now in Berlin. I like to hack things.