A whale, a bird and two colors enter a bar

DateTime.AfterPost - DateTime.Now = 5 minutes

Well, it’s not a joke. Sorry if you are disappointed. Today I want to talk about blue & green deployment using docker-compose and canary strategy

Description

  • This paper will introduce the idea of deploying a new version of a microservice system that runs from docker-compose (without using K8S or alike)
  • This could be useful for testing / Stand-Alone servers / companies that don’t want or have the tech-strength to manage a cluster.
  • The outcome of this architecture is a stable system that can be upgraded and rolled back without concerns.
Blue Green
Blue green canary release with a load balancer

Blue / Green deployment

Many articles have been written about blue/green deployment and I won’t be writing another one. I do wish to write a few sentences about it so that we are speaking the same language.

Blue / Green deployment is a type of deployment that allows upgrading systems with minimal down-time and maximum confidence.

To do this, the method does as follows:

Let’s assume blue is up

  1. We turn on green – this is done without ANY conflicts to blue
  2. We transition our clients to work with green.
    This can be done all at once or canary-wise.
  3. After verifying green works and all clients now work with green, we take blue down.
  4. Repeat the procedure whilst having green up (and now old) and wanting to make blue the new and stable.

Many services provide this “as a service” (For example, AWS or K8S)

You’ll probably prefer using “as a service” solutions but sometimes we have environments where it is impossible. We are not always able to use AWS (or other cloud providers) as we want to be offline and K8S might be “too much” for our needs.

Having said that, we still want a manageable environment and upgrades.

Let’s get to it

Assumptions

Before we start, we need to understand we are speaking the same language.

  • There won’t be more than two versions in the air at any time. We don’t want / see the need to manage more than a “stable” version and a “new” version.
  • We are currently in the “blue” version – This will be helpful terminology-speaking.
  • Each version has a unique VersionId and they are incremental.
  • The system can be brought up using “docker-compose up -d” and brought down using “docker-compose down”.
  • Your services communicate with each other via IP:PORT or RabbitMq (or alike)

Avoiding collision

  • The port to be used for a service that exposes its interface will be an Environment Variable of the form “<ServiceName>_PORT”
    • For example, let’s say I have a service that exposes a service via HTTP API.
    • In normal situations, it would expose its service using a constant port (8080 for example)
    • We now change this so that it exposes its service using port “${MyApi_PORT}”
    • All services that communicate with that service, will change their configuration from “8080” to “${MyApi_PORT}”
  • Service container’s name will be an Environment Variable of the form
    “<ServiceName>_${VERSION_COLOR}”
  • If the services use a host path for working, the path will look as follows: “/opt/${VERSION_COLOR}/…

Release Pipeline

  • The release pipeline will receive a variable telling the color of the new version to be released.
  • Using that color it will set the following variables:
    • MyApi_PORT=${MyApi_<color>}
      • MyApi_blue and MyApi_green are constant values
    • VERSION_COLOR=${color}

Root directory

  • /blue/
    • <BlueVersionID>
    • docker-compose.yml

Now we want to upgrade our system from the blue version to the green version.

We’ll run the release pipeline with “green” as input.

Side note

I think the best thing to do here, but as a “bonus”, is to create a simple service that will manage which version should be released now.

It will do the following:

  • GET the current stable version
  • GET the current “new” version (or null if none)
  • SET the current stable version
  • SET the current “new” version
    • Won’t let you set a new version if there are still clients on the stable version – meaning not all clients have moved to the new version.
    • We do not support more than 1 “new” version.
    • This service can be automatically called from the pipeline.

The deployment will create a new directory

  • /green/
    • <GreenVersionID> (for debugging purposes)
    • docker-compose.yml

We can safely start our system now

docker-compose -f /green/docker-compose.yml up -d

and we’ll see that both our versions are up and running.

Canary release

We now have two versions up and running and we want to move the clients from the old one (blue) to the new one (green).

Moving them all at once is an option and will cut down the time it takes to release a new version. Even so, moving them all at once puts us at risk of stumbling upon a bug and then we’ll have to move them all back. Although this can be “fast”, it still means our clients will experience the bug / or some downtime.

A better way, in my opinion, will be to “canary” deploy the version. By this, the upgrade will not be a 0% or 100% rollover but a 0% to 100%.
As before, many services will give you this option “as a service” (AWS Route53 for example) but we might not have that “luxury”.

For manual canary, you could do some porting of your own according to the IP of the client. If you have some “workers” you manage, you could configure some of them to work with the new version. Another idea is to use the routing-key feature of RabbitMq to decide which version to send the requests to, and more.. I think you got the idea.

First of all, in the canary scenario, I could run my E2E / Integration tests on the new version (green) in the production environment (as crazy as it sounds, this is actually manageable and quite easy). Of course I did this before in the testing environment, but nothing is more accurate than the real environment.

Then, I could turn into doing tests in the real world without worrying about paying customers. I could use some test clients or move directly to the alpha-testers. There is a huge advantage of doing the final tests in the production environment, and if you can do it without worries, better yet.

After all is well, I’ll move to the beta-testers / the next large portion. This could also be made geographically – Maybe I’ll want to roll the upgrade to the parts of the world that are currently awake to get feedback as fast as possible or to the parts where there is less activity if I’m worried this version will break stuff.

Finally and inevitably, all the customers will receive the green version and it is time to take down the blue version. Blue version can stay up for as much time as you think is necessary in case a rollback is required. This could be a few hours to a few days, but I don’t recommend any more than that.

Now that the green version is stable, it’s time to take down the blue version to prepare the environment for a new blue version. This time, blue will be the new version whilst green is the stable version.

In the end, with a bit more effort, we are receiving a more stable environment which will introduce more confidence to make changes. Stability in the form of a paced upgrade and a quick rollback. Although it sounds like a lot of work, it can be mostly automated with “canaries” in different places to trigger a rollback if needed.

Comments: 1

  1. Someone who knows says:

    sounds familiar 🙂

Add your comment