Blue-green Deployments and Canary Releases on Kubernetes

  

Peter Palaga    

@ppalaga       

      Marek Jelen

    @marek_jelen

 

Peter PalagaMarek Jelen

"Java guy"

JBoss Fuse, JBoss EAP

"Kubernetes & OpenShift guy"

Developer Advocate

Agenda

  • A Spring Boot app

  • Bring it to the cloud

  • Deployment strategies

A Spring Boot app

https://github.com/ppalaga/horse-ride-service

  • Model with a few Entities

  • CRUD endpoints provided by Spring Data

  • Usual @SpringBootTest

Minishift/Minikube

First steps with Minishift

# You may need more or less CPU and memory
# depending on what you do
minishift start --memory=8GB --cpus=4 --disk-size=50GB

oc login -u developer -p whatever

# Open the OpenShift web console in browser
minishift console

Jenkins Pipeline on OpenShift

# make sure you are in the default project
$ oc project
myproject
$ oc new-app https://github.com/ppalaga/horse-ride-service.git

  • If the git repo contains a Jenkinsfile

    • Os creates a BuildConfig with a special Pipeline strategy

    • Provisions a Jenkins instance

    • Triggers the first build

Preparations for a real pipeline

  • Git repo inside the cluster

  • Separate projects for CI, stage and production

  • Some permissions for Jenkins

  • Done by provision.sh script:

git clone https://github.com/ppalaga/horse-ride-service.git
cd horse-ride-service
git reset --hard provision.sh
oc login -u developer -p whatever
cd src/main/openshift; ./provision.sh

Pipeline

Build and test

Provision a database so that the @SpringBootTest can pass

cd horse-ride-service; git checkout master
git reset --hard testdb
git diff HEAD^ HEAD # study what changed
# Alias the cluster's internal git repo
URL=http://team:team@gogs-horse-ride-cicd.$(minishift ip).nip.io
git remote add minishift ${URL}/team/horse-ride-service.git
# Push to the cluster's internal git repo
git push -f minishift master
# Go to horse-ride-cicd project in OpenShift console
# and check that the pipeline started under Builds > Pipelines

Build and test Docker image

git reset --hard test-image
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Look at horse-ride-cicd > Builds > Pipelines
# Open the Jenkins log

Stage → approval → production

git reset --hard stage-prod
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Look at horse-ride-cicd > Builds > Pipelines
# Visit the stage and prod sites via links in Jenkins log
# Approve the promotion from stage to production

Blue/green deployments

blue green

  • Release in a predictable manner

  • Reduce downtime

    • Easy to switch to roll out a new version

    • Easy to roll back if the new version does not behave properly

https://martinfowler.com/bliki/BlueGreenDeployment.html

Blue/green demo (1/2)

git reset --hard blue-green
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Look at horse-ride-cicd > Builds > Pipelines
# The UI of new deployment should be blue

Blue/green demo (2/2)

git reset --hard blue-green-green-bg
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Look at horse-ride-cicd > Builds > Pipelines
*# The UI of new deployment should be green

Blue/green Caveats

  • Long running sessions/connections need to be handled gracefully.

    Migrating to Microservice Databases

  • Database schema conversions

    • Ideally back/forwards compatible - see Edson Yanagas’s book

    • Downtime may be needed in some cases

Canary releases

canary

  • Roll out gradually

  • Observe canary’s health

  • Rollback if the canary dies

https://martinfowler.com/bliki/CanaryRelease.html

Canary demo (1/2)

git reset --hard canary
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Observe the iterative checking of canary's health in Jenkins log
# This canary survives and its UI background is yellow

Canary demo (2/2)

git reset --hard canary-dead
git diff HEAD^ HEAD # study what changed
git push -f minishift master
# Observe the iterative checking of canary's health in Jenkins log
# This canary dies and its gray UI won't be served after
# the rollback

Wrap up

@ppalaga  ·  @marek_jelen