Creating and automating Concourse CI pipelines

Concourse CI is a great CI/CD platform for self-hosting your different development pipelines but adding and maintaining them can be a real pain. Our goal today is to ease this process.

Photo by Simon Fitall on Unsplash

How do we set pipelines in Concourse CI?

Concourse requires us to use the fly CLI tool to configure pipelines to the Concourse server by using .yml configuration files.

fly -t ci set-pipeline -p booklit -c pipeline.yml

We need to do this every time we want to create or change a pipeline, there needs to be a better way.

Automating your pipelines

Creating a self-sustaining pipeline

Concourse pipelines are triggered by checking the Git repository for changes. We can take this to our advantage, by creating a self-sustaining pipeline. To remain using the same airplane terminology as Concourse, we name our new pipeline fleetmanager.

fleetmanger.yml

Concourse representation of the pipeline fleetmanager.yml

---
resources:
- name: councourse-ci
  type: git
  icon: gitlab
  source:
    uri: https://gitlab.com/user/concourse-ci.git
jobs:
- name: set-self
  public: true
  plan:
    - get: councourse-ci
      trigger: true
    - task: init
      config:
        platform: linux
        image_resource:
          type: registry-image
          source: {repository: ubuntu}
        inputs:
          - name: councourse-ci
        run:
          path: /bin/sh
          args:
            - -cx
            - |
              cd councourse-ci/
              chmod +x setup_pipelines.sh
              ./setup_pipelines.sh set-fleetmanager
- name: configure-pipelines
  public: true
  plan:
    - get: councourse-ci
      trigger: true
      passed: [set-self]
    - task: init
      config:
        platform: linux
        image_resource:
          type: registry-image
          source: {repository: ubuntu}
        inputs:
          - name: councourse-ci
        run:
          path: /bin/sh
          args:
            - -cx
            - |
              cd councourse-ci/
              chmod +x setup_pipelines.sh
              ./setup_pipelines.sh set-fleet

The pipeline checks the Git repository containing its own configuration for changes. The first job name set-self will apply the new configuration to itself (the pipeline fleetmanager). When this is successful, the next job configure-pipelines will be executed.

Centralize your pipline configuration files

Most of the CI/CD tools require to have the pipeline configuration file to remain in the same repository as the code that needs to be processed by this pipeline. Concourse cannot use this method because it is not directly integrated in the Git repository.

We can however keep the pipeline configuration file from different projects in the same Git repository.

Both fleetmanager pipeline jobs use the setup_pipelines.sh to execute the necessary commands to create and update the Concourse pipelines. In our example we have made a folder CI which contains the configuration files of the pipelines we want to manage.

concourse-ci
    - fleetmanager.yml
    - setup_pipelines.sh
    - CI
      - backend.yml
      - frontend.yml
      - deployment.yml     
      - build.yml 
      - vars
        - vars.yml

Note that the same directory contains a folder vars with underneath the configuration file vars.yml containing all variables needed by the pipelines in the CI folder.

The setup_pipelines.sh script has two options:

• set-fleetmanager (creates/updates the self-sustaining managing pipeline)
• set-fleet (creates/updates the pipeline(s) located in the CI directory)

Both options work the same way but set-fleetmanager looks for the fleetmanager.yml file, while set-fleet iterates over the different pipeline configuration files in the CI directory.

Please note that the script unpauses any just created pipelines, so they can be directly set in motion.

setup_pipelines.sh

#!/bin/bash
# Description: CI setup script
# Author: TroublingFox125
# Version 1.0
PROGRAM="fly"
PROGRAM_URL="https://github.com/concourse/concourse/releases/download/v7.0.0/fly-7.0.0-linux-amd64.tgz"
CONCOURSE_HOST="https://ci.example.com"
USER="user"
PASSWORD="pass"
RED='\033[0;31m'
NC='\033[0m' # No Color
BLINK='\e[5m'
echo "Initializing program"
check_enviroment () {
echo "- Checking permissions"
if [ "$EUID" -ne 0 ]; then
    printf "${RED}Please run as Administrator${NC}\n";
    exit 1
fi
}
check_program () {
  ./$PROGRAM  &> /dev/null
  if [ "$?" = "0" ];
  then
  echo "  -> $PROGRAM" found
  else
      echo "  -> $PROGRAM" not found. Downloading...
      curl -L -O -J $PROGRAM_URL &> /dev/null
      tar -xzvf fly-7.0.0-linux-amd64.tgz &> /dev/null
      if [ "$?" = "0" ];
      then echo "  -> $PROGRAM" installed successfully.
      fi
  fi
  check_success $? &> /dev/null
}
check_curl() {
  which curl &> /dev/null
  if [ "$?" != "0" ];
  then
    apt-get update
    apt-get -y install curl
  fi
}
check_dependencies (){
   echo "- Checking dependencies"
   check_curl &> /dev/null
   check_program
}
concourse_login () {
  echo "- Authenticating with $CONCOURSE_HOST"
  ./fly --target main login --concourse-url $CONCOURSE_HOST -u $USER -p $PASSWORD &> /dev/null
  echo "- Syncing CLI version"
  ./fly -t main sync &> /dev/null
}

set_pipelines () {
  echo "- Setting up pipeline(s)"
  FILES=$(pwd)/CI/*.yml
  dir="$(pwd)/CI"
  for f in $FILES
  do
    set_pipeline $f $dir
  done
}
set_pipeline_maintainer () {
    echo "- Setting up pipeline(s)"
    f="fleetmanager.yml"
    dir="$(pwd)"
    set_pipeline $f $dir
}
set_pipeline () {
    echo "  -> Processing $1 pipeline..."
    filename=$(basename -- "$1")
    filename="${filename%.*}"
    ./fly -t main set-pipeline --pipeline $filename --config $1 --load-vars-from $(pwd)/CI/vars/vars.yml -n &> /dev/null
    check_success $? &> /dev/null
    ./fly -t main unpause-pipeline -p $filename &> /dev/null
    check_success $?
}
check_success () {
    if [ "$1" = "0" ];
    then
    echo "  -> Done"
    else
    echo "ERROR: Check configuration of $f"
    exit 1
    fi
}
check_enviroment
check_dependencies
concourse_login
case "$1" in
  #State
  set-fleetmanager) 
  set_pipeline_maintainer ;;
  #Options
  set-fleet)
  set_pipelines ;;
  *)
  echo "unknown option"
  ;;
esac
printf "${BLINK}End of program\n"

It is obvious that this script needs to be executed one time before the self-sustaining pipeline is initiated. This can be done by running the script on your local machine.

sudo ./setup_pipelines.sh set-fleetmanager

Now you should be able to directly edit the configuration files, pushing it to the Git repository and seeing that your pipeline will be automatically added or updated.