Sign in

Creating and automating Concourse CI pipelines

Concourse CI is a great CI/CD platform for self-hosting your different development pipelines but adding and maintaining them can be a real pain. Our goal today is to ease this process.

Photo by Simon Fitall on Unsplash

How do we set pipelines in Concourse CI?

Concourse requires us to use the fly CLI tool to configure pipelines to the Concourse server by using .yml configuration files.

We need to do this every time we want to create or change a pipeline, there needs to be a better way.

Automating your pipelines

Creating a self-sustaining pipeline

Concourse pipelines are triggered by checking the Git repository for changes. We can take this to our advantage, by creating a self-sustaining pipeline. To remain using the same airplane terminology as Concourse, we name our new pipeline fleetmanager.

fleetmanger.yml

Concourse representation of the pipeline fleetmanager.yml
jobs:
- name: set-self
public: true
plan:
- get: councourse-ci
trigger: true
- task: init
config:
platform: linux
image_resource:
type: registry-image
source: {repository: ubuntu}
inputs:
- name: councourse-ci
run:
path: /bin/sh
args:
- -cx
- |
cd councourse-ci/
chmod +x setup_pipelines.sh
./setup_pipelines.sh set-fleetmanager
- name: configure-pipelines
public: true
plan:
- get: councourse-ci
trigger: true
passed: [set-self]
- task: init
config:
platform: linux
image_resource:
type: registry-image
source: {repository: ubuntu}
inputs:
- name: councourse-ci
run:
path: /bin/sh
args:
- -cx
- |
cd councourse-ci/
chmod +x setup_pipelines.sh
./setup_pipelines.sh set-fleet

The pipeline checks the Git repository containing its own configuration for changes. The first job name set-self will apply the new configuration to itself (the pipeline fleetmanager). When this is successful, the next job configure-pipelines will be executed.

Centralize your pipline configuration files

Most of the CI/CD tools require to have the pipeline configuration file to remain in the same repository as the code that needs to be processed by this pipeline. Concourse cannot use this method because it is not directly integrated in the Git repository.

We can however keep the pipeline configuration file from different projects in the same Git repository.

Both fleetmanager pipeline jobs use the setup_pipelines.sh to execute the necessary commands to create and update the Concourse pipelines. In our example we have made a folder CI which contains the configuration files of the pipelines we want to manage.

Note that the same directory contains a folder vars with underneath the configuration file vars.yml containing all variables needed by the pipelines in the CI folder.

The setup_pipelines.sh script has two options:

  • set-fleetmanager (creates/updates the self-sustaining managing pipeline)
  • set-fleet (creates/updates the pipeline(s) located in the CI directory)

Both options work the same way but set-fleetmanager looks for the fleetmanager.yml file, while set-fleet iterates over the different pipeline configuration files in the CI directory.

Please note that the script unpauses any just created pipelines, so they can be directly set in motion.

setup_pipelines.sh

PROGRAM="fly"
PROGRAM_URL="https://github.com/concourse/concourse/releases/download/v7.0.0/fly-7.0.0-linux-amd64.tgz"
CONCOURSE_HOST="https://ci.example.com"
USER="user"
PASSWORD="pass"
RED='\033[0;31m'
NC='\033[0m' # No Color
BLINK='\e[5m'
echo "Initializing program"check_enviroment () {
echo "- Checking permissions"
if [ "$EUID" -ne 0 ]; then
printf "${RED}Please run as Administrator${NC}\n";
exit 1
fi
}check_program () {
./$PROGRAM &> /dev/null
if [ "$?" = "0" ];
then
echo " -> $PROGRAM" found
else
echo " -> $PROGRAM" not found. Downloading...
curl -L -O -J $PROGRAM_URL &> /dev/null
tar -xzvf fly-7.0.0-linux-amd64.tgz &> /dev/null
if [ "$?" = "0" ];
then echo " -> $PROGRAM" installed successfully.
fi
fi
check_success $? &> /dev/null
}
check_curl() {
which curl &> /dev/null
if [ "$?" != "0" ];
then
apt-get update
apt-get -y install curl
fi
}
check_dependencies (){
echo "- Checking dependencies"
check_curl &> /dev/null
check_program
}
concourse_login () {
echo "- Authenticating with $CONCOURSE_HOST"
./fly --target main login --concourse-url $CONCOURSE_HOST -u $USER -p $PASSWORD &> /dev/null
echo "- Syncing CLI version"
./fly -t main sync &> /dev/null
}
set_pipelines () {
echo "- Setting up pipeline(s)"
FILES=$(pwd)/CI/*.yml
dir="$(pwd)/CI"
for f in $FILES
do
set_pipeline $f $dir
done
}
set_pipeline_maintainer () {
echo "- Setting up pipeline(s)"
f="fleetmanager.yml"
dir="$(pwd)"
set_pipeline $f $dir
}
set_pipeline () {
echo " -> Processing $1 pipeline..."
filename=$(basename -- "$1")
filename="${filename%.*}"
./fly -t main set-pipeline --pipeline $filename --config $1 --load-vars-from $(pwd)/CI/vars/vars.yml -n &> /dev/null
check_success $? &> /dev/null
./fly -t main unpause-pipeline -p $filename &> /dev/null
check_success $?
}
check_success () {
if [ "$1" = "0" ];
then
echo " -> Done"
else
echo "ERROR: Check configuration of $f"
exit 1
fi
}
check_enviroment
check_dependencies
concourse_login
case "$1" in
#State
set-fleetmanager)
set_pipeline_maintainer ;;
#Options
set-fleet)
set_pipelines ;;
*)
echo "unknown option"
;;
esac
printf "${BLINK}End of program\n"

It is obvious that this script needs to be executed one time before the self-sustaining pipeline is initiated. This can be done by running the script on your local machine.

Now you should be able to directly edit the configuration files, pushing it to the Git repository and seeing that your pipeline will be automatically added or updated.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store