Sign in

Creating and automating Concourse CI pipelines

Concourse CI is a great CI/CD platform for self-hosting your different development pipelines but adding and maintaining them can be a real pain. Our goal today is to ease this process.

Photo by Simon Fitall on Unsplash

How do we set pipelines in Concourse CI?

Concourse requires us to use the fly CLI tool to configure pipelines to the Concourse server by using .yml configuration files.

We need to do this every time we want to create or change a pipeline, there needs to be a better way.

Automating your pipelines

Creating a self-sustaining pipeline

Concourse pipelines are triggered by checking the Git repository for changes. We can take this to our advantage, by creating a self-sustaining pipeline. To remain using the same airplane terminology as Concourse, we name our new pipeline fleetmanager.


Concourse representation of the pipeline fleetmanager.yml

The pipeline checks the Git repository containing its own configuration for changes. The first job name set-self will apply the new configuration to itself (the pipeline fleetmanager). When this is successful, the next job configure-pipelines will be executed.

Centralize your pipline configuration files

Most of the CI/CD tools require to have the pipeline configuration file to remain in the same repository as the code that needs to be processed by this pipeline. Concourse cannot use this method because it is not directly integrated in the Git repository.

We can however keep the pipeline configuration file from different projects in the same Git repository.

Both fleetmanager pipeline jobs use the to execute the necessary commands to create and update the Concourse pipelines. In our example we have made a folder CI which contains the configuration files of the pipelines we want to manage.

Note that the same directory contains a folder vars with underneath the configuration file vars.yml containing all variables needed by the pipelines in the CI folder.

The script has two options:

  • set-fleetmanager (creates/updates the self-sustaining managing pipeline)
  • set-fleet (creates/updates the pipeline(s) located in the CI directory)

Both options work the same way but set-fleetmanager looks for the fleetmanager.yml file, while set-fleet iterates over the different pipeline configuration files in the CI directory.

Please note that the script unpauses any just created pipelines, so they can be directly set in motion.

It is obvious that this script needs to be executed one time before the self-sustaining pipeline is initiated. This can be done by running the script on your local machine.

Now you should be able to directly edit the configuration files, pushing it to the Git repository and seeing that your pipeline will be automatically added or updated.