Continuous integration: definition

Anyone who works in software development knows the problems associated with integration. New code is written, the actual task is finally done! Now, the source code just needs to be inserted into the overall project – and then the problems start appearing. Many teams opt for continuous integration to ensure that a disaster does not occur at the end of a long development phase because nothing fits together.

Continuous integration is – just like continuous delivery – very popular, especially when it comes to agile software development. The aim of this modern approach is to work in small steps in order to make the development process itself more effective and to be able to react more flexibly to changes. Continuous integration was mentioned for the first time in the description of the agile method “Extreme Programming” by Kent Beck.

Continuous integration creates a solution for a situation like the following: a company is working on a large project for which the customer should receive comprehensive, complex software. Individual teams develop different sections of the application and the developers program individual functions. After months of work, perhaps years, everything has to be put together – and then problems arise. In this situation, it can take months until the bugs are detected and fixed, and all code snippets are put together. All the while, final testing and deployment is rapidly approaching.

With continuous integration, integrating new code takes place much earlier, not when all participants have completed their sections. Instead, developers add their finished code to the mainline once or several times a day – the source code is then open to all programmers. Since these are always relatively small sections of code, the integration is also rather short. A developer should only need a few minutes to make their work available to the rest of the team. If an error is discovered, it can be detected immediately, and in the best case, quickly corrected.

In practice, continuous integration usually goes like this: A developer has the task of creating a function. Before they start programming, they download the current version of the application’s complete code, the so-called mainline. They can now work on this version, which is on their personal computer (also called a working copy). Once they have completed their task, they test the program, remove any errors, and can now put the new version back on the mainline. 

However, the developer is not working on the program alone. Whilst they are making their changes, their colleagues have probably also made changes, so each developer on the team now has a different version on their workstation. The version on the mainline might have changed in the meantime, so the developer has to insert all changes into their working copy first. If an error occurs, it is up to them to correct it. Only then can the programmer add their changes to the mainline. Then they test the program again. Only if no changes occur – which can happen if they have not updated their working copy correctly – the process is actually completed and the developer can focus on the next task.

If you choose to use continuous integration, you have to follow some rules. Most programmers follow the principle that Martin Fowler (a software developer) created for successful continuous integration. First and foremost, they make sure that everyone involved is on the same level and that no one causes chaos by deviating from them.

It sounds natural, but it is one of the most important factors: all team members must use the same source (repository) when working on the code. This is not only true of the source code itself, since a working application requires additional elements, like databases. These too must be bundled together in one place. Martin Fowler therefore recommends building a repository so that a developer can work with a completely new worker computer and find all the required files in one central location.

To make a working program out of the source code, you have to compile it, update databases, and move files to the right place. This process can be automated. Ideally, it should be possible to run a build process with just one command.

A team benefits even more from automation and the acceleration of continuous integration if it integrates testing mechanisms into the build process. Just like the build process itself, the tests (best practice is to implement a whole set of different test mechanisms) must be executed with as little effort as possible.

Continuous integration can only work if all team members are supporting the system. As soon as one developer fails to permanently integrate their code into the mainline, other developers assume false premises. Everyone assumes that they are working on a stable system, but if someone withholds the code for too long (more than a day), the subsequent debugging can become a problem. As is so often the case in life, continuous integration communication is an important factor. If developers keep each other up to date, minor issues can be resolved quickly.

The program code found in the mainline should always be tested and be functional. Therefore, each developer may only integrate their code into this main branch and not work in their own branch. In this context, every developer should make sure that their own contribution definitely functions. To do this, they have to test the code and the build after integration. If a bug occurs, they must fix it. This ensures that the code is always error-free.

One of the most important things when it comes to continuous integration is that a faulty version must never remain on the mainline. For developers, this means that the repair must not be postponed. According to Martin Fowler, it isn’t a problem if builds don’t work and the code needs to be reworked, but the continuous integration system always requires the repair to be done immediately. This is because all developers must be able to assume that the code works in the mainline – if it doesn’t, the whole team builds in unstable systems, causing an avalanche of bugs.

The actual integration of the code (including testing) should take place as quickly as possible. Extreme programming (XP) provides just 10 minutes for this. Since a developer may integrate several times a day, long build times would lead to high working time losses. It is therefore necessary to establish mechanisms that accelerate the process. To enable these high speeds, you have to accept that you won’t be able to carry out every conceivable test directly. Instead, a two-stage system is implemented: in the first phase, which occurs in the daily work, tests are carried out in such a way that short build times can be achieved. The second test phase, on the other hand, lasts several hours and includes in-depth checks.

In principle, it makes sense to carry out tests in an outsourced and safe environment. However, you also need to ensure that it is configured exactly the same way as it is in the production environment. This can be very costly because the machine would have to be set up exactly the same way. However, virtualisation of complete computers means that this cost factor is becoming smaller and smaller.

All developers in a team should have easy access to the latest version and be able to run the program. This is relatively easy to do: since continuous integration requires all files to be stored in one repository anyway, developers just need to know the location. There are several good reasons for this: testers can start additional tests during the programming process, stakeholders can use executable files for demonstration purposes, and quality managers can check the number.

Not only is it important that everyone involved has access to the source code, it must also be clear to see what changes have been made and by whom. Communication also means that developers communicate with each other when they are in a build process. For this, some teams use separate displays or visuals that show that they are in the middle of an integration.

Finally, software distribution should also be automated. Files need to be moved from one environment to another, which can be time consuming. It makes sense to use scripts for this process. These will automate and accelerate the process.

Despite its positive characteristics, continuous integration is often also shown to have some disadvantages in everyday working life. Although it saves you a long and tedious integration phase at the end of the project and allows you to tackle problems early on, it can be very difficult for well-rehearsed teams to switch to continuous integration. In this case, the process can end up costing more time instead of saving time.

In principle, it is possible to manage continuous integration without the need for special tools, since all work steps can be performed manually. However, this requires a lot of discipline and even more effort. You can also make your work easier by using useful tools. These usually provide a server and help with building and version control.

• Jenkins: The very popular Java program is a form from Hudson that is no longer being further developed. The open source software works with various build tools and version control systems.
• Travis CI: This continuous integration tool is particularly popular because it works seamlessly with GitHub. The online repository informs Travis about changes to the code. The software is available in a free version for open source projects and in a paid version for all other projects.
• Bamboo: The Bamboo server enables developers to integrate, deploy, and manage releases. The manufacturer Atlassian offers the software as a web interface based on Java. Bamboo supports developers through automation and works with different build tools. For open source projects, the normally fee-based offer is available free of charge.
• Gitlab CI: GitLab offers its own program for continuous integration, which works with the well-known management version. The pipeline can be individually designed and therefore adapted to each project. GitLab also supports CI Docker.
• CircleCI: There are two different versions of this continuous integration program. Either you choose a variant directly in the provider’s cloud, or you create your own local server for use.
• CruiseControl: Originally developed by ThoughtWorks (a company Martin Fowler is involved in), CruiseControl is now an independent project. Amongst other things, CruiseControl offers developers a dashboard – their own website – on which build status can be seen.
• Codeship: Codeship tries to provide developers with a very simple way to continuous integration. On the basis of container technology, automatisms can easily be created. The company offers two different versions for this task: basic and pro.
• TeamCity: The commercial software TeamCity attaches great importance to interoperability with other tools. Many programs are already supported in the standard version, but the range can also be extended by plugins. The pre-tested commit is another special feature of theirs. TeamCity checks the rewritten code independently before integrating it into the mainline and informs the developer of any errors.