Software Development Automation
The Most Important Tool for a Successful Software Project
Speeding up developers, reducing downtime, and lowering project costs with CI/CD
Continuous integration & continuous delivery (CICD) is one of the most important aspects of a modern software development project. Without the confidence that a mature, modern CICD suite provides, developers are left blindly hoping that their code holds up as they march towards production.
Hope is not a strategy.
In a recent project, my team was tasked with a brand new, brown-field cloud uplift project - a fresh start for an existing system. We designed a rough, high-level design, chose our development language and tooling, and planned for the first couple of iterations of the system. The project would create a number of interconnected microservices which meant that we would have many different applications to manage, and the integration between the applications would be key.
It was clear to me that CICD pipelines would be absolutely critical to the success of the project. Without regularly checking that the codebase is tested, deployable, and healthy, we would be continually adding to a mountain of tech debt, integration problems, and bugs in business logic. The development team discussed and agreed to gating the acceptance of code into the codebase. For code to be accepted, it would have to pass two requirements:
1. New code must be approved by at least two people
2. The entire codebase must pass CI
Such a strict policy may provoke eye-rolling or sarcasm relating to the policy falling by the wayside at the first sign of trouble. However, it held firm and proved to be a massive asset to the project. This is because a CICD suite was built to support this gating policy, and it was created for the developers, by the developers.
These were not requirements forced upon the development team, they were ones that we imposed to keep ourselves honest and to keep our codebase healthy. Technology would be key in assisting us to comply with our own development process.
It started out as merely running some tests, but evolved into a suite of pipelines that were created to resolve developer frustration - code passing tests failing to be deployed, concerns about dependency vulnerabilities, deployed microservices failing to integrate together, and so on.
CICD is all about automating away tasks from developers to make their lives easier. They don’t have to worry about their code-breaking deployment, breaking integration layers, degrading the system, etc. They can rely on a CICD system to give them confidence that their code is healthy …or improve the CICD system if it doesn’t give them such reassurance.
You can’t buy an effective CICD pipeline off the shelf, and there’s by no means a “best pipeline”, or “best CICD platform”. That being said, I’ve been using GitLab a lot lately for my CICD needs and I’m a huge fan of what it can do.
The best CICD pipeline is the one made by your team, to serve your team.
To show you the evolution of a simple CICD pipeline - and give you some ideas to mull over - let’s take a look at how the pipeline evolved in the project I mentioned earlier. You’ll see the process of continual improvement that provided developers with more and more confidence in what they were making.
Bare basics
To start us off we asked the question, what must happen before code is merged into the codebase? The answer was resounding, it must pass tests. The project was in its infancy and few tests were actually written, however stating this requirement early on meant that our codebase would always pass tests.
So long as developers were writing accurate tests then we could make the assumption that any code that was accepted into the codebase had also passed the base-level of quality check provided by the unit tests, integration tests, etc. To provide the assurance that tests were being written, we relied on peer review acceptance (two people must approve a merge) and the controversial ‘code-coverage’ metric (a blog post could be written about the usage of this alone).
Ready to go
All code was now being tested against tests that covered the majority of the functionality. The next question was, what do we do with this code? It works, it’s tested a little bit, so what? As part of the process of choosing a coding language, tooling, etc, we decided to containerise the code using Docker so that it could be deployed (almost) anywhere. Each microservice received a Dockerfile to define the container build process and as a developer was working they could at any time test the code in a container on their local machine.
Now that the CI was live and kicking to enforcing testing, the question was asked again - “what must happen before code is merged into the codebase?”. We like to make sure that the codebase is always able to be built into a container - a great next step to add to our CI pipeline.
Live and kicking
Now that the microservices were being routinely built into containers, deployment of those containers was the next step on the agenda. This could, and was, done manually for a short while and informal scripts were made as the team grew tired of such a repetitive task. Putting this task into the pipeline would be the beginning of us considering what should happen after new code was merged into the codebase.
Thus, a post-merge pipeline was born. If we assume that a new commit into the codebase has passed a base-level of tests and can be built into a container, let’s deploy that container!
Is it healthy?
By this point, a good development cadence had been established. People were reviewing each other’s code in pull requests, code was passing unit testing and every time a change was merged into the codebase, it was deployed to a developer environment. Was it healthy though?
Microservices were passing unit tests and even some newly introduced end-to-end tests to check that ‘expected request = expected reply’ for a given microservice. We were finding that the services, once deployed, may not be acting as expected though. They might be missing an expected environment variable, they need more resources to function, or they are dependent on a missing resource like Redis.
To alleviate the pain of realising that a microservice has been broken for the last x commits, we introduced a test after deployment to check the health of the service. This could be as simple as it responding with an HTTP 200 when a health-check endpoint is called, or you could go a step further and make a request then check what it replies with.
Either way, it’s a great addition to the pipeline as we will know that the deployed microservice is acting as we expect.
Will it be healthy?
The CICD pipeline tells us now if deployment of the codebase is healthy or not, which is great as we get feedback quickly instead of waiting until we try to use the deployed microservice. However, a savage loop develops when merging code into the codebase - the pull request pipeline passes all checks and is merged, then the post-merge pipeline deploys the microservice but it fails the health check. Now the developer has to find and fix the problem, open another pull request, get two people to review it, and merge hoping it will pass health-check.
What if we could predict if a pull request was going to break the codebase and prevent it from happening in the first place?
Why not add the deployment & health-check steps to the pre-merge pipeline which gates acceptance into the codebase. That way we can be sure that the code passes all existing tests can be deployed successfully, and that the deployment is healthy.
Cleaning up
The slight problem now, with every pull request we will have a new version of the microservice deployed. We are going to have hundreds of pull requests across all of the microservices, so we are going to have hundreds of versions deployed *screaming internally*.
Let’s add a step to clean that up. For a pull request pipeline, we’ll deploy the microservice, check its health, and then pull it back down. Pull request deployments are short-lived so they won’t clog up the environment or cost us too much money while retaining the boost of confidence that when the code is merged then the codebase will still be healthy.
Multiply
The codebase health is reasonably consistent now, and that means that our deployments are pretty healthy too. We can explore the reusability of both the microservice and our CICD pipeline by recycling or templating our deployment code and deploying it into multiple environments. That’ll prepare us for minting stable releases and deploying them into different environments - testing, pre-production, production, user acceptance testing, etc.
Only the beginning
There is so much you can do with a CICD pipeline beyond the regular process of test & deploy, but this is a great start! You will have seen here how the CICD pipeline grew based on the desires of the developers to:
- Solve problems they were repeatedly encountering
- Make the system as a whole more stable and deployments more consistent
You could take this pipeline in so many different directions from here based on the interests of your team, your stakeholders…everyone directly involved in the project. Here are some ideas to get you started:
- Non-blocking security jobs to monitor application & container dependency vulnerabilities
- Jobs to deploy multiple microservices and test that they integrate with each other as expected
- Automatically create, test, and publish release candidates every fortnight from the last known stable version of a microservice
- Deploy the entire system in a temporary environment for end-to-end testing, then pulldown after testing
- Automatically document the outcomes of tests in a knowledge-base
That’s all for now, thanks for reading!
Stay tuned for more posts on CICD, Data, and the Cloud
About the author
Matthew Grey is a principal technology engineering consultant at Servian specialising in Google Cloud.
You can reach him on LinkedIn or check out his other posts here on Medium.
