Tech

Many of my peers within the software world know the value of Continuous Integration and don't need convincing.
This article is for everybody else.

Introduction

In my first job out of college we had what you'd recognise as CI, though the term wasn't so popular then. It was powerful, very useful, but a source of Byzantine complexity.

I've also worked for people who didn't think CI was worth doing because it was too expensive to set up and maintain. This is not totally unreasonable; the real question is to figure out where the value for your project might lie.

Recently, a friend wrote:

I don't really know very much about CI. I would be interested in knowing
more and might even use some of the quick wins (...)
I do not want to become completely reliant upon GitHub for anything.

So let's start with a primer.

Terminology: What is CI?

Unfortunately the term "CI" is sometimes misused and/or confused.

The short answer is that it's automation that regularly (continuously) does something useful with your codebase. These actions might take place on every commit, nightly, or be activated by some external trigger.

CI usually refers to a spectrum of practices, each step building on the last:

Exactly what these phases mean for your project, and how far you go with them, depends on your project. What suits my embedded firmware probably won't suit your cloud app or that other person's desktop app.

The lines between the phases are blurry. For example, it may not make sense to build and integrate everything in one go.

❔ Why CI

If deployed appropriately, CI can save time, reduce costs and improve quality. Even on a hobby project, there is often value in saving your time.

1. Automating stuff, so the humans don't have to

You could use your engineers to do the repetitive drudge work of creating a release across multiple platforms.

You could have them run a full barrage of tests before committing a code change... but should you?

Engineers are expensive and generally dislike boring stuff, so the smart business move is usually to automate away the repetitive parts and have them focus where they can deliver most value.

If you're not sure, consider this: how much time does your team spend per release cycle on the repetitive parts? Consider your expected frequency of release cycles, that should lead you to the answer.

2. Automatic analysis and status reporting

One place I worked had a release process which relied on an engineer reading multiple megabytes of log file to see if things had been successful. Many things could go wrong and leave the final output in a plausible but half-broken state. Worse, it wasn't as simple as running the script in stop-on-error mode, because some of the steps were prone to false alarms.

You may be ahead of me here, but I didn't think much of that setup.

Compilation failed? Show me the compiler output from the file that failed.

A test failed? I want to see the result of that test (expected & observed results).

Everything passed? Great, but don't spend megabytes to convey _one single bit_ of information.

At its simplest, a small project will have a single main branch, and the operational information you need can be boiled down to a small number of states:

Something is broken	Non-critical warning (not all projects use this)	Everything is working

In a non-remote workplace it might make sense to set up some sort of status annunciator.

• Some people use coloured lava lamps or similar.
• At one place I worked the machinery in the factory had physical traffic light (andon) lamp sets. We set one of these up, driven by a Raspberry Pi wired in to the build server.
• Some projects build more elaborate virtual dashboards that suit their needs. Multiple branches, multiple build configurations, whatever makes sense.

3. Improved quality

This one might be self-evident, but I'll spell it out anyway.

A good CI system will let you incorporate tests of many different types, with variable pass/fail criteria.
Think beyond unit and integration testing:

• Regression (check that your bugs stay fixed)
• Code quality (code/test coverage analysis; static analysis; dynamic memory leak analysis; automated code style checks)
• Security analysis (are there any known issues in your dependencies?)
• License/SBOM compliance
• Fuzz testing (how does it handle randomised, unexpected inputs?)
• Performance requirements
• "Early warning" performance canaries
• Standards compliance
• System data migration
• On-device testing (might be real, emulated or simulated hardware)

If you can automate a really robust set of tests, you can have a lot of confidence in the state of your code at any given time. This gives incredible agility: you can release at any time, if the tests pass. This is the key to moving quickly, and is how many tech companies operate.

For a success story involving physical devices, check out the HP LaserJet team's DevOps transformation.

4. Reduced time to resolve issues

If there's one thing I've learned in the software business, it's that it's cheaper to find bugs closer to development - by orders of magnitude.

In other words, reduce the feedback cycle to reduce your costs. This is where automated tests and checks have great value.

• If there is something wrong in code I modified a minute ago, I'm still in the right headspace and can usually fix it pretty quickly.
• If it takes a few days to get a test result, I won't remember all the detail and will have to refresh my memory.
• If it takes several months to hear that something's wrong, I may be working on a totally different part of the system and it will take longer to context switch.
• If a bug report comes in from the field a year or two later, I might as well be starting again from scratch.

But - as ever - engineering is a trade-off. You can't write a test to catch a bug you haven't foreseen. It may be prohibitively expensive to test all possible combinations before release.

❌ Why not CI

CI is not suitable for all software projects.

If you're writing a scratch throw-away project that won't live for very long, even simple CI may not be worth it.

If you have a legacy codebase that was written without testing in mind, it might be prohibitively expensive to refactor to set these up. Nevertheless, in such projects there is often still some value to be found in a continuous build.

Let's be pragmatic.

Tests aren't everything

On the face of it, more testing means greater quality, right? Well... maybe?

Keep the end goal in sight. It's up to you to decide what makes sense for your situation; I recommend taking a whole-of-organisation view.

• You need to balance test runtime against overall feedback cycles. If the tests take too long to run, you're slowing people down.
• Some tests are expensive in terms of time or consuming resources, so you might not want to run them daily.
• Tests involving physical devices can be difficult to automate, and risk creating a process bottleneck. (Consider emulation and/or simulation where appropriate.)
• Beware of over-testing; you may not need to exhaustively check all the combinations. Statistical techniques might help you out here.
• Beware of making your black-box tests too strict; this can lead to brittle tests that are more hassle to maintain than they are worth.

Costs and maintenance

It will take time and effort to set up CI. How much time and effort, I can't say.

In times past, CI was quite the bespoke effort.

These days there is good tooling support for many environments, so it is usually pretty quick to get something going. From there you can decide how far to go.

It might be too big for your platform

CI platforms are designed for small, lightweight processes. Think seconds to minutes, not hours.

If you need to build a large application or a full Yocto firmware image, it's going to be tough to make that fit within the limits of a cloud-hosted CI platform. Don't despair! There are ways out, but you need to be smart. Alternative options include:

• self-hosting CI runners that are take part in a cloud source repository;
• self-hosting the CI environment (e.g. Gitlab, Jenkins, CircleCI), noting that most source code hosting platforms have integrations;
• split up the task into multiple smaller CI jobs making good use of artefacts between stages;
• reconsidering what is truly worth automating anyway.

👷 Steps you can take

1. Build your units

In most projects you already had to set up a buildsystem. Automating this is usually pretty cheap though you will need to get the tooling right.

Sidebar: Tooling on cloud platforms

On-cloud CI (as provided by Github, Gitlab, Bitbucket and others) is generally containerised. What this means is that your project has to know how to install its own tooling, starting with a minimal (usually Linux) container image.

This is really good practice! Doing so means your required tools are themselves expressed in source code under revision control.

Where this might get tricky is if you have multiple build configurations (platforms or builds with different features). Don't be surprised if automating reveals shortcomings in your setup.

If you have autogenerated documentation, consider running that too. (In Rust, for example, it could be as easy as adding a cargo doc step.)

2. Test your units

Adding unit tests to CI is usually pretty cheap though it will depend on the language and available test frameworks.

If you want to include language-intrinsic checks (e.g. code style, static analysis) this is a good time to build them in. Some analyses can be quite expensive so it may not make sense to run all the checks at the same frequency.

3. Integrate it

If you're pulling multiple component parts (microservices, standalone executables) together to make an end result, that's the next step. Do they play nicely? Do you want to run any isolated tests among them before you move to delivery-level tests?

4. Add more checks

This is where things stop being cheap and you have to start thinking about building out supporting infrastructure.

5. Deliver it

Now we're getting quite situation-specific. Think about what it means to deliver your project.

Are you building a package for an ecosystem (Rust crate / Python pypi / npm.js / ...) ? You might be able to automate the packaging steps and that might be pretty cheap.

Are you building an application? Perhaps you can automate the process of building the installer / container / whatever shape it takes. If you have multiple build configurations or platforms, it could get very tedious to build them all by hand and there is often a win for automation.

Sidebar: Caution!

Where there's code signing involved, you'll need to decide whether it makes sense to automate that or leave it as a manual release step. Never put private keys or other code signing secrets directly into source! Some platforms have secrets mechanisms that may be of use, but it pays to be cautious. If your secrets leak, how will you repair the situation?

Closing thoughts

Most projects will benefit from a little CI. You don't need to have unit tests, though they are a good idea.

You're going to have to maintain your CI, so build it for maintainability like you do your software.

Apply agile to your CI as you do to your deliverables. Perfect is the enemy of good enough. Build something, get feedback, iterate!

CI vendors want to lock you in to their platform. Keep your eyes open.

Don't let CI become an all-consuming monster that prevents you from delivering in the first place!

This is a tale from many months ago, working on an embedded ARM target.

In my private journal I wrote:

Today I feel like I saddled up and rode my horse to the literal ends of the earth. I was fault-finding in the setting-up-of-the-universe that happens before your program starts up, and in the tearing-it-down-again that happens after you declare you're finished.

If you know C++, you might guess that this was a story about static object allocation and deallocation. You'd be right. So, destructors belonging to static-allocated objects. You'd never think they'd run on a bare-metal embedded target.

Well, they can. If your target supports exit() - e.g. if you are running with newlib - then an atexit handler is set up for you, and that will be set up to run the static destructors. If your program then calls exit() (as, say, your on-silicon unit tests might, at the end of a test run) then things are at risk of turning to custard.

You might have enabled an interrupt for some peripheral on the silicon. In order to do anything really useful, the ISR might reference a static object. If you do this, you'd damn well better make sure the object has a static destructor that disables the interrupt, or hilarity is one day going to ensue. You know, the sort of hilarity that involves being savaged by a horde of angry rampaging badgers, or your socks catching fire.

But wait, I hear you say, it called exit! The program no longer exists! Well, sure it doesn't; but what happens on exit? On this particular ARM target, running tests via a debugger as part of a CI chain, when the atexit handlers have run the process signals final completion with a semihosting call, which is a special flavour of debug breakpoint. It is... not fast. If your interrupt happens regularly, the goblins are going to get you before the pseudo-system-call completes. Your test framework will fail the test executable for hanging, despite somehow passing all of its tests.

There was an actual bug in there, and it was mine. Class X, which contained an RTOS queue and enabled an interrupt, only had a default destructor. On exit, somewhen between static destructors and completion of the semihosting exit call, the ISR fired. It duly failed to insert an item into the now-destroyed queue, so jumped to the internal panic routine. That routine contained a breakpoint and then went nowhere fast, waiting for a debugger command that was never going to arrive --- hence the time-out. Maybe it would have been useful to have a library option to skip the static destructors, but I probably wouldn't have been aware of it ahead of time anyway.

The static destructor ordering fiasco can also be yours for the taking, but thankfully that hadn't bitten me. Nevertheless, it was a rough day.

Cover image: Cyber Bug Search, by juicy_fish on Freepik

This is an article about software tooling and the developer experience.

Background

At the time of writing this article I am working on an embedded project. It is C++ running on bare-metal target hardware. I've got all of 768K of RAM to play with, but that's still a tad more than my first home micro had.

Our tooling and infrastructure grew mostly organically. I was the one who initialised the git repo.

CI chain 🔗

We have a CI chain, of course. It has grown over time and does a few tricks:

• build for the target hardware (various targets; RAM and flash builds)
• build on Linux native port and run unit tests
• perform various analyses (clang-tidy, sonarcloud, code coverage, valgrind)
• code style enforcement (clang-format) 👮
• prepare a signed upgrade image with the development signing key
• assemble a whole-of-flash image for the factory 🏭
• build and run some unit tests on a development board

The dev board is on-premises but the rest happens in the cloud, taking a matter of minutes on every push. It's reasonably slick.

You can run all the build and analysis steps locally. We have a script that spins up a container emulating our CI provider so you don't even need to have the dev tools installed, you only need to pull our in-house container image.

How much effort to spend on tooling? 💸

Many a savvy manager or product owner will tell you to not spend too much time on tooling. They have a point; time spent working on tooling and infra is not delivering features, after all. Many managers have been bitten by infrastructure blow-outs and would rather we focus on building the stuff they want us to build.

But here's the thing. When they say that, they don't mean "forget about tooling and never speak of it again". You are allowed to bring the subject up again in future, and you are allowed to try to educate them as to the value.

The frog boils 📉

We now have over 300 source files that get compiled into the image, which includes some vendor and autogenerated code.

There's the rub. We didn't start out with anywhere near that many. The CI chain used to be really fast, under one minute. Local builds were fast, and even running the clang-tidy script wasn't that awful (though I did build a cache for it, which helped).

The size and complexity grew over time, and here we are today.

It's like the old tale about boiling frogs. (Disclaimer: I don't even know if this is true, and I definitely don't suggest you try it, but it's an interesting thought experiment.)

What happens if you drop a frog into a pot of boiling water?

The frog doesn't like it and jumps out.

But what happens if you drop a frog into a pot of cold water, then light the stove? The frog doesn't realise anything is wrong. It boils to death before it thinks to jump out.

Builds with a cold cache were taking a couple of minutes locally, and about 8 minutes in the cloud. (This happens depressingly often when you change a low-level header that is included all over the place... there's another observation, that maybe our architecture needs work, but that's for another day.)

A full static analysis with cold cache was taking over 5 minutes locally, and 22 in the cloud. (The difference is because our company-provided laptops have 16 cores, while our CI provider runs our container with 4 vCPU cores and tightly limited RAM.)

Goodness me, 300 compile units now?
This water sure is getting warm! 😰

It's a poor developer experience. Even running the code style checker script takes five whole seconds. That's 5 seconds in which you can get distracted, pop over to Slack, and whoops! ten minutes later, you're trying to remember what it was you were doing.

I've been in this business long enough to know that keeping myself and my team focussed is vitally important to getting things done and keeping the stakeholders happy. An efficient DX is part of that puzzle.

Like the mythical frog, I don't remember noticing these tasks growing longer.

A concrete walk-through 🔧

OK, so you've decided you need to jump out of the pan and spend some time sharpening your tools. Great.

The most important advice I can offer is to understand where the pain points are. 🔍

If this is stuff you work with every day, you maybe already know where the pain points are but are blind to them because you haven't noticed how poor the DX has become. Open your eyes, and you may find it is in plain sight. Don't optimise for its own sake; optimise where it will help.

Here's what I noted about our tooling:

• The compilation time itself is already pretty well optimised and parallelised thanks to meson, ninja and ccache.
• The code style checker is something we run very frequently as part of our workflow (it's in our pre-commit script). At 5 seconds, that's like sand in your gears.
• The static analyser is a different type of pain point. We run it interactively but we don't like to because it's a slow ol' beast. On the cloud, it is the slowest part of the build; it is probably the reason why we're regularly incurring overage on our CI build minutes. (Now, that's not a big cost compared to an engineer's salary, but it's not zero and it has been noticed, so maybe worth spending a little time on.)
• The unit test run on a dev board is a bottleneck, practically by definition, as it uses a self-hosted CI runner plugged into our pipeline.

Code style checks: Parallelise!

We use clang-format with an in-house style file.

The heart of our checker script is a shell function.

The tl;dr version is:

• We were invoking clang-format in a tight loop, once per input file. 🤦
• Now, we count the files, divide by (slightly more than) the number of CPU cores available, and have xargs parcel out the work into that many chunks. 🚅

Static analysis: Reconsider the load, cache, and parallelise where it will help!

The learnings from this one are:

• Do the heavy lifting in parallel (we were already)
• Cacheing positive analysis results helps (we were doing that too)
• Don't analyse unit tests or test harness/mocks/fakes unless you have a good reason
• Don't analyse target-specific files for obsolete hardware targets unless you have a good reason
• Where you have a string-and-glue shell phase to prepare the analysis run, it can be worth parallelising that as well.

Results 🚀

(Timed on my work laptop, which has 16 cores. It's running Ubuntu 22.04 on WSL on Windows 11.)

My next steps

• The on-target testing phase is a bottleneck, so we're going to add more hardware.
• There might be some mileage in using using [pre-compiled headers](https://mesonbuild.com/Precompiled-headers.html) to speed up cold-cache compilation.
• This isn't something the places I have worked have traditionally done, but I know it's popular when you're dealing with the Win32 C++ api.
• Our codebase has a "platform" layer; this smells like a good starting point for precompilation.
• Coverage analysis. We already do this, but there's another pain point while `genhtml` crunches the data and generates the HTML report. This one could be trickier; it's a single-threaded perl script, invoked by meson.
• Stack usage analysis. We've got some code paths that can be provoked to overflow on debug builds. This is a different sort of pain point; while we've configured the silicon to detect and hard-fault on overflow, I'd like to set up some analysis to automatically find these situations ahead of time.

Closing thoughts

How long to spend on infrastructure? 💸

Circling back to a question I asked earlier, how long is a piece of string?

It depends where you are and what your current DX is like.

10-20% of your developer time seems a reasonable benchmark; it's enough that you can get a handful of DX tickets or story points through every iteration without becoming too much of a drag on progress. After all, if you're working agile (and who isn't, these days?) constant improvements are the key.

How's your developer experience looking?