Blog

Many of my peers within the software world know the value of Continuous Integration and don't need convincing.
This article is for everybody else.

Introduction

In my first job out of college we had what you'd recognise as CI, though the term wasn't so popular then. It was powerful, very useful, but a source of Byzantine complexity.

I've also worked for people who didn't think CI was worth doing because it was too expensive to set up and maintain. This is not totally unreasonable; the real question is to figure out where the value for your project might lie.

Recently, a friend wrote:

I don't really know very much about CI. I would be interested in knowing
more and might even use some of the quick wins (...)
I do not want to become completely reliant upon GitHub for anything.

So let's start with a primer.

Terminology: What is CI?

Unfortunately the term "CI" is sometimes misused and/or confused.

The short answer is that it's automation that regularly (continuously) does something useful with your codebase. These actions might take place on every commit, nightly, or be activated by some external trigger.

CI usually refers to a spectrum of practices, each step building on the last:

Exactly what these phases mean for your project, and how far you go with them, depends on your project. What suits my embedded firmware probably won't suit your cloud app or that other person's desktop app.

The lines between the phases are blurry. For example, it may not make sense to build and integrate everything in one go.

❔ Why CI

If deployed appropriately, CI can save time, reduce costs and improve quality. Even on a hobby project, there is often value in saving your time.

1. Automating stuff, so the humans don't have to

You could use your engineers to do the repetitive drudge work of creating a release across multiple platforms.

You could have them run a full barrage of tests before committing a code change... but should you?

Engineers are expensive and generally dislike boring stuff, so the smart business move is usually to automate away the repetitive parts and have them focus where they can deliver most value.

If you're not sure, consider this: how much time does your team spend per release cycle on the repetitive parts? Consider your expected frequency of release cycles, that should lead you to the answer.

2. Automatic analysis and status reporting

One place I worked had a release process which relied on an engineer reading multiple megabytes of log file to see if things had been successful. Many things could go wrong and leave the final output in a plausible but half-broken state. Worse, it wasn't as simple as running the script in stop-on-error mode, because some of the steps were prone to false alarms.

You may be ahead of me here, but I didn't think much of that setup.

Compilation failed? Show me the compiler output from the file that failed.

A test failed? I want to see the result of that test (expected & observed results).

Everything passed? Great, but don't spend megabytes to convey _one single bit_ of information.

At its simplest, a small project will have a single main branch, and the operational information you need can be boiled down to a small number of states:

Something is broken	Non-critical warning (not all projects use this)	Everything is working

In a non-remote workplace it might make sense to set up some sort of status annunciator.

• Some people use coloured lava lamps or similar.
• At one place I worked the machinery in the factory had physical traffic light (andon) lamp sets. We set one of these up, driven by a Raspberry Pi wired in to the build server.
• Some projects build more elaborate virtual dashboards that suit their needs. Multiple branches, multiple build configurations, whatever makes sense.

3. Improved quality

This one might be self-evident, but I'll spell it out anyway.

A good CI system will let you incorporate tests of many different types, with variable pass/fail criteria.
Think beyond unit and integration testing:

• Regression (check that your bugs stay fixed)
• Code quality (code/test coverage analysis; static analysis; dynamic memory leak analysis; automated code style checks)
• Security analysis (are there any known issues in your dependencies?)
• License/SBOM compliance
• Fuzz testing (how does it handle randomised, unexpected inputs?)
• Performance requirements
• "Early warning" performance canaries
• Standards compliance
• System data migration
• On-device testing (might be real, emulated or simulated hardware)

If you can automate a really robust set of tests, you can have a lot of confidence in the state of your code at any given time. This gives incredible agility: you can release at any time, if the tests pass. This is the key to moving quickly, and is how many tech companies operate.

For a success story involving physical devices, check out the HP LaserJet team's DevOps transformation.

4. Reduced time to resolve issues

If there's one thing I've learned in the software business, it's that it's cheaper to find bugs closer to development - by orders of magnitude.

In other words, reduce the feedback cycle to reduce your costs. This is where automated tests and checks have great value.

• If there is something wrong in code I modified a minute ago, I'm still in the right headspace and can usually fix it pretty quickly.
• If it takes a few days to get a test result, I won't remember all the detail and will have to refresh my memory.
• If it takes several months to hear that something's wrong, I may be working on a totally different part of the system and it will take longer to context switch.
• If a bug report comes in from the field a year or two later, I might as well be starting again from scratch.

But - as ever - engineering is a trade-off. You can't write a test to catch a bug you haven't foreseen. It may be prohibitively expensive to test all possible combinations before release.

❌ Why not CI

CI is not suitable for all software projects.

If you're writing a scratch throw-away project that won't live for very long, even simple CI may not be worth it.

If you have a legacy codebase that was written without testing in mind, it might be prohibitively expensive to refactor to set these up. Nevertheless, in such projects there is often still some value to be found in a continuous build.

Let's be pragmatic.

Tests aren't everything

On the face of it, more testing means greater quality, right? Well... maybe?

Keep the end goal in sight. It's up to you to decide what makes sense for your situation; I recommend taking a whole-of-organisation view.

• You need to balance test runtime against overall feedback cycles. If the tests take too long to run, you're slowing people down.
• Some tests are expensive in terms of time or consuming resources, so you might not want to run them daily.
• Tests involving physical devices can be difficult to automate, and risk creating a process bottleneck. (Consider emulation and/or simulation where appropriate.)
• Beware of over-testing; you may not need to exhaustively check all the combinations. Statistical techniques might help you out here.
• Beware of making your black-box tests too strict; this can lead to brittle tests that are more hassle to maintain than they are worth.

Costs and maintenance

It will take time and effort to set up CI. How much time and effort, I can't say.

In times past, CI was quite the bespoke effort.

These days there is good tooling support for many environments, so it is usually pretty quick to get something going. From there you can decide how far to go.

It might be too big for your platform

CI platforms are designed for small, lightweight processes. Think seconds to minutes, not hours.

If you need to build a large application or a full Yocto firmware image, it's going to be tough to make that fit within the limits of a cloud-hosted CI platform. Don't despair! There are ways out, but you need to be smart. Alternative options include:

• self-hosting CI runners that are take part in a cloud source repository;
• self-hosting the CI environment (e.g. Gitlab, Jenkins, CircleCI), noting that most source code hosting platforms have integrations;
• split up the task into multiple smaller CI jobs making good use of artefacts between stages;
• reconsidering what is truly worth automating anyway.

👷 Steps you can take

1. Build your units

In most projects you already had to set up a buildsystem. Automating this is usually pretty cheap though you will need to get the tooling right.

Sidebar: Tooling on cloud platforms

On-cloud CI (as provided by Github, Gitlab, Bitbucket and others) is generally containerised. What this means is that your project has to know how to install its own tooling, starting with a minimal (usually Linux) container image.

This is really good practice! Doing so means your required tools are themselves expressed in source code under revision control.

Where this might get tricky is if you have multiple build configurations (platforms or builds with different features). Don't be surprised if automating reveals shortcomings in your setup.

If you have autogenerated documentation, consider running that too. (In Rust, for example, it could be as easy as adding a cargo doc step.)

2. Test your units

Adding unit tests to CI is usually pretty cheap though it will depend on the language and available test frameworks.

If you want to include language-intrinsic checks (e.g. code style, static analysis) this is a good time to build them in. Some analyses can be quite expensive so it may not make sense to run all the checks at the same frequency.

3. Integrate it

If you're pulling multiple component parts (microservices, standalone executables) together to make an end result, that's the next step. Do they play nicely? Do you want to run any isolated tests among them before you move to delivery-level tests?

4. Add more checks

This is where things stop being cheap and you have to start thinking about building out supporting infrastructure.

5. Deliver it

Now we're getting quite situation-specific. Think about what it means to deliver your project.

Are you building a package for an ecosystem (Rust crate / Python pypi / npm.js / ...) ? You might be able to automate the packaging steps and that might be pretty cheap.

Are you building an application? Perhaps you can automate the process of building the installer / container / whatever shape it takes. If you have multiple build configurations or platforms, it could get very tedious to build them all by hand and there is often a win for automation.

Sidebar: Caution!

Where there's code signing involved, you'll need to decide whether it makes sense to automate that or leave it as a manual release step. Never put private keys or other code signing secrets directly into source! Some platforms have secrets mechanisms that may be of use, but it pays to be cautious. If your secrets leak, how will you repair the situation?

Closing thoughts

Most projects will benefit from a little CI. You don't need to have unit tests, though they are a good idea.

You're going to have to maintain your CI, so build it for maintainability like you do your software.

Apply agile to your CI as you do to your deliverables. Perfect is the enemy of good enough. Build something, get feedback, iterate!

CI vendors want to lock you in to their platform. Keep your eyes open.

Don't let CI become an all-consuming monster that prevents you from delivering in the first place!

The QUIC Copier (qcp) is an experimental high-performance remote file copy utility for long-distance internet connections.

Source repository: https://github.com/crazyscot/qcp

Features

•  🔧 Drop-in replacement for `scp`
•  🛡️ Similar security to `scp`, using existing, well-known mechanisms
•  🚀 Better throughput on congested networks

 📖 About qcp

qcp is a hybrid protocol combining ssh and QUIC.

We use ssh to establish a control channel to the target machine, then spin up the QUIC protocol to transfer data.

This has the following useful properties:

• User authentication is handled entirely by ssh
• Data is transmitted over UDP, avoiding known issues with TCP over "long, fat pipe" connections
• Data in transit is protected by TLS using ephemeral keys
• The security mechanisms all use existing, well-known cryptographic algorithms

For full documentation refer to qcp on docs.rs.

Motivation

I needed to copy multiple large (3+ GB) files from a server in Europe to my home in New Zealand.

I’ve got nothing against ssh or scp. They’re brilliant. I’ve been using them since the 1990s. However they run on top of TCP, which does not perform very well when the network is congested. With a fast fibre internet connection, a long round-trip time and noticeable packet loss, I was right in the sour spot. TCP did its thing and slowed down, but when the congestion cleared it was very slow to get back up to speed.

If you’ve ever been frustrated by download performance from distant websites, you might have been experiencing this same issue. Friends with satellite (pre-Starlink) internet connections seem to be particularly badly affected.

Results

Typical results (updated in July 2025) for a 1GB download from Europe to New Zealand. Both endpoints have nominal 1Gbit connectivity (server in a data centre, client on residential 1Gb fibre).

The Internet is a complex system of systems; your mileage may vary.

💻 Getting qcp

You can install it:

• as a package from the release archive;
• as a Rust binary crate, with cargo install --locked qcp (you will need to have the rust toolchain installed);
• by cloning and building the qcp source repository.

You will need to install qcp on both machines. Please refer to the README for more.

References

RFC9000 "QUIC: A UDP-Based Multiplexed and Secure Transport"

This is a tale from many months ago, working on an embedded ARM target.

In my private journal I wrote:

Today I feel like I saddled up and rode my horse to the literal ends of the earth. I was fault-finding in the setting-up-of-the-universe that happens before your program starts up, and in the tearing-it-down-again that happens after you declare you're finished.

If you know C++, you might guess that this was a story about static object allocation and deallocation. You'd be right. So, destructors belonging to static-allocated objects. You'd never think they'd run on a bare-metal embedded target.

Well, they can. If your target supports exit() - e.g. if you are running with newlib - then an atexit handler is set up for you, and that will be set up to run the static destructors. If your program then calls exit() (as, say, your on-silicon unit tests might, at the end of a test run) then things are at risk of turning to custard.

You might have enabled an interrupt for some peripheral on the silicon. In order to do anything really useful, the ISR might reference a static object. If you do this, you'd damn well better make sure the object has a static destructor that disables the interrupt, or hilarity is one day going to ensue. You know, the sort of hilarity that involves being savaged by a horde of angry rampaging badgers, or your socks catching fire.

But wait, I hear you say, it called exit! The program no longer exists! Well, sure it doesn't; but what happens on exit? On this particular ARM target, running tests via a debugger as part of a CI chain, when the atexit handlers have run the process signals final completion with a semihosting call, which is a special flavour of debug breakpoint. It is... not fast. If your interrupt happens regularly, the goblins are going to get you before the pseudo-system-call completes. Your test framework will fail the test executable for hanging, despite somehow passing all of its tests.

There was an actual bug in there, and it was mine. Class X, which contained an RTOS queue and enabled an interrupt, only had a default destructor. On exit, somewhen between static destructors and completion of the semihosting exit call, the ISR fired. It duly failed to insert an item into the now-destroyed queue, so jumped to the internal panic routine. That routine contained a breakpoint and then went nowhere fast, waiting for a debugger command that was never going to arrive --- hence the time-out. Maybe it would have been useful to have a library option to skip the static destructors, but I probably wouldn't have been aware of it ahead of time anyway.

The static destructor ordering fiasco can also be yours for the taking, but thankfully that hadn't bitten me. Nevertheless, it was a rough day.

Cover image: Cyber Bug Search, by juicy_fish on Freepik

We recently created a graphics package for a fencing broadcast, which provides a read-out of the scoreboard to the Captivate (formerly Titler Live) software.

The package can be seen in action on the Commonwealth Junior & Cadet 2024 programmes.

The details read from the scoreboard cover the "hit" lamps, scores, clock (including fractional seconds in the last 10s), period, red/yellow cards, and the priority indicator. On top of that, the package provides a place to enter the fencer names and nationalities, and set colours for them.

This is all made possible by the scoreboard, a Favero FA-07, offering a data feed over an RS-422 interface.

We can build on this experience to create graphics packages for other scoreboards and for other titling packages. Most graphics packages offer an API that we can use.