There is a well-known observation in software engineering that the number of bugs per thousand lines of code has stayed roughly constant for decades. Our tools have got better and more mistakes get caught automatically. But despite that, the bug rate has barely moved since the 1970s! The reason seems to be that when better tools eliminate one category of error, we spend the freed-up headspace on more complex logic, which introduces its own bugs. Basically complexity fills the space.

Personally I find this reassuring. It means your code is not unusually buggy. But it also means that just concentrating harder will probably not reduce the error rate. What works is building in structure: design patterns that limit where bugs can hide, sharing your code so other people can run it and spot what you missed, writing functions that do one thing so the logic is easier to check. This thread runs through everything I wrote this month.

What I wrote this month

Applying the Open-Closed Principle to Research Code

Last month I wrote about the Single Responsibility Principle, the idea that each function should do exactly one thing, and how keeping your functions focused makes research code easier to read, test, and reuse. This month I have moved to the next letter in SOLID: the Open-Closed Principle. Once a function has produced a reported result, it should be closed for modification. New behaviour comes from new functions, not from adding elif branches to existing ones. If you have ever rerun an analysis months later and got a slightly different number, this post is for you. I walk through a practical example and show how the principle connects to ideas researchers already understand, like pre-registration and versioned protocols.

Why Synthetic Data is Good for Open Science

The most common reason researchers give me for not sharing their code is that they cannot share their data, so sharing the code feels kind of pointless. Synthetic data can be a bridge here, and the counterintuitive part is that it does not need to be perfect. It just needs to be structured enough that someone can run your code against it. This post covers what we actually need from a synthetic dataset for reproducibility purposes, and why the biggest win is in checking data cleaning and manipulation code, which is often the most complex and error-prone part of an analysis.

What I am building

I am putting together a course on codelists. A codelist is a carefully curated list of clinical codes that defines what counts as a particular condition, medication, or procedure when you are working with health data. Getting one wrong is surprisingly easy and surprisingly consequential: miss a single code and you can silently drop thousands of patients from a study. The course will cover the full process from understanding the coding systems through to validation and sharing, built around realistic exercises rather than toy examples. You can follow along at courses.carolinemorton.co.uk.

Something worth reading

A Large-Scale Study on Research Code Quality and Execution by Trisovic et al. (2022). The authors retrieved over 2000 replication datasets containing more than 9000 R files from the Harvard Dataverse repository and tried to re-execute them in a clean environment. 74% of files failed to complete without error on the first attempt, and even after automated code cleaning, 56% still failed. The most common errors were exactly the kind of thing that good practices prevent: wrong directory paths, missing dependencies, and undefined variables. The paper also found that journals with stricter data sharing policies had noticeably higher re-execution rates. If you want a concrete, evidence-based case for why the things we talk about in this newsletter matter, this is it. I reference this paper a lot when I speak about science at conferences and the numbers still shock me!

Caroline Morton is a doctor, epidemiologist, and senior software developer. She writes about research code quality and general software engineering at carolinemorton.co.uk.