CentOS, as most who would use it should know, is a binary-compatible re-creation of Red Hat Enterprise Linux. In theory, it is the same, minus trademarks and branding-related items removed (generally, the term “Red Hat” and its associated logos are removed from documentation, text, and interfaces). However, it takes time for CentOS to release an equivalent update to its upstream’s version. Case in point, Red Hat 7.7 was released around 8th of August 2019. CentOS just released their update on th 18th of Setpember - a month and a half later. Why the delay?

There are a lot of questions asked online about the differences between the two distributions, but the delay seen is never really addressed adequately (as far as my searching has found). I was prompted to do another little bit of digging this morning after noticing the release announcement, and found that the answer is hinted at in the FAQ, though some extrapolation with knowledge of build processes is helpful to actually understanding.

Note that this is all my own extrapolation from the FAQ and documentation of the CentOS project, not any specific knowledge of the actual build/release cycle of either project.

Firstly, it is important to note that CentOS is built from the same sources as Red Hat, but not in the same environment:

CentOS Linux is built from publicly available source code provided by Red Hat®, Inc for Red Hat® Enterprise Linux in a completely different (CentOS Project maintained) build system.

– FAQ, point 8

From point 2, we are directed to see the Rebuild and Release Process page which highlights in commentary the delay:

The goal for release of a new point release or update set is four to eight weeks after the release by upstream. It may move + or - about two weeks. A new major version will be expected to take longer than a mid-cycle point release; a mid-cycle point release similarly takes more time than the ‘entry into maintenance mode’ point release.

So the point release delay is known to take over a month, hopefully no more than two and a half months. But why?

This is where awareness of large project build processes is useful - there are two main things that need to happen, and successfully:

  • The actual build of all the sources into packages
  • The verification that all software passes through the test suites to ensure its built version does what it needs to, and that they integrate correctly

This is on top of doing the work to remove the branding, and running whatever sanity checks need to be run to be sure any new trademark-related items have been removed.

Since the build system is different, the support for both of these processes must be re-created separately for CentOS, and maintained to support new point releases. In an ideal world, this should be relatively trivial for a point release: pull some updates, run the same process as was done previously, and be done. For major releases, infrastrucutral changes in the project may lead to build procedures needing significant overhaul, or for new build procedures to be created outright, as well as corresponding functional and integration tests to be developed.

Point 5 of the FAQ indicates that in many cases, what looks like trademarks are actually required names in source code, or must remain so that third-party scripts targeting Red Hat can also run on CentOS - the binary compatbility claim relies on this to be true. So work also has to be done to ensure that these source-related names are not caught up in the trademark removal process. In all likeliness, this is something that needs adjusting every so often, most notably when new packages are introduced in major releases.

Software building in itself takes a while. Rebuilding after issues arise simply add to the release’s timeline.

All in all, this all makes for any number of configurations, updates, and testing to add to the re-release time of otherwise known source. Writing good software may be hard, but verifying it is equally difficult, especially if you are trying to place guarantees on it that will be depended on worldwide.

So for me this morning, I feel I have gained some new insight into what is involved simply in forking a project and rebuilding it for a worldwide audience. I thank the CentOS project for the efforts they put into this excellent system. You’re doing awesome work.