2023 12 23 Build Gripes

I was chatting with Dan Lorenc from Chainguard (who I adore as a company) on Twitter about the trevails of patching open source software last week, and immediately thought of many stories-from-the-trenches I’ve accumulated during the past 25 years or so of working at Akamai Technologies of Cambridge, MA.

We are heavy users of open source software and in many situations will use our own builds of stuff rather than distro packages. While this obviously creates some extra overhead, it means that we have a lot of in-house expertise in building and releasing third-party libraries that are critical to us if the need arises.

Probably the biggest challenge in this model is the initial effort required to get a new library into our build system. Many open-source libraries take little more than running ./configure; make install to be usable, but sometimes that process can be agonizing if the package is using some hand-rolled scheme for passing site-specific information to the build.

One of the best pieces of advice I ever read about this topic (I can’t remember who it was but I know they were from Redhat), was basically “use autoconf or you’re an <expletive>”. That was probably 10 or 12 years and today you can use CMake or Meson too without worrying about being an <expletive>, but their point– that it is best to adhere to popular conventions when interfacing with outside users of your software –is still valid.

Back then, circa 2000 or so, builds for packages fell into a handful of categories. First was autoconf-based GNU stuff or non-gnu stuff that just used autoconf because that’s what you did. Then there were a small number of Cmake early adopters, no big whoop. There were projects who managed to meet all their configuration needs with cpp and just shipped with a Makefile. Maybe you’d have to pass it some target name or some variable definitions for your platform, but you could usually just run make and maybe they didn’t understand DESTDIR or honor a –prefix type variable. But fundamentally its just a Makefile so we can hack around it. Sometimes a component would fail to follow the autoconf wisdom and wouldn’t work in an out-of-tree build. Usually these are correctable with some minor patching, but it starts to get annoying, particularly if the maintainer(s) wouldn’t accept the patches that Debian or Redhat maintainers needed to add to their packaging harnesses to work around these nuisances.

The absolute worst, however, were projects where the maintainer had some grudge against autoconf. I get it because autoconf and everything in its orbit, automake, m4, libtool, yada yada, are all a pain, but this maintainer would put a phony configure script in the dist tarball containing a bunch of goofy shell code that they wrote because they couldn’t be bothered to figure out how to do it in autoconf. Most of the time they would get everything wrong. They don’t handle –enable-static or –disable-shared, they don’t know how to call pkg-config –static, they halfway implemented DESTDIR except for some things they forgot, etc. They hardcode CFLAGS/LDFLAGS et al so that there’s no easy way to avoid patching the Makefile by passing those via the environment. Then, maybe they don’t handle something like LIBS="…" which gets postfix-appended to the linker invocation in order to resolve some missing symbol that is not easily corrected via LDFLAGS.

When multicore CPUs began their ascendance and we started to realize that make -j was an option, packages in this category were also often offenders with respect to including implicit dependencies in Makefiles that would break parallel builds, although it’s easy to screw up an automake input in that way as well. I have spent countless hours stepping through other people’s busted Makefiles one rule at a time trying to correct an implicit dependency, and although its satisfying to be able to send out the patch that fixes it, ideally maintainers would invest some time in learning how the build tool works and treating it like a first-class element of your library, the likelihood that professionals will use and love it increases dramatically.

Most newer components I encounter now are using either CMake or Meson and get most of the distro support right. They export both .cmake metadata and pkg-config files, make sensible choices about option flags, and CMake and Meson make it easier on maintainers when it comes to stuff like static or out-of-tree builds. Maintainers may never need to build their library in that way, and so if the build tool is written in such a way that those things work without any need for the maintainer to adhere to a set of arcane practices, the world is better for everyone.

A more nefarious problem that I have begun to observe recently, though, is management of build-time dependencies in C and C++ libraries like grpc and protobuf. If I’m a googler using Bazel, this stuff is mostly magically taken care of for me, but if I’m building in some other context, there is thi somewhat terrifying web of dependencies I need to build. In some cases the sources for these dependencies are vendored into the source tree, by either actually including them in the repo (as gnulib is typically used), by including them as git submodules, or by relying on features in CMake or Meson which retrieve the dependency sources at build time. I am a believer in fully hermetic builds for reasons that would fill a separate blog post, and while I understand that there are ways to safely extend the trusted part of the build enclave to include retrieving sources at build time, these often don’t cut the mustard when it comes to reliability and reproducibility of builds, and also leak a signal about who is using what when building.

The real core of my gripe is then that I want maintainers of systems of this type to recognize that its important for packages to be capable of being built on airgapped systems. I don’t mind for the onus to be on me when it comes to retrieving and somehow storing these dependencies in advance, but I don’t want to need to read through a cobbled-together Makefile which is different in every project to figure out what combination of random flags or commands that I need to run in order to prefetch those sources in order to build your project.

Also, if I am going to build a project whose dependencies have overlapping transitive dependencies, for example two libraries which both depend on a vendored version of abseil, maintainers need to have their eyes opened to the fact that people are building this stuff at other places and in somewhat different ways to suit different needs. Fortunately on most occasions where I have run into a problem of this sort, I’ve found that distro maintainers found it sooner than I did, and because most successful distro package maintainers quickly learn to be good citizens in their interactions with their upstream maintainers, they usually blaze a path to a solution successfully and then randos like me can assist with minor additional fixes.

Having seen how the batteries-included style dependency management in toolchains like Go and Rust have greatly reduced much of the toil related to problems of this type, and seeing some early progress in the world of c++ modules, Conan, etc. inspired by other languages, I suspect that many of these issues will start to diminish about a month after I retire 10 or 12 years from now. Also CMake and Meson keep improving and although I wish it was just one or the other, the fact that there are a handful means that for any of them to be convenient, they need to assume a world in which there are a diversity of build systems and that they will be better citizens if they find ways to interoperate more consistently. This is beginning to sound like a feel good Christmas message, and perhaps it is.

Anyhow I hope Dan gets a chance to read this, would love to hear some of the Chainguard image patching stories and am looking forward to squashing many more CVEs in 2024!