Hard Build System

Everything should be made as simple as possible, but not simpler.

Build systems for operating systems implement a complex task and tend to be highly customizable. It's crucial to pick good abstractions to manage this complexity. A source code tree has many aspects of configuration, such as the target processor architecture, what optimizations and warnings are used, where the code is installed, which compiler is used, and so on. No solution fits all.

It's tempting to fall in the trap of providing an easy build system. The user merely runs a script with administrator access, causing system-wide changes to occur, source code to be built, programs installed, and a somehow an operating system image appear on the other end. No effort, just waiting. This approach is too simple and ends up being too complex.

Example

Cross-building a fully-featured Unix-like operating system with an user-space and ports could be done using these steps (requiring decisions):

• Download the source code (toolchain sources, core operating system, ports).
  • Decision: What mirrors to download from.
  • Decision: Where to install the source code.
• Install required programs as described in the documentation.
  • Decision: Where to get the programs from (package management (deb, rpm, ...), build from source, ...).
  • Decision: What version of the programs are used.
  • Decision: Where to install the programs.
• Build and install custom programs for this operating system.
  • Decision: What compiler is used and the compiler options.
  • Decision: Where to install the programs.
• Prepare a System Root containing the standard library headers (needed to make a Hosted GCC Cross-Compiler).
  • Decision: Where the system root is located.
  • Decision: What processor architecture is used.
• Build and install a cross toolchain:
  • Decision: What compiler and compiler options are used to compile the cross-compiler.
  • Decision: Where to install the cross-compiler.
• Build the core operating system and ports and install into the system root.
  • Decision: What cross-compiler to use and the compiler options used to compile your operating system and ports.
  • Decision: Whether this is a debug build and whether compiler sanitation and security measures are used.
  • Decision: What ports is to be built.
• Create a bootable image from the sysroot contents.
  • Decision: What boot method is used (cdrom, harddisk, floppy).
  • Decision: What to include and what not to (if making a minimal build).
  • Decision: What drivers are needed in the initrd.
• Run the operating system in a virtual machine.
  • Decision: Which virtual machine is used.
  • Decision: What processor features, simulated hardware and how much memory is available.
  • Decision: Whether hardware acceleration is enabled.
• Install the operating system on real hardware.
  • Decision: What bootloader is used.
  • Decision: What harddisk and partition is used.
  • Decision: What filesystem is used.

It is obvious there are many more decisions to make and that users will make different decisions. You shouldn't make most of these decisions for the user. It is the user's responsibility to install software and make system-wide changes, not your build system's. Rather, you should document what needs to be done and let the user do it.

Hard Build System

An Hard Build System just builds the operating system. The user prepares the build environment manually by following the documentation, or perhaps using convenience scripts. It trusts the user is able to install prerequisite programs and libraries. It trusts the user to download the necessary source code.

The user is in control, the user decides what to do and does it manually or invokes the convenient scripts if applicable. This is in contrast to the easy build system, where the script is in control and the user might be consulted. The goal isn't to make things hard, but to embrace that particular decisions must be made by the user and make the design no simpler.

Help the User

Help the user do things:

• Document the process.
• Link to the primary source code mirror.
• Document prerequisite programs and list package names for common Linux distributions.
• Document how to create a cross-compiler (Maybe offer a pre-built toolchain for common systems).
• Have an easy way to create a system root and populate it with standard library headers.
• Default to sane optimization settings if unspecified.

Document the primary way of doing things, and offer secondary short-cuts for those that desire.

Cross-Compilation is Hard

A cross-compilation inherently have more complexity than a native compilation. For instance, two toolchains exist: The one that makes local programs, and the one that cross-compiles. A System Root is used. You have to prepare the compilation specially.

The solution is to accept this fact. Cross-compiling your operating system is the secondary way to build your operating system and it can only be so simple. The primary way to build your operating system is on itself. You might not be there yet, but design your build system to run on your own operating system primarily.

Have your build system always compile for the current system by default. This is the natural choice when you primarily develop under your OS. Have the user explicitly request cross-compilation code by setting CC, HOST, or perhaps passing a --host option to your build system. When your operating system supports multiple processors, it's not clear which platform to target by default when cross-compiling. What if you are on an ARM system compiling your MIPS and x86-64 OS? Solve the issue by targeting the local operating system (even if it is non-sensible to compile your OS with a Linux compiler) and thus forcing the user to specify the intended cross target.

Separate the Toolchain

When you develop your operating system under itself, you use the system compiler to compile your operating system. The system compiler is merely a program installed in a system directory. It is natural for the same to be the case when cross-compiling: The cross-compiler is just a program installed somewhere.

Store the cross-compiler in a directory the user chooses, such as $HOME/opt/x86_64-myos and put its bin in the PATH. Don't put it inside the operating system's source directory, it's unrelated to that.

The compiler built by your toolchain should be the native compiler on your operating system, cross-compiled by the cross-compiler, when you get this far.

Doing is Learning

The user learns by having to manually do operations following documentation. This information is crucial in operating system's development, where the developer is required to be an expert to succeed. Being able to make a cross-compiler is great practice for porting a compiler to your OS. Understanding the compilation process is helpful when it fails.

Good Users Script

You may find that you always make particular decisions (even though some people might take other decisions). In such cases, you can simply write personal scripts that make things easy for you. You can share those scripts, others can be adapt them to their needs or just use them.

Configuration is Personal

Don't require contributors to edit files that are part of the build system (those that would show as changed by version control if edited) to customize trivial decisions. Environment variables and options passed to the build system are examples of personal configuration that exists outside of the general purpose build system.

For instance, you may be tempted to hard-code the path to the cross-compiler directory in your project's Makefile. Don't do this, instead assume it is already in the PATH.

Easy Build System

An Easy Build System is responsible for as much as possible, avoiding any user effort. Degrees of this exist, of course. The ultimate easy build system is when the user blindly pipes an URL into an administrative shell:

download-to-stdout http://example.com/myos.sh | sudo sh

It'll attempt to do as much work as possible to reliably understand the local operating system. It'll figure out what software needs to be installed, detect the distro and install the correct packages, download and compile source code if needed, might make global changes if needed, and generally try to pick sensible defaults. The script will be very complex because the systems out there are very different. The user can merely hope it works and doesn't screw up the local system too much.

Interactive Build System

A step up from the blind easy build system is the interactive build system. It asks questions underway, such as where the source code should be installed. A good one might ask a lot of questions and present sensible defaults. Such scripts tend to be even more complex than the blind scripts.

An alternate way is for the user to download a script and simply edit variables at the top. This avoids the complexity of interaction, at least.

Easy Contributions Fallacy

It's easier to attract contributors if it requires less effort for them to build the project. Preferably no effort at all and no decisions on their part. But consider: Are these contributors are worth it? If they are unable to manually build a cross-toolchain, install necessary programs, download the source code, and everything up - will they be able to make any non-trivial contributions to an operating system?

If you really wish to make things easy, instead of supporting the many wildly different systems they normally use, you should become self-hosting. You can then simply give them an image of your operating system containing the development tools and source code already set up perfectly. They can install it locally or run it in a virtual machine.

Perhaps the goal is only that they can try your OS easily? In that case, you simply need to do occasional releases or set up a nightly build service. Building from source is always harder than just running an existing image.

No Error Recovery

Unfortunately things tend to go wrong when compiling an operating system. A naive script might seriously malfunction if this happens. A reliable script will detect this and simply fail out. This might mean the user has to completely start over, repeating the same work building a cross-compiler, just because you ran out of harddisk space linking the kernel.

A better interactive script might just ask if the user wants to try again or skip to a particular step, at the cost of more script complexity. Compare this with the user just entering commands manually in the shell.

Sourcing Scripts

Some solutions create a small script that the user sources (a shell equivalent of #include) in the current shell. These scripts usually set a few environment variables and adds stuff to the PATH. A hard build system would simply have the user add directories permanently to the PATH in ~/.profile or such.

Trust Issues

It's harder to trust an easy build system, when you have to comprehend complex shell scripts running as root. It's easier to verify commands you enter yourself, compile as an unprivileged user, read the Makefiles and build scripts before invoking them. It's easier to ensure the easy build system doesn't restart renaming files in /usr and other scary things.

See Also

Articles

• Meaty Skeleton - A minimal operating system template with a hard build system.
• Makefile - A makefile tutorial

External Links

• GNU Make manual (PDF is freely available as well)
• Sortix Cross Development Guide — a guide on how to build one of the hobby OSes
• Recursive Make Considered Harmful — A white paper by Peter Miller on the hazards of using a recursive make build and a suggested alternative

Note: While the purpose of Peter Miller's paper is sound, the dependency configuration outlined in section 5.4 is considered inferior to the dependency configuration outlined in Makefile.