Make files not war

This article tries to explain make and it’s makefiles in slightly different terms than the original documentation or even most tutorials. It took me a long time to understand how to write short, simple and understandable makefiles that work, and I hope this article may help people by providing another view.

This article assumes a Linux system and GNU make, though I will discuss some of the differences concerning Windows and BSD.

Table of contents


Prelude

But all makefiles I’ve seen are ugly messes!

As most projects just copy theirs from another project or a “one-size-fits all” makefile in order to get it to work - or even worse, generate one - that comes as no surprise.

These heirlooms are often huge, hard to read and near-impossible to debug, thereby contributing a great deal to make’s reputation as a bad build system in some circles and scaring away people interested in learning how things work.

The biggest source of trouble and complexities is the variety of implementations of the basic makefile convention, resulting in different feature sets being supported across systems. This can be circumvented by concentrating on the GNU implementation of make, which is supported on pretty much any system and provides arguably the widest range of features.

Other than that, I have been very happy with a broadly supported, minimally intrusive build system that has stood the tests of time and is still considered by many to be THE major build automation tool there is.

Why use make

  • It works
  • Easy to set up in new and existing projects
  • Your system probably ships with a version of make
    • or it can be easily installed
  • It’s tiny (and doesn’t depend on much)
  • makefiles can be short, succinct and beautiful
  • Does not need mysterious working or resource folders
  • No opaque magic performed

When to use make

Most projects, be they software development, writing a book or publishing a blog, at some point need to generate “end products” (of some kind) from manually created raw files.

The point of a build (automation) system is to encode how to do that for one item, and apply that rule to one or more objects of the same type, updating only what’s changed.

One non-software example would be rendering changed book chapters to PDFs and subsequently concatenating them to form the final book.

Of course, this can be (and often is) done via a custom script, written in any one programming language, but make’s powerful rule-based system greatly simplifies the process of doing that.

Sometimes it may be useful to run some tools (for example, a test suite or a spell checker) against the complete project, and the built-in bash-like scripting support allows to do that, too.

In fact, the articles on this website are automatically generated from markdown files using make - everytime I push changes to my web server, it runs make against the directory containing the raw articles using this makefile, which we will come back to in the course of this article.


Writing makefiles

Let’s get to discussing how make works by starting with some basic terms and a simple makefile.

Starting out simple

Starting out, let’s create an empty file named makefile (with capital or lowercase M, make accepts both).

The content of a basic makefile can be thought of as rules describing:

  • If any of these files change
    • This file should, too

In make terminology, the second item is called a target - a file to be made by somehow processing some or all of the prerequisites, which form the first item.

The aforementioned rules for processing the prerequisites can be arbitrarily complex, and can consist of calling out to other tools such as compilers, parsers, etc.

The basic syntax for defining a target is

target: prereq1 prereq2 ...
    command1
    command2
    ...
    <blank line>

File name: makefile

Note that you need to indent the command section with tabs, not spaces.

The commands tell make what to do to create the target file. They are executed when we ask to create or update the target file and make concludes that the prerequisites have changed or the target file does not yet exist.

In most cases, you will want to process multiple files of the same type according to the same rules. One example of this would be creating HTML pages from markdown articles. This is done with a pattern rule. Pattern rules are one of the main things that set make apart from just scripting the whole thing in bash.

Pattern rules work by matching file extensions. For example, make knows how to create object files (.o) from C source files (.c) - by compiling them and passing the compiler the flag -c. make ships with quite a few pattern rules built in, with the most popular of those probably being the ones for compiling C and C++ code.

We can now simplify the makefile by omitting the commands for targets where make already knows what to do. Just don’t forget the blank line after the target - make needs that to know where one target ends and the next one begins.

In most cases, we can even omit the prerequisites: make’s internal rules assume that in order to, for example, build somefile.o via the C source -> Object file route, you’ll need somefile.c.

Be careful: When you supply your own command list, make will not search for a pattern rule to run for this target. It will instead rely on your command section to create the target file. It will however still run the “Do I need to update or create this” check and the subsequent search for matching pattern rules for your prerequisites.

We can - and will, in a later section - define our own pattern rules, which is one of the most powerful tools make offers.

For now, lets have a quick look at how to run a makefile.

Invoking make

Start make in a directory by running

make

If there is a makefile present, the first target make can find will be created (built). If there is no makefile (or one does exist, but contains no targets), make will complain about that.

To request a specific target, run

make target

One interesting feature right of the bat is that make is able to try to infer, using it’s built-in rules, what to do even without having any further information.

Try it by creating a test.c file in an empty folder and running make test. make will then try to build a target named test, recognize that it can do so by compiling test.c using a built-in pattern rule and do that. This even works over multiple steps of generated intermediate files.

Some special targets

In most makefiles in the wild, you’ll find a couple of special targets. Most commonly seen are probably

  • all to build the entire project
  • clean to remove all generated artefacts
  • install to install the generated files to the system
  • release or dist for preparing release-ready or distributable files (such as packages or tarballs)

Keep in mind that these targets are not required to be present in a makefile, but most build processes in the software world expect at least the first three to work.

These special targets obviously do not produce files named all or clean. Since make normally decides whether to run anything at all based on its view of whether the target would change, creating such a file would prevent make from doing anything at all for these targets.

To prevent that, GNU make allows us to mark these targets as phony in order to have them run anyway. We can do this by adding these targets as prerequisites to the internal .PHONY target like so:

.PHONY: all clean run

File name: makefile

The all target normally simply has the main executable in it’s prerequisite - sometimes with some additional commands to post-process the generated files. Since you’ll want the all target to be the one executed by default in most cases, it should also be the first in the file.

For the clean target, you’ll normally add a command list removing all generated files - which can be greatly simplified using variables, which we will learn about in the next section.

Variables and functions

Just as in most programming languages, makefiles can be made more readable by using variables.

make also imports it’s environment, allowing us to assign variables inside the makefile from the outside by passing them with the invocation or setting them in the external environment.

Basic operations

You can assign and reference variables with the syntax

NAME = value
FOO = baz
bar = $(FOO) frob

File name: makefile

References to variables are made in the form $(NAME) or ${NAME}, with both being exactly the same. When omitting the braces, make assumes that the variable name is one character long.

Variables can be appended to with the += operator. We can also set variables conditionally (only if they do not already have a value) with the ?= operator. This is extremely useful to allow the external environment to overwrite a variable while still providing a default value.

Finally, most implementations of make allow us to set a variable to the output of a shell command with the != operator. This comes at the cost of one subshell spawn per operation.

Passing arguments to the built-in pattern rules

Up to this point we have seen some of the built-in pattern rules in action, though they always ran the same basic commands. Of course build requirements are not always the same, and tools may require different flags for different things to be done.

For example, a software project might want to link against different libraries or with different optimizations for a release build than a debug build.

The built-in rules in make therefore include a number of well-known variables at important locations in their commands. We can set these to our liking, either from within the makefile or even from the outside.

This allows us to, for example, run the same makefile with different compilers by supplying make with the name of the compiler binary to execute. This is done by setting the CC (C compiler) environment variable like so

CC=clang make

Some of the most popular variables, which you most probably have seen if you’ve glimpsed into a makefile before are

  • $(CC) / $(CXX)
    • The C and C++ compiler binaries make uses to build the target.
  • $(CFLAGS) / $(CXXFLAGS)
    • Flags to pass to the compilers, such as optimizations and warning levels.
  • $(LDLIBS)
    • Libraries to link against.

Program variables

make also keeps some common programs in variables, primarily for the purposes of being able to overwrite them in special conditions.

The most important one of these is $(MAKE), which should be used when calling make from within a makefile (called a recursive make). It takes into considerations such things as command line arguments from the original invocation.

In a clean target, where your main job is to remove files, consider using the $(RM) variable instead of directly calling rm. $(RM) expands to rm -f (at least in GNU make), providing a safer rm call and the ability to, for example, substitute shred.

Variable functions

GNU make defines some very useful functions, most of which operate on the notion of words, them being space-separated strings of characters.

This makes it easy to work with variables containing lists of files.

Functions are called by using them like you would a variable of the same name, but with their arguments appended before the closing brace.

Some of the more interesting and useful functions are

  • $(wildcard pattern)
    • Returns a space-separated list of filenames matching the pattern, which may also be a relative path. This is obviously problematic with filenames containing spaces, and your best bet is to avoid them when using make. The pattern may contain the special match-all character *.
  • $(patsubst search pattern, replacement pattern, list of words)
    • Substitute all words in the list of words that match the search pattern to according to the replacement pattern. Both patterns use % as a wildcard character.
  • $(filter-out search pattern, list of words)
    • Returns a list of words with all words matching search pattern filtered out.
  • $(notdir list of words)
    • Returns list of words, but each entry is reduced to its basic filename (ie. the directory part, if there is any, is filtered out).
  • $(shell command)
    • Run a command in a subshell and capture the standard output, similar to the != operator. The shell used to run the command is determined by the $(SHELL) variable.

Advanced variable usage

Variables can be referenced in just about any context within a makefile. You could even construct the name of an executable to call from within a command list by concatenating multiple variables.

This enables us to use variables as targets or as prerequisites as we see fit, allowing for succinct and easy-to-understand constructs such as the following:

OBJECTS = $(patsubst %.c,%.o,$(wildcard *.c))
all: $(OBJECTS)

File name: makefile

The above makefile creates a list of all C source files in the directory, replaces their .c suffix with .o using the $(patsubst …) function and then goes on to use that list of files as prerequisite list for the all target. When running make, it will try to build the all target because it’s the first one defined. Since the target depends on a bunch of object files that may not yet exist (or may need updates), and make knows how to make these from C source files, all requested files are built.

This makes for an extremely powerful tool in combination with pattern rules, as we can automatically gather prerequisites, which are then implicitly used as new targets.

Substituting suffixes

In order to be compatible with other implementations, GNU make provides an alternative syntax for a limited call to the $(patsubst …) function called a substitution reference. This feature allows us to substitute suffixes of a list-of-words with another suffix.

The makefile from the previous example could be expressed in a style that would work on almost any implementation like so:

FILES != echo *.c
OBJS = $(FILES:.c=.o)
all: $(OBJS)

File name: makefile

Note the use of the != assignment operator instead of the $(wildcard …) function.

Target-specific variables

Another interesting feature leading to a great saving in lines of code are target-specific variables, which allow us to set variables to different values depending on the current target.

FOO = bar
target1: FOO = frob
target2: FOO += baz

File name: makefile

The previous example would have set $(FOO) to bar globally, to frob within target1 and to bar baz in target2.

As you can see, we can use any variable assignment operator conditionally, allowing us for example to create a release target with a different set of compiler flags by simply setting the $(CFLAGS) variable differently for that target.

Integration into external processes

Since most Linux systems ship with some kind of package manager, and many packages are created from projects that use make as their build system, a kind of consensus was reached regarding the use of a set of variables.

This mostly concerns the install targets of a project, as binary packages often consist of the results of a make install run packed in a common format.

The most important variables to keep in mind are

  • $(DESTDIR)
    • which should be empty by default and should never be set (only read) from inside the makefile. This is used by maintainers to inject a path that should be prepended to any file installed to the system.
  • $(PREFIX)
    • which your makefile should set to /usr/local or any path prefix where it makes sense to install things. This will be used by users that want to install to a different path on their system. Keep in mind to only set this if it has not been provided from the environment (by using the ?= operator).

Defining pattern rules

The syntax for writing a pattern rule is a lot like the normal target syntax. The major difference comes in using the the wildcard % in the target name. This wildcard will match what is called the stem of a target.

The wildcard may also be used in the prerequisite list, where it will be replaced by the stem to form implicit prerequisites. One of the more interesting things you can do is to extend the prerequisite list of a pattern rule with explicit prerequisites. These work just as if you would have added them to any target using the rule.

You can also simply overwrite any of the built-in rules by defining a rule with the same target and prerequisites.

Of course, since a pattern rule needs to be written to be independent of the actual file names it is called with, make provides some special variables within a pattern rule definition. These are called automatic variables.

Automatic variables

The names of automatic variables are all one more-or-less memorable character long, so that we don’t need braces to reference them. Since there are quite a few automatic variables, we’ll take a quick look at the most useful ones:

A pattern rule that converts markdown files to HTML using markdown might then look like:

%.htm : %.md
    markdown $^ > $@

File name: makefile

Suffix rules

Suffix rules are an older way of defining something that works a lot like pattern rules, though their syntax is a lot less expressive. When using GNU make, defining pattern rules as explained in this article is far preferrable, as we’ll see.

However, the POSIX standard and the BSD implementation only support suffix rules, so I’ll give a short introduction.

Suffix rules operate in a fashion similar to a basic pattern rule where the stem does not exclude any prefixes of the file name and thus contains everything up to the file’s suffix.

A suffix rule is then declared by concatenating the prerequisite suffix and the target suffix to form the suffix rule target name. In order to have make recognize this as a suffix rule and not just a weird target name, the suffixes used must be added as prerequisites to the special .SUFFIXES target.

For example, the previous example of converting markdown articles to HTML would look as follows

.SUFFIXES: .md .htm
.md.htm:
    markdown $^ > $@

File name: makefile

Advanced features such as are mentioned in the section on pattern rules are not possible with suffix rules.

The GNU documentation provides further in-depth information on how to define suffix rules and why not to do so.

Putting it all together

In the previous sections, you have learned some of the more complex things you can do with makefiles. The moderately complex, but still useful makefile for the articles on this website reads like this

The script generate_article.py implements a minimalistic template engine using index.htm as skeleton for embedding the HTML generated from the input files. The template’s presence as prerequisite of the pattern rule ensures that a change in the template file will also trigger regeneration of all article files. As the comments point out, you could add the script as well as the makefile itself to the prerequisites, too.

I think you’ll agree that while maybe not functionally complete, this still qualifies as readable - at least when you have mastered the basic syntax of make. And it works, too ;)


Troubleshooting

Nothing to be done for target

make tries to minimize it’s workload by only building targets whose prerequisites have changed. It does this by looking at the filesystem timestamps for all files involved. The response nothing to be done implies that make did not think anything would change in the target files if were to run the rules. This could have several reasons.

If target relies on a pattern rule

It the target uses a pattern rule, a prerequisite is automatically inferred even if the prerequisite list contains other files already. For an example, consider this makefile trying to compile a binary named test from foo.c:

test: foo.c

File name: makefile

The built-in pattern rule to compile binaries from C source will generate an additional prerequisite called test.c from the stem, which may not exist, causing make to ignore the pattern rule from consideration (since it does not match). make continues to search for rules that apply and, failing to find any, does nothing.

To solve this, use the stem of the primary source file (the first prerequisite) as the output prefix to have the built-in pattern rule match. Use additional prerequisites to trigger a rebuild when they are being changed.

If target does not use a pattern rule or contains a command list

If a target with an empty prerequisite list does not match an internal pattern rule (which infer some prerequisites automatically), the only source of information available to make is the target file itself. The timestamp on this file is then of course comparable to only itself and therefore found not to be in need of an update.

The solution to this is to properly record the prerequisites of the target file, or mark the target as phony.

If your target does contain a prerequisite list, your command list will only be run if make considers them to have changed since the last run (as determined by the file timestamps). Should make have determined that the target files already incorporate the latest changes in the prerequisites, it will do nothing.

To solve this, you can

In a very unlikely scenario, make not recognizing updated prerequisites could also indicate a problem with your hard drive or partition, so it may make sense to check your filesystem if you are encountering persistent problems.


Inconsistencies between implementations

GNU make

GNU make is arguably the most feature-complete implementation of make. Many of the advanced things that make life easier and makefiles less ugly started life as a GNU extension to the core make specification and have been gradually picked up by other implementations.

Conversely, GNU make also took up many features first seen in other implementations, not only in order to stay compatible to a wide range of other makefiles, but also to extend what is possible with a makefile.

Regrettably, this also means that if you want your makefile to stay compatible to the widest possible range of implementations, you will have to forgo using some functionality. Non-portable functionality mentioned in the article should have a note next to it. Please remind me to add one if you find functionality that does not work with some implementations.

Another possibility of supporting as many implementations as possible is to create multiple makefiles for your project. GNU make will first search for a file named GNUmakefile, in order to support exactly this scenario.

POSIX / BSD make

As most implementations choose to support at least some extensions to the basic protocol, there is no pure implementation of only POSIX-mandated functionality (unless strict POSIX compliance is enforced by defining the .POSIX target). I will therefore focus this comparison on the differences to BSD make, which is one of the more restrictive ones.

As noted in the article, POSIX does not mandate variable functions, as useful as they are, and BSD does not implement them. Most functionality provided by this feature can potentially be replaced by calls to the shell (using the != operator) at the cost of additional subshell executions.

Most prominently, the $(wildcard …) function may be replaced by a call to either of the following commands

find . -type f -name '*.suff'
echo *.suff

The $(patsubst …) function’s power is still available in a somewhat limited way by using substitution references, which are supported both in the GNU and BSD implementations.

Another major source of incompatibilities in makefiles is the amount and names of supported automatic variables within the pattern/suffix rules. While all implementations agree on the use of $< as the name of the first prerequisite, there are differing opinions on how to read the entire prerequisite list. Within a GNU makefile, we can use $^ for this purpose, while BSD uses $>. The POSIX standard does not actually define any variable for this purpose. Since neither defines the other, a construct such as $^$> would work in both implementations, though it does nothing for readability.

If you don’t want to stay within the limitations of core make, the BSD repositories provide a port of GNU make under the name of gmake.

Windows

The build process under Windows is generally a bit more involved than under Unix-like systems, as Windows users are usually accustomed to a much higher level of tooling integration (ie. expect their projects to be able to build using Visual Studio).

There is a port of GNUmake for Windows, though when writing your own command lists they most probably will not work without some modifications as most core tools are named differently on Windows.

Writing a special windows target might be a solution for most problems.

Alternatively, installing the Cygwin projects port of the core linux tools should also do the trick.


Wrapping up

You’ve made it to the end of this article. Thanks for reading! I’ve aggregated some links to further details, if this tickled your interest.

Links

Once you get accustomed to the terminology, the GNU make manual is a very good read. In addition to the sections already linked from their context in the article, the following parts are a good source of interesting information:

The POSIX make documentation contains information on the common denominator to be supported.

The manpage of BSD make provides information on the peculiarities of that specific implementation.

As for an in-depth discussion of the differences between GNU make and others, the GNU documentation has pages on features and their origins (which includes features only GNU supports), as well as features missing in comparison to other implementations.

Feedback

If you liked this, found it useful, found an error and have a correction, or simply want to say hi - drop me an email to fjs@fabianstumpf.de!

Errata

The reddit comment thread for this article produced some interesting feedback. I’ll address a few points raised here.

  • Examples are not copy-pastable because they are indented with spaces instead of the tabs make expects.
    • Completely valid complaint and an oversight on my part. It seems the person developing the markdown compiler I use just hates tabs with all their heart, at least according to the source comments. I’ll try to fix this in the next few days.
  • The pattern rule in the example makefile does not depend on generate_article.py and makefile itself.
    • You could do that if you change your generation script frequently. I don’t though. I hope that the article explained how to add prerequisites quite thoroughly, so if you don’t want to have to run a make clean all yourself once in a blue moon, by all means add it. For software projects, where you might change up things frequently, this is actually a good thing to consider.
  • The makefile can be used to execute arbitrary commands by running it on files with specially crafted filenames (eg. ;echo HelloWorld;.md).
    • That’s true - and due to the nature of variable handling in makefiles, very hard to prevent (short of hardcoding your files, which it was kind of the point not to). I would however argue that if you have a kind of control where you can create files in the target directory, you might as well just edit the makefile itself to do something nefarious. I acknowledge that this is a flaw in make though (not properly separating data and code).