Make files not war
This article tries to explain make
and it’s makefiles in slightly different terms than the
original documentation or even most tutorials. It took me a long time to understand
how to write short, simple and understandable makefiles that work, and I hope this article
may help people by providing another view.
This article assumes a Linux system and GNU make
, though I will discuss some of the differences
concerning Windows and BSD.
Table of contents
- Prelude
- Writing makefiles
- Troubleshooting
- Implementation differences
- Wrapping up
Prelude
But all makefiles I’ve seen are ugly messes!
As most projects just copy theirs from another project or a “one-size-fits all” makefile in order to get it to work - or even worse, generate one - that comes as no surprise.
These heirlooms are often huge, hard to read and near-impossible to debug, thereby contributing
a great deal to make
’s reputation as a bad build system in some circles and scaring away
people interested in learning how things work.
The biggest source of trouble and complexities is the variety of implementations of the basic
makefile convention, resulting in different feature sets being supported across systems.
This can be circumvented by concentrating on the GNU implementation of make
, which is supported
on pretty much any system and provides arguably the widest range of features.
Other than that, I have been very happy with a broadly supported, minimally intrusive build system that has stood the tests of time and is still considered by many to be THE major build automation tool there is.
Why use make
- It works
- Easy to set up in new and existing projects
- Your system probably ships with a version of
make
- or it can be easily installed
- It’s tiny (and doesn’t depend on much)
- makefiles can be short, succinct and beautiful
- Does not need mysterious working or resource folders
- No opaque magic performed
When to use make
Most projects, be they software development, writing a book or publishing a blog, at some point need to generate “end products” (of some kind) from manually created raw files.
The point of a build (automation) system is to encode how to do that for one item, and apply that rule to one or more objects of the same type, updating only what’s changed.
One non-software example would be rendering changed book chapters to PDFs and subsequently concatenating them to form the final book.
Of course, this can be (and often is) done via a custom script, written in any one programming
language, but make
’s powerful rule-based system greatly simplifies the process of doing that.
Sometimes it may be useful to run some tools (for example, a test suite or a spell checker) against the complete project, and the built-in bash-like scripting support allows to do that, too.
In fact, the articles on this website are automatically generated from markdown files
using make
- everytime I push changes to my web server, it runs make
against the
directory containing the raw articles using this makefile, which we will
come back to in the course of this article.
Writing makefiles
Let’s get to discussing how make
works by starting with some basic terms and a simple
makefile.
Starting out simple
Starting out, let’s create an empty file named makefile (with capital or lowercase M,
make
accepts both).
The content of a basic makefile can be thought of as rules describing:
- If any of these files change
- This file should, too
In make
terminology, the second item is called a target - a file to be made by somehow
processing some or all of the prerequisites, which form the first item.
The aforementioned rules for processing the prerequisites can be arbitrarily complex, and can consist of calling out to other tools such as compilers, parsers, etc.
The basic syntax for defining a target is
target: prereq1 prereq2 ...
command1
command2
...
<blank line>
File name: makefile
Note that you need to indent the command section with tabs, not spaces.
The commands tell make
what to do to create the target file. They are executed when we ask to
create or update the target file and make
concludes that the prerequisites have changed or
the target file does not yet exist.
In most cases, you will want to process multiple files of the same type according to the same rules.
One example of this would be creating HTML pages from markdown articles. This is done with a pattern rule.
Pattern rules are one of the main things that set make
apart from just scripting the whole thing in bash.
Pattern rules work by matching file extensions. For example, make
knows how to create object files
(.o) from C source files (.c) - by compiling them and passing the compiler the flag -c.
make
ships with quite a few pattern rules built in,
with the most popular of those probably being the ones for compiling C and C++ code.
We can now simplify the makefile by omitting the commands for targets where make
already knows what
to do. Just don’t forget the blank line after the target - make
needs that to know where one target ends and
the next one begins.
In most cases, we can even omit the prerequisites: make
’s internal rules assume that in order to, for example,
build somefile.o via the C source -> Object file
route, you’ll need somefile.c.
Be careful: When you supply your own command list, make
will not search for a pattern rule
to run for this target. It will instead rely on your command section to create the target file.
It will however still run the “Do I need to update or create this” check and the subsequent search for
matching pattern rules for your prerequisites.
We can - and will, in a later section - define our own pattern rules, which is one of the most powerful
tools make
offers.
For now, lets have a quick look at how to run a makefile.
Invoking make
Start make
in a directory by running
make
If there is a makefile present, the first target make
can find will be created (built).
If there is no makefile (or one does exist, but contains no targets), make
will
complain about that.
To request a specific target, run
make target
One interesting feature right of the bat is that make
is able to try to infer,
using it’s built-in rules, what to do even without having any further information.
Try it by creating a test.c file in an empty folder and running make test
.
make
will then try to build a target named test, recognize that it can do so by compiling
test.c using a built-in pattern rule and do that. This even works over multiple steps of
generated intermediate files.
Some special targets
In most makefiles in the wild, you’ll find a couple of special targets. Most commonly seen are probably
all
to build the entire projectclean
to remove all generated artefactsinstall
to install the generated files to the systemrelease
ordist
for preparing release-ready or distributable files (such as packages or tarballs)
Keep in mind that these targets are not required to be present in a makefile, but most build processes in the software world expect at least the first three to work.
These special targets obviously do not produce files named all or clean.
Since make
normally decides whether to run anything at all based on its view of whether the target
would change, creating such a file would prevent make
from doing anything at all for these targets.
To prevent that, GNU make
allows us to mark these targets as phony in order to have them run anyway.
We can do this by adding these targets as prerequisites to the internal .PHONY
target like so:
.PHONY: all clean run
File name: makefile
The all
target normally simply has the main executable in it’s prerequisite - sometimes with
some additional commands to post-process the generated files. Since you’ll want the all
target to
be the one executed by default in most cases, it should also be the first in the file.
For the clean
target, you’ll normally add a command list removing all generated files - which
can be greatly simplified using variables, which we will learn about in the next section.
Variables and functions
Just as in most programming languages, makefiles can be made more readable by using variables.
make
also imports it’s environment, allowing us to assign variables inside the makefile from the
outside by passing them with the invocation or setting them in the external environment.
Basic operations
You can assign and reference variables with the syntax
NAME = value
FOO = baz
bar = $(FOO) frob
File name: makefile
References to variables are made in the form $(NAME) or ${NAME}, with both being exactly
the same. When omitting the braces, make
assumes that the variable name is one character long.
Variables can be appended to with the +=
operator. We can also set variables conditionally (only if
they do not already have a value) with the ?=
operator. This is extremely useful to allow the external
environment to overwrite a variable while still providing a default value.
Finally, most implementations of make
allow us to set a variable to the output of a shell command with
the !=
operator. This comes at the cost of one subshell spawn per operation.
Passing arguments to the built-in pattern rules
Up to this point we have seen some of the built-in pattern rules in action, though they always ran the same basic commands. Of course build requirements are not always the same, and tools may require different flags for different things to be done.
For example, a software project might want to link against different libraries or with different optimizations for a release build than a debug build.
The built-in rules in make
therefore include a number of well-known variables at important locations in
their commands. We can set these to our liking, either from within the makefile or even from the outside.
This allows us to, for example, run the same makefile with different compilers by supplying make
with the name of the compiler binary to execute. This is done by setting the CC
(C compiler) environment
variable like so
CC=clang make
Some of the most popular variables, which you most probably have seen if you’ve glimpsed into a makefile before are
- $(CC) / $(CXX)
- The C and C++ compiler binaries make uses to build the target.
- $(CFLAGS) / $(CXXFLAGS)
- Flags to pass to the compilers, such as optimizations and warning levels.
- $(LDLIBS)
- Libraries to link against.
Program variables
make
also keeps some common programs in variables, primarily for the purposes of being able
to overwrite them in special conditions.
The most important one of these is $(MAKE), which should be used when calling make
from within
a makefile (called a recursive make). It takes into considerations such things as command line
arguments from the original invocation.
In a clean
target, where your main job is to remove files, consider using the $(RM) variable
instead of directly calling rm
. $(RM) expands to rm -f
(at least in GNU make), providing
a safer rm
call and the ability to, for example, substitute shred
.
Variable functions
GNU make
defines some very useful functions,
most of which operate on the notion of words, them being space-separated strings of characters.
This makes it easy to work with variables containing lists of files.
Functions are called by using them like you would a variable of the same name, but with their arguments appended before the closing brace.
Some of the more interesting and useful functions are
- $(wildcard pattern)
- Returns a space-separated list of filenames matching the pattern, which may also be a relative path. This is obviously problematic with filenames containing spaces, and your best bet is to avoid them when using make. The pattern may contain the special match-all character *.
- $(patsubst search pattern, replacement pattern, list of words)
- Substitute all words in the list of words that match the search pattern to according to the replacement pattern. Both patterns use % as a wildcard character.
- $(filter-out search pattern, list of words)
- Returns a list of words with all words matching search pattern filtered out.
- $(notdir list of words)
- Returns list of words, but each entry is reduced to its basic filename (ie. the directory part, if there is any, is filtered out).
- $(shell command)
- Run a command in a subshell and capture the standard output, similar to the
!=
operator. The shell used to run the command is determined by the $(SHELL) variable.
- Run a command in a subshell and capture the standard output, similar to the
Advanced variable usage
Variables can be referenced in just about any context within a makefile. You could even construct the name of an executable to call from within a command list by concatenating multiple variables.
This enables us to use variables as targets or as prerequisites as we see fit, allowing for succinct and easy-to-understand constructs such as the following:
OBJECTS = $(patsubst %.c,%.o,$(wildcard *.c))
all: $(OBJECTS)
File name: makefile
The above makefile creates a list of all C source files in the directory, replaces their .c suffix with
.o using the $(patsubst …) function and then goes on to use that list of files as prerequisite
list for the all
target. When running make
, it will try to build the all
target because it’s the first one
defined. Since the target depends on a bunch of object files that may not yet exist (or may need updates), and
make
knows how to make these from C source files, all requested files are built.
This makes for an extremely powerful tool in combination with pattern rules, as we can automatically gather prerequisites, which are then implicitly used as new targets.
Substituting suffixes
In order to be compatible with other implementations, GNU make
provides an alternative syntax
for a limited call to the $(patsubst …) function called a substitution reference. This feature allows
us to substitute suffixes of a list-of-words with another suffix.
The makefile from the previous example could be expressed in a style that would work on almost any implementation like so:
FILES != echo *.c
OBJS = $(FILES:.c=.o)
all: $(OBJS)
File name: makefile
Note the use of the !=
assignment operator instead of the $(wildcard …) function.
Target-specific variables
Another interesting feature leading to a great saving in lines of code are target-specific variables, which allow us to set variables to different values depending on the current target.
FOO = bar
target1: FOO = frob
target2: FOO += baz
File name: makefile
The previous example would have set $(FOO) to bar globally, to frob within target1
and to
bar baz in target2
.
As you can see, we can use any variable assignment operator conditionally, allowing us for example to
create a release
target with a different set of compiler flags by simply setting the $(CFLAGS)
variable differently for that target.
Integration into external processes
Since most Linux systems ship with some kind of package manager, and many packages are created from projects
that use make
as their build system, a
kind of consensus was reached regarding the use of a
set of variables.
This mostly concerns the install
targets of a project, as binary packages often consist of the results
of a make install
run packed in a common format.
The most important variables to keep in mind are
- $(DESTDIR)
- which should be empty by default and should never be set (only read) from inside the makefile. This is used by maintainers to inject a path that should be prepended to any file installed to the system.
- $(PREFIX)
- which your makefile should set to
/usr/local
or any path prefix where it makes sense to install things. This will be used by users that want to install to a different path on their system. Keep in mind to only set this if it has not been provided from the environment (by using the?=
operator).
- which your makefile should set to
Defining pattern rules
The syntax for writing a pattern rule is a lot like the normal target syntax. The major difference comes in using the the wildcard % in the target name. This wildcard will match what is called the stem of a target.
The wildcard may also be used in the prerequisite list, where it will be replaced by the stem to form implicit prerequisites. One of the more interesting things you can do is to extend the prerequisite list of a pattern rule with explicit prerequisites. These work just as if you would have added them to any target using the rule.
You can also simply overwrite any of the built-in rules by defining a rule with the same target and prerequisites.
Of course, since a pattern rule needs to be written to be independent of the actual file names it is called with,
make
provides some special variables within a pattern rule definition. These are called automatic variables.
Automatic variables
The names of automatic variables are all one more-or-less memorable character long, so that we don’t need braces to reference them. Since there are quite a few automatic variables, we’ll take a quick look at the most useful ones:
- $@: The complete target name
- $<: The name of the first prerequisite (this may also be the one implicitly generated by a pattern match)
- $^: Space-separated list of all prerequisites (GNU extension)
- $>: Space-separated list of all prerequisites (BSD extension)
A pattern rule that converts markdown files to HTML using markdown
might then look like:
%.htm : %.md
markdown $^ > $@
File name: makefile
Suffix rules
Suffix rules are an older way of defining something that works a lot like pattern rules, though their syntax
is a lot less expressive. When using GNU make
, defining pattern rules as explained in this article is far
preferrable, as we’ll see.
However, the POSIX standard and the BSD implementation only support suffix rules, so I’ll give a short introduction.
Suffix rules operate in a fashion similar to a basic pattern rule where the stem does not exclude any prefixes of the file name and thus contains everything up to the file’s suffix.
A suffix rule is then declared by concatenating the prerequisite suffix and the target suffix to form
the suffix rule target name. In order to have make
recognize this as a suffix rule and not just a weird target
name, the suffixes used must be added as prerequisites to the special .SUFFIXES target.
For example, the previous example of converting markdown articles to HTML would look as follows
.SUFFIXES: .md .htm
.md.htm:
markdown $^ > $@
File name: makefile
Advanced features such as are mentioned in the section on pattern rules are not possible with suffix rules.
The GNU documentation provides further in-depth information on how to define suffix rules and why not to do so.
Putting it all together
In the previous sections, you have learned some of the more complex things you can do with makefiles. The moderately complex, but still useful makefile for the articles on this website reads like this
.PHONY: all clean
ARTICLES = $(patsubst %.md,%.htm,$(wildcard *.md))
%.htm : %.md index.htm
./generate_article.py $< > $@
all: $(ARTICLES)
clean:
$(RM) $(ARTICLES)
File name: makefile
The script generate_article.py
implements a minimalistic template engine using index.htm as skeleton for embedding the
HTML generated from the input files. The template’s presence as prerequisite of the pattern rule ensures that a change in
the template file will also trigger regeneration of all article files. As the comments point out, you could add
the script as well as the makefile itself to the prerequisites, too.
I think you’ll agree that while maybe not functionally complete, this still qualifies as readable - at least when you have
mastered the basic syntax of make
. And it works, too ;)
Troubleshooting
Nothing to be done for target
make
tries to minimize it’s workload by only building targets whose prerequisites have changed. It does this by
looking at the filesystem timestamps for all files involved. The response nothing to be done implies that make
did not think anything would change in the target files if were to run the rules. This could have several reasons.
If target
relies on a pattern rule
It the target uses a pattern rule, a prerequisite is automatically inferred even if the prerequisite list contains other files already. For an example, consider this makefile trying to compile a binary named test from foo.c:
test: foo.c
File name: makefile
The built-in pattern rule to compile binaries from C source will generate an additional prerequisite called test.c from
the stem, which may not exist, causing make
to ignore the pattern rule from consideration (since it does not match).
make
continues to search for rules that apply and, failing to find any, does nothing.
To solve this, use the stem of the primary source file (the first prerequisite) as the output prefix to have the built-in pattern rule match. Use additional prerequisites to trigger a rebuild when they are being changed.
If target
does not use a pattern rule or contains a command list
If a target with an empty prerequisite list does not match an internal pattern rule (which infer some prerequisites
automatically), the only source of information available to make
is the target file itself. The timestamp on this
file is then of course comparable to only itself and therefore found not to be in need of an update.
The solution to this is to properly record the prerequisites of the target file, or mark the target as phony.
If your target does contain a prerequisite list, your command list will only be run if make
considers them to have
changed since the last run (as determined by the file timestamps). Should make
have determined that the target files
already incorporate the latest changes in the prerequisites, it will do nothing.
To solve this, you can
- add any files that should trigger a rule run to the prerequisites
- if necessary, record the prerequisites of your prerequisites
- delete any generated prerequisites (eg. by running
make clean
) to forcemake
to build those again - run
make
with the -B flag, forcing it to rebuild the requested target unconditionally
In a very unlikely scenario, make
not recognizing updated prerequisites could also indicate a problem with your
hard drive or partition, so it may make sense to check your filesystem if you are encountering persistent problems.
Inconsistencies between implementations
GNU make
GNU make
is arguably the most feature-complete implementation of make. Many of the advanced things that
make life easier and makefiles less ugly started life as a GNU extension to the core make
specification and
have been gradually picked up by other implementations.
Conversely, GNU make
also took up many features first seen in other implementations, not only in order to
stay compatible to a wide range of other makefiles, but also to extend what is possible with a makefile.
Regrettably, this also means that if you want your makefile to stay compatible to the widest possible range of implementations, you will have to forgo using some functionality. Non-portable functionality mentioned in the article should have a note next to it. Please remind me to add one if you find functionality that does not work with some implementations.
Another possibility of supporting as many implementations as possible is to create multiple makefiles
for your project. GNU make
will first search for a file named GNUmakefile, in order to support exactly
this scenario.
POSIX / BSD make
As most implementations choose to support at least some extensions to the basic protocol, there is no pure
implementation of only POSIX-mandated functionality (unless strict POSIX compliance is enforced by defining
the .POSIX target). I will therefore focus this comparison on the differences to BSD make
, which is
one of the more restrictive ones.
As noted in the article, POSIX does not mandate variable functions, as useful as they are, and BSD does
not implement them. Most functionality provided by this feature can potentially be replaced by calls to the
shell (using the !=
operator) at the cost of additional subshell executions.
Most prominently, the $(wildcard …) function may be replaced by a call to either of the following commands
find . -type f -name '*.suff'
echo *.suff
The $(patsubst …) function’s power is still available in a somewhat limited way by using substitution references, which are supported both in the GNU and BSD implementations.
Another major source of incompatibilities in makefiles is the amount and names of supported automatic variables within the pattern/suffix rules. While all implementations agree on the use of $< as the name of the first prerequisite, there are differing opinions on how to read the entire prerequisite list. Within a GNU makefile, we can use $^ for this purpose, while BSD uses $>. The POSIX standard does not actually define any variable for this purpose. Since neither defines the other, a construct such as $^$> would work in both implementations, though it does nothing for readability.
If you don’t want to stay within the limitations of core make
, the BSD repositories provide a port of GNU
make
under the name of gmake
.
Windows
The build process under Windows is generally a bit more involved than under Unix-like systems, as Windows users are usually accustomed to a much higher level of tooling integration (ie. expect their projects to be able to build using Visual Studio).
There is a port of GNUmake for Windows, though when writing your own command lists they most probably will not work without some modifications as most core tools are named differently on Windows.
Writing a special windows
target might be a solution for most problems.
Alternatively, installing the Cygwin projects port of the core linux tools should also do the trick.
Wrapping up
You’ve made it to the end of this article. Thanks for reading! I’ve aggregated some links to further details, if this tickled your interest.
Links
Once you get accustomed to the terminology, the GNU make manual is a very good read. In addition to the sections already linked from their context in the article, the following parts are a good source of interesting information:
- Chains of implicit rules, which
explains exactly how and when
make
will build (and delete) intermediate files for implicit rules. - List of variables used by built-in rules
- List of automatic variables in GNU
make
- List of built-in functions
- The complete documentation on pattern rules
- Makefile conventions, a list of things most makefiles should do in a similar fashion.
The POSIX make documentation contains information on the common denominator to be supported.
The manpage of BSD make
provides information on the peculiarities
of that specific implementation.
As for an in-depth discussion of the differences between GNU make
and others, the GNU documentation
has pages on features and their origins (which includes
features only GNU supports), as well as features missing in comparison to other implementations.
Feedback
If you liked this, found it useful, found an error and have a correction, or simply want to say hi - drop me an email to fjs@fabianstumpf.de!
Errata
The reddit comment thread for this article produced some interesting feedback. I’ll address a few points raised here.
- Examples are not copy-pastable because they are indented with spaces instead of the tabs
make
expects.- Completely valid complaint and an oversight on my part. It seems the person developing the markdown compiler I use just hates tabs with all their heart, at least according to the source comments. I’ll try to fix this in the next few days.
- The pattern rule in the example makefile does not depend on generate_article.py and makefile itself.
- You could do that if you change your generation script frequently. I don’t though. I hope that the article explained
how to add prerequisites quite thoroughly, so if you don’t want to have to run a
make clean all
yourself once in a blue moon, by all means add it. For software projects, where you might change up things frequently, this is actually a good thing to consider.
- You could do that if you change your generation script frequently. I don’t though. I hope that the article explained
how to add prerequisites quite thoroughly, so if you don’t want to have to run a
- The makefile can be used to execute arbitrary commands by running it on files with specially crafted filenames (eg.
;echo HelloWorld;.md
).- That’s true - and due to the nature of variable handling in makefiles, very hard to prevent (short of hardcoding your files, which it
was kind of the point not to). I would however argue that if you have a kind of control where you can create files in the target
directory, you might as well just edit the makefile itself to do something nefarious. I acknowledge that this is a flaw in
make
though (not properly separating data and code).
- That’s true - and due to the nature of variable handling in makefiles, very hard to prevent (short of hardcoding your files, which it
was kind of the point not to). I would however argue that if you have a kind of control where you can create files in the target
directory, you might as well just edit the makefile itself to do something nefarious. I acknowledge that this is a flaw in