For comments on this page, contact: lboggs@mail.bordoon.com

Portable GNU MAKE Example

Would you like to write code that is portable to both linux and unix and you are primarly a linux/unix developer?

If you haven't already figured out to use GNU MAKE in such a way that you can easily write makefiles that work in both environments, you can use examples documented here.

AND! These examples include automatic makefile dependency generation!

Yes, you could always do this if you were a makefile guru and if you had a copy of a good "depend" program, but the examples provided here show how do all this using only the base install of the os, your compiler and GNU make. It turns out that both gcc and the Visual Studio compiler have options that let you produce either makefile dependency files direcly or (in the case of Visual Studio) after processing with a simple script.

Most people have 3 environments to deal with:

Linux
Windows (native)
Cygwin (Windows plus bash and Linxu tools)

The GNU MAKE examples documented here work in all three environments. You can develop on one platform, copy your code to the other an pick right up like nothing was wrong.

You can get the example and the gory details at the following link:
Here

Stop! The instructions for testing your installation of the examples are meant to work "out of the box" in the following environments:

Linux with gcc/g++ installed and in your path
Windows with GNU MAKE and the Visual Studio compiler in your path
Cygwin with the Visual Studio compiler in your path
If you want to compile with Cygwin and gcc, or on some other unix flavor, you'll have to do some tweaking.

To compile the examples, extract the .zip file to some parent directory and you will end up with a child named "example". CD into that directory and run "make". You should see something like the following (on linux anyway).

If the compile looks successful, then you can run "make tests" and see something that looks liek this:

For detailed instructions on using the GNU MAKE examples in this distribution, goto examples.

For a very high level view of how the makefiles work and what the trade-offs for using this approach are, continue reading:

Overview

------------ Introduction ------------ This directory contains an easy to use and understand but full-featured EXAMPLE of using GNU make to build a software project that is portable to Linux, Windows, and the Cygwin environment. This file describes the "big picture" of why this directory was created. For explicit examples of using the make rules documented here, see file MAKEFILES_FOR_SMARTIES.txt You can get GNU MAKE for Win32 on www.sourceforge.net. Look in the win32 section if you have trouble finding it. Other than the make.rules file and the *.bat files in this directory, there are no other tools you need to use examples in this directory. GNU MAKE can be easy! --------------------- GNU MAKE is a powerful and highly flexible tool that can be used by any one in any supported environment to build any project that has deterministic build rules. To accomplish this, it must be flexible and very powerful which leads to difficulty in explaining the tool. That is, when you look at it's documentation, you see hundreds of make features -- each of which is absolutely essential to someone in some environment with some problem. The totality of which, though, can add to to high learning curve. This directory seeks provide easy to understand and easy to use GNU MAKE examples that hide the complexity without sacrificing the power of GNU MAKE. This is accomplished by separating the declarative aspects of Makefile definition from the algorithms that make the declarations work. Declarative Makefiles --------------------- The portable examples found in this directory structure allow you to limit your Makefiles to nothing more than a set of assertions: 1. I have these source files 2. I want these libraries and executables built 3. I want these test programs run and their outputs checked. 4. AND DON'T ASK ME NO STUPID QUESTIONS! Despite this simplicity, the examples contained here give you automatic make dependency generation and a high degree of configurability with a minimum of knowledge and typing. Here's an example Makefile, it can be found in the src/program subdirectory: # FILE = src/program2/Makefile include ../../make.rules OUTPUTPROGRAM:= $(BINDIR)program2.exe all:: $(OUTPUTPROGRAM) allObjects:= allObjects+= $(patsubst %.c,%.$(OBJ),$(wildcard *.c)) allObjects+= $(patsubst %.cpp,%.$(OBJ),$(wildcard *.cpp)) allObjects+= $(patsubst %.cxx,%.$(OBJ),$(wildcard *.cxx)) $(OUTPUTPROGRAM): $(allObjects) \ $(LIBDIR)lib1.$(STATICLIB) \ $(LIBDIR)lib2.$(STATICLIB) $(LINK.cxx) This example is almost 100% declarative in nature: 1. it declares what rules should be used for implementing the declarations found in this file (../../make.rules) 2. it declares that a program, not stored in the current directory (../../bin/program2.exe) should be built from source files in the current directory. 3. it declares that the program should be constructed from all the source files in the current directory (after compilation into objects) and 2 libraries whose source code is defined in some other directory. The only "procedural" aspect to the above is the linker recipe: $(LINK.cxx) This particular line selects among the many possible build types the writer's desire to produce a C++ program. Note that this linker invocation does not require that the objects and libraries appear on the same line. Instead, the definition of the macro, LINK.cpp is very complex and it knows how to get the *.$(OBJ) and the two library names that are defined on the target predecessor line. Automatic dependency generation is also happening behind the scenes. Because of the definitions in ../../make.rule, whenever C or C++ code is is compiled, the makefile dependencies on header files used by object files are automatically created for GNU MAKE's use. At What Cost? ------------- Nothing is free, but the costs of obtaining the benefits of declarative only (well mostly) makefiles are fairly small. You have to do the following: 1. You must structure your project hierarchy such that the make.rules file's built in logic will know how to find things and help you recursively visit all interesting subdirectories for the purpose of compilation and testing. 2. YOU MUST NOT USE SPACES IN FILENAMES - at least not in the names of subdirectories or source files that GNU MAKE will be using for targets and target selection. You CAN use spaces in the names of the project top level directory, but it is safer not to use spaces in any of the the project paths. Data files that are not actual targets won't cause trouble, but sources and generated objects MUST not have spaces in their names. 3. You EITHER have to limit yourself to one directory building EXACTLY ONE target file (or collection of files that add up to one thing). OR you have to learn a little GNU MAKE string manipulation magic. For big projects, you SHOULD limit yourself to the one major target per directory approach. This gives you the power to simply copy the example makefiles in this directory (see src/lib1/Makefile or src/program1/Makefile) to your working directory filled with source and then make only a single customization: change the name of the target to match your actual library name or program name. If you choose to learn a little GNU MAKE string magic then you can mix anything kinds of things together in any directory. The make.rules file makes heavy use of make magic. You can use it for examples. 3. If you want portablity between Linux and Windows, you must restrict your Makefiles to using MACROS that define the names of commands that do ALMOST EVERYTHING. The following table relates commands you might want to type use at the command line to the macros defined in make.rules that replace them. See MAKEFILES_FOR_SMARTIES.txt.

Examples

This document describes how to use the make rules defines in this directory as examples from which you can easily create powerful makefiles which allow you write code on Linux and Windows interchangeably. You can even develop different parts of the project on both at the same time (if you are careful). See file README.txt for a general introduction of why this directory exists, why it works and what the trade offs are for using the makefile strategy, described there, are. You can get GNU MAKE for Win32 on www.sourceforge.net. Look in the win32 section if you have trouble finding it. It alread comes with Linux, and if you want to use this approach on solaris, hpux, or aix, you should be able to download it from gnu.org. Note that on those platforms, the built in make program WILL NOT WORK with the examples found in this directory. Other than a supported compiler, the make.rules file, and the *.bat files in this directory, there are no other tools you need to use examples in this directory. Caveat: only if you have the zip executable installed, can you use the backup: target found in the make.rules file. If you are too lazy to read all this, see the examples in subdirectories: ./Makefile -- Defines a top level directory makefile that recurses to child directories. You could put program build logic here too, but for demonstration purposes, there isn't any. The top level makefile of many projects also includes product installation make targets as well. None are shown here. Not immediately obvious, however, there is a backup make target that will use zip to export the source code to a .zip file. bin/Makefile -- These makefiles exist only so that a recursive lib/Makefile make clean will delete the binaries pushed into the bin and lib directories but the Makefiles in the src subtree. src/Makefile -- Another recursive only makefile. It exists only to include it's subdirectories (lib1, lib2, program1, and program2) into the build. src/lib1 -- Source code and Makefile that builds lib/lib1.$(STATICLIB). The Makefile shows you how to mix source code languages in one Makefile. You can use the makefile found in that directory as a template for building any library in any directory. You only have to change the name of the library being built! Just copy and tweak. Note: to do this, you must have only the source code for one library in the directory to which you copy src/lib1/Makefile. src/lib2 -- A second but more trivial library building directory. It exists only so we can have two libraries for demonstrative purposes. src/program1 -- Filled with source code and a Makeile for a program that will be linked with lib1 and lib2 to produce bin/program1.exe. src/program2 -- Ditto, so we can have a second program. mixed -- A directory whose makefile builds both programs and libraries. Use only for tiny projects because you have know more GNU MAKE syntax than the makefiles in the src subtree. Note how much more typing it takes to create this kind of Makefile: you can't just copy and tweak, you actually have to WRITE the Makefile. tests -- A directory full of test expected values files and the make rules which invoke programs in other directories and compares the outputs against the expected values For more details, see below and comments in the above named Makefiles. Basics ------ The file, make.rules, defines all the compilation logic needed to build makefiles portable to * Linux (with gcc) * Windows (with either Visual Studio C++) * Windows/Cygwin (with Visual Studio C++) The rules file detects your operating system and selects the default compiler for that operating system. You can also create a file, compiler_selection.rules, that superseeds the selection logic. The rules file also reads another optional file, project.config, that lets you add additional compiler options, macro definitions, rules, and commonly used library lists so that you do not have to actually modify make.rules itself. Warning: parallel builds are not supported. Don't use GNU MAKE's -j option. The make.rules file defines compilation logic, including automatic header file dependency generation for the following file types: *.c *.cxx *.cpp Out of the box, it makes the assumption that on windows, even with cywin, you want to use the Microsoft Visual Studio Compiler. You must then ru Structuring Your Code --------------------- The make.rules file hides's a lot of logic from your makefiles but at the cost of requiring you to structure your source code directories like this: 1. You must have a single root directory that serves as the root of a tree. All source code must appear either at the root or in a subdirectory of the root. YOU MUST NOT HAVE SPACES in the names of source files or any file that is target or dependency in a Makefile. You can over come this, in the main by using relative paths to do things: ../../../fred.c The project top level directory can have spaces in it's name but it is really painful for any other file in any makefile to have spaces. (You can use them if you ABSOLUTELY have to but you must escape the spaces. The $(subst ,\ ,$(v)) macro can help with this). 2. In this root directory, you must put the make.rules file and the windows*.bat files. You many optionally choose to define a project.config file in this root directory that defines common compiler options, libraries etc. See comments in project.config. 3. Your Makefiles should begin with an inclusion the make.rules file from your project. The pathname should be directory relative like one of the following: include make.rules # in the root directory include ../make.rules # in the first level subdirectories include ../...../make.rules # in deeper subdirectories 4. If you need to process subdirectories then you must declare the subdirectory list before you include the make.rules file: SUBDIRS:= src programs tests include ../make.rules You can mix subdirectory processing with normal target building. 5. You must hook into the all:: target to enable the compilation of most of your object, libraries, and executables, and into the test:: target to define a set of tests to be run: include ../make.rules allObjects:= allObjects+= $(patsubst %.c,%.$(OBJ),$(wildcard *.c)) allObjects+= $(patsubst %.cxx,%.$(OBJ),$(wildcard *.cxx)) allObjects+= $(patsubst %.cpp,%.$(OBJ),$(wildcard *.cpp)) all:: oneLib.$(STATICLIB) ############ here! oneLib.$(STATICLIB): $(allObjects) $(MKLIB) tests:: test1 ############ here! test1: someprog.exe $(RUNTEST) # runs the program $(CHECKFORDIFFS) # compares against expected # outputs Note that you can add more than one all:: target dependency in the same directory -- but then the task of creating the target predecessors gets a little tougher. 6. You can optionally hook into the clean:: target if you wish to add additional directory clean up logic over and above the default: clean:: $(RMALL) otherfiles || true $(RMDIR) otherdirs || true Alternatively, you can append to the string following strings to let the automated clean target do it for you: CLEANLIST += otherfiles CLEANDIRS += otherdirs Note that the CLEANLIST can contain wildcards but not directories. If you want directories deleted put them in the CLEANDIRS list. Note that the above examples are provided for explanatory purposes, see the rest of the subdirectories found here for more examples. Standard Targets and Rules -------------------------- The make.rules file defines the following standard make targets: all:: # an extensible list of the main things to be done # to build your project. Each Makefile should # include additional targets as predacessors to # all:: clean:: # an extensible list of rules for deleting objects # libraries, executables, log files, trash files, # etc that get created during a build or manually. backup: # a single rule for exporting your entire project # into a .zip file. This target should only be used # from the root directory of your project. tests:: # an extensible list of tests which will be run. # forces the all:: target to be run automatically nothing::# debug only target. Lets you make nothing, # recursively. Showing Hidden Make Actions --------------------------- Normally, most compilations / deletions do not get logged to standard out to reduce the visual noise. However, you may need to see what is going on if you are attempting to write your own make rules. Here's how to invoke make and see every command it is executing: make HIDE= make HIDE=clean ... This basically just overrides the definition of the HIDE variable which normally suppresses most output. You define it to be nothing and no suppression occurs. Showing the Compiler's Preprocessed Output ------------------------------------------ To force the creation of a preprocessed output for a given source file, use command like this: make file.i If there is a file,c, file.cpp, or file.cxx, then it will be preprocessed and stored in file.i. Writing your own build rules and recipes ---------------------------------------- Writing make target definitions is too complex to document here. There are some very sophisticated examples to be found in the example subdirectories. The Makefiles of interest are described at the top of this file. See the following excellent website for detailed explantions of GNU MAKE language, rule writing, etc. http://www.gnu.org/software/make/manual/make.html The following advice focuses on making your rules and recipes portable. The best advice for writing rules is to keep things as simple as possible but no simpler (per Albert Einstein). Manually writing long complex chains of dependencies is NOT the best solution even though it seems simplest if you don't know much about GNU MAKE. compilation is almost completely automated in these examples. If you follow the strategy shown here, you will only be forced to manually construct make rules and recipes for tests and installations. An important bit of guidance here is to use the macros defined in make.rules instead of normal shell commands (or windows command line commands). Unfortunately this takes some thought, sometimes, on how to make the rules portable. Here are the macros of of interest: Command Makefile Line MACRO Form Alternative ------------------------------------------------- rm -f $(RMALL) del $(RMALL) rm -fr $(RMDIR) rmdir $(RMDIR) mkdir -p $(MKDIR) mkdir cp -r $(CPDIR) xcopy /e /y cp $(CP) copy mv $(MV) move $(MV) echo $(ECHO) lib $(MKLIB) ar $(MKLIB) diff $(FILE_COMPARE) gcc ... *.c $(RAW_COMPILE).c.gcc g++ ... *.cpp $(RAW_COMPILE).cpp.gcc g++ ... *.cxx $(RAW_COMPILE).cxx.gcc cl ,,, .c $(RAW_COMPILE).c.cl cl ,,, .cpp $(RAW_COMPILE).c.cl cl ,,, .cxx $(RAW_COMPILE).c.cl These macros do not work magic! If you start writing your own recipes in GNU MAKE language, you will have to learn to write them in such a way that the same statements work on all the target platforms of interest. This can be a challenge. Luckily, you can search the make.rules file for examples... An obvious problem is that Windows loves \ characters and GNU MAKE dislikes them. When writing targets, you should generally use / instead, but when you execute any of the dos commands, you will need to convert the forward slashes to backslashes. Here's an example: copiedTarget: c:/program files/library/stuff/file cmd /c copy $(subst /,\\\\,$+) $@ The $+ means "all of the predecessors of the current target". The $(subst macro is documented below). It replaces strings. Note here that you must use four backslashes due to the GNU MAKE macro expansion escaping requirements. Linux and users: beware the standard tools: grep, sed, tr, sort, uniq etc. They exist in cygwin but no without it. Unfortunately, the dos alternatives either do not exist or are broken with respect to using them from inside GNU MAKE. The find command, for instance, requires that the first parameter be passed with surrounding double quotes. However, GNU MAKE simply will not do that no matter you to trick it. (unless the string has spaces or special characters in it). Anyway, in some cases it may make more sense to write a separate set of rules for linux/cygwin than is used Windows without cygwin. Here's an example: ifeq ("$(OS)","Windows") rule: $(subst /,\\\\,$(PROJECT_TOP_DIR))windows_helper.bat ... else rule: sed -e "stuff" | tr -d "\n" | sort -n >$@ endif In this example, you really just can't mimic the desired rule from inside GNU MAKE but there may be a way for a smartie, like yourself, to write a batch command that does work -- if you study the commands available on windows long enough. So, you write your helper .bat file and put it either in the current directory or that the project top directory, or some other convienient location and invoke it. Finally, if you are going to write your own rules, you need to learn to to effectively use the following built-in GNUM MAKE functions: $(filter pattern,list) -- given a space separated list, discard all but interesting items $(wildcard patternlist) -- get a possibly empty list of files matching one or more file name patterns (*.c *.h *.cxx, etc). $(patsubst %.ext,%.other,list ) -- transform the members of a list by replacing the suffix of filenames. (does more than just this!) $(foreach v,list,output) -- generate strings for each member in a list. "output" uses $(v) to produce targets, predecssors, etc. $(basename path) -- remove the suffix from path (Unlike the basename shell command, it only removes the suffix, no matter what it is). $(dir path) -- remove the filename from a path leaving the trailing slash $(notdir path) -- remove the directory part of a path leaving the basename and suffix $(subst a,b,s) -- replace all references to string a with string b in string s. (Very useful in making recipes portable to dos and unix. GNU MAKE strongly prefers / and strongly resists \ as file name separators but native dos commands require \. The substitute command helps you bridge between the worlds. In GNU MAKE macros, to might want to perform this substitution on any variable that is a filename: $(subst /,\\\\,$(name)) Note the 4 backslashes equal 1 by the name the dos command finally gets run Cool Stuff in GNU MAKE on WIN32 ------------------------------- Despite the fact that bash is not available on Windows without cygwin, the GNU MAKE implementers have helped makefile writers immensely by supporting the if-statement construct from bash. AND (YAY!) they support some of the conditional tests: -s filename # file exists and is not zero size -f filename # file exists (others?) The if-statement command, even under windows lets you write code like this: target: if SomeCommandInvocation >log ;\ then cmd /c echo YAY IT WORKED! ;\ cmd /c del log ;\ else cmd /c echo BOO IT FAILED! ;\ cmd /c echo Investigate... ;\ fi cmd /c echo Some Other Statement To use the conditionals, write code like this: target2: predecessor if [ ! -f $@ ] ; then echo $@ already exists fi cmd /c echo. >$@ Note also that the bash magic that lets you specify failure and success works in GNUM MAKE on windows without bash. Here are some example command lines: false # this command line fails the current target build true # basically a no opt because it means the current line does not fail any command || true # runs command and pretends it succeeded any command && false # runs the command and prentends it failes the target Here are some cool uses: 1. implementing a file compare recipe that cleans up after itself if fc filea fileb >$@.diff ; \ then cmd /c echo YAY THEY ARE EQUAL ; \ cmd /c del $@.diff ; \ else cmd /c echo ERROR THEY ARE DIFFERENT ; \ cmd /c echo SEE FILE $@.diff for differences ;\ fi 2. running a program that produces and log and deleting the log if it does not exit with an error code if program options >log ; then cmd /c echo YAY; cmd /c del log ; fi 3. making sure a program produced a needed file cmd /c del neededFile program if [ ! -f neededFile ] ; then cmd /c ERROR program did not create neededFile ; fi Portable Test Programs and Data Sets ------------------------------------ Source code portability is great but you also need to be run your tests in all environments of interest. There are several aspects to acheiving this: 1. Your source code and project compilation make rules have to be portable. 2. The steps you take to run your test programs and compare program it's outputs against expected values have to be portable. 3. The test program behavior has to be flexible with respect to the natural variations between operating systems. For example: * floating point string conversions have slight variations between operating systems. Sometimes you + or space on one operating system where as you get the oppose character on other. Some possible variants: 1.2e46 1.2e+46 1.2 e-46 * ctime() on some systems puts leading zeros on the day of the month but others do not (see tests/heisenprogl.cpp). Compare: Wed Feb 1 01:01:01 2012 Wed Feb 01 01:01:01 2012 * $HOME is not always /home/<userid>! Cygwin's $HOME is not the same as %HOMEPATH%... * PATH variable structure various between operating systems. The compilation rules in make.rules are portable to Linux, Windows, and Cygwin. Other operating systems should be supportable by following the patterns in make.rules. Test program execution and data comparison, at a basic level, can be made to be portable if you are careful. See tests/Makefile. In general it is safer to write tests in C or C++ and have them make appropriate comparisons and simple exit with a single pass/fail status. However, this is not always convenient -- the if statements may make the program impossibly complex to understand and maintain. An alternative is to have the program simply log output to stdout and use diff or FC to compare actual outputs against expected values. When doing this, it may become necessary to have multiple sets of expected output values: one per operating system type. The tests/Makefile shows how to write make macros and test targets that compare test program outputs against expected values and supports operating system specific overrides of the expected values. See $(CHECKDIFF)