Make and subdir

[Mauricio Ramirez]

I have a project with about 30 modules. We've been using the following as
our Makefile.am

ACLOCAL_FLAGS = -I buildtools # To find out custom macros (for aclocal)

SUBDIRS = \
mylibrary\
prog1 prog2 prog3 prog4 prog5 prog6 prog7 prog8 \
prog9 prog10 prog11 prog12 prog13 prog14 prog15 prog16 \
prog17 prog18 prog19 prog20 prog21 prog22 prog23 prog24 prog25 prog26 \
prog27 prog28 prog29 prog30 prog31


And the following as our Configure.ac

AC_PREREQ(2.69)
AC_INIT(system, 1.0, ***@myemailaddress.com)
AM_INIT_AUTOMAKE
AM_CONFIG_HEADER([config.h])

REQUIRE_TIBRV
REQUIRE_ORACLE

AC_PROG_LIBTOOL

# Our subdirectories
AC_DEFUN([SUBDIR],[
if test -d $1; then
AC_CONFIG_SUBDIRS($1)
fi
])

AC_CONFIG_SUBDIRS(mylibraru)
SUBDIR(prog1 )
SUBDIR(prog2 )
SUBDIR(prog3 )
SUBDIR(prog4 )
SUBDIR(prog5 )
SUBDIR(prog6 )
SUBDIR(prog7 )
SUBDIR(prog8 )
SUBDIR(prog9 )
SUBDIR(prog10 )
SUBDIR(prog11 )
SUBDIR(prog12 )
SUBDIR(prog13)
SUBDIR(prog14)
SUBDIR(prog15 )
SUBDIR(prog16 )
SUBDIR(prog17 )
SUBDIR(prog18 )
SUBDIR(prog19 )
SUBDIR(prog20 )
SUBDIR(prog21 )
SUBDIR(prog22 )
SUBDIR(prog23 )
SUBDIR(prog24 )
SUBDIR(prog25 )
SUBDIR(prog26 )
SUBDIR(prog27 )
SUBDIR(prog28 )
SUBDIR(prog29 )
SUBDIR(prog30 )
SUBDIR(prog31 )
AC_OUTPUT(Makefile)


This has worked great on RedHat 5, but when we're moving to Redhat 7, it
fails when building mylibrary. It can't find some includes. If I go into
the mylibrary directory, and do the make, it works just fine. Not
understanding why I can no longer build the whole thing in one shot.

[Warren Young]

Post by Mauricio Ramirez
This has worked great on RedHat 5, but when we're moving to Redhat 7, it
fails when building mylibrary. It can't find some includes. If I go into
the mylibrary directory, and do the make, it works just fine. Not
understanding why I can no longer build the whole thing in one shot.

I’d suggest that you take this as an opportunity to move to a single top-level Makefile and single top-level configuration system. It has significant benefits, especially in today’s world of real hardware parallelism:

http://lcgapp.cern.ch/project/architecture/recursive_make.pdf

In that paper’s day, only high-end systems had multiple physical processor cores, highly parallel storage subsystems, etc., so its advice held less weight than today.

I regularly build my software with a 1.5x oversubscription on the cores. That is, on an 8-core system, I use “make -j12”. A parallel make with that many possible parallel build steps going on at once is far more reliable when there is a single instance of make(1) controlling the whole build process.

I’ve wrapped that functionality into a command called mmake:

https://tangentsoft.com/pidp8i/doc/trunk/tools/mmake

That is in turn dependent on another script I wrote called corecount, which hides away the local mechanism for counting the number of CPU cores:

https://tangentsoft.com/pidp8i/doc/trunk/tools/corecount

I welcome patches to that script that work on systems I haven’t encountered yet. I fully expect that corecount incorrectly returns “1” all the time on Solaris, for instance.

I’ve been using this pair of scripts long enough now that I reflexively type “mmake” rather than “make.” I’ve also got Vim set to use mmake by default:


set makeprg=mmake

map <F5> :make run<CR>
map <F7> :make<CR>
map <F8> :make<CR>
map <F9> :make run<CR>


Those hot keys may be familiar to you if your work experience extends beyond the Autotools world.

If the “Recursive Make Considered Harmful” argument bothers you, you may want to read this paper next:

https://www.microsoft.com/en-us/research/publication/non-recursive-make-considered-harmful/

But realize, that paper isn’t telling you to keep using recursive Makefiles. They’re reporting that they couldn’t get make to work for them at all at that scale, so they replaced it entirely. If you’re going to do that, this is not the right mailing list for you. :)

Non-recursive make has never been unmanageable at the scales I work at. My largest Automake-managed project has 30 top-level build products, so it may be of comparable scale to your project.

[Václav Haisman]

Post by Mauricio Ramirez
I have a project with about 30 modules. We've been using the following as
our Makefile.am
ACLOCAL_FLAGS = -I buildtools # To find out custom macros (for aclocal)
SUBDIRS = \
mylibrary\
prog1 prog2 prog3 prog4 prog5 prog6 prog7 prog8 \
prog9 prog10 prog11 prog12 prog13 prog14 prog15 prog16 \
prog17 prog18 prog19 prog20 prog21 prog22 prog23 prog24 prog25 prog26 \
prog27 prog28 prog29 prog30 prog31
And the following as our Configure.ac
AC_PREREQ(2.69)
AM_INIT_AUTOMAKE
AM_CONFIG_HEADER([config.h])
REQUIRE_TIBRV
REQUIRE_ORACLE
AC_PROG_LIBTOOL
# Our subdirectories
AC_DEFUN([SUBDIR],[
if test -d $1; then
AC_CONFIG_SUBDIRS($1)
fi
])
AC_CONFIG_SUBDIRS(mylibraru)
SUBDIR(prog1 )
SUBDIR(prog2 )
SUBDIR(prog3 )
SUBDIR(prog4 )
SUBDIR(prog5 )
SUBDIR(prog6 )
SUBDIR(prog7 )
SUBDIR(prog8 )
SUBDIR(prog9 )
SUBDIR(prog10 )
SUBDIR(prog11 )
SUBDIR(prog12 )
SUBDIR(prog13)
SUBDIR(prog14)
SUBDIR(prog15 )
SUBDIR(prog16 )
SUBDIR(prog17 )
SUBDIR(prog18 )
SUBDIR(prog19 )
SUBDIR(prog20 )
SUBDIR(prog21 )
SUBDIR(prog22 )
SUBDIR(prog23 )
SUBDIR(prog24 )
SUBDIR(prog25 )
SUBDIR(prog26 )
SUBDIR(prog27 )
SUBDIR(prog28 )
SUBDIR(prog29 )
SUBDIR(prog30 )
SUBDIR(prog31 )
AC_OUTPUT(Makefile)
This has worked great on RedHat 5, but when we're moving to Redhat 7, it
fails when building mylibrary. It can't find some includes. If I go into
the mylibrary directory, and do the make, it works just fine. Not
understanding why I can no longer build the whole thing in one shot.

I, too, suggest that you explore non-recursive build. And if you have
many directories/components, I suggest that you take a look at Autogen
(https://www.gnu.org/software/autogen/) to make it easier to add new ones.

If you need examples, I have (biased, author) one in log4cplus'
Autotools build system.