configure speedup proposal: add --assume-c99 and --assume-posix2008 flags

[John Spencer]

there are many configure scripts out there that still check for things
that are standard since at least 10 years, and doing this extensively
and over and over (people building software themselves usually build
more than one package) consumes a lot of time (especially due to the
non-parallel nature of configure scripts).
often configure scripts take longer to finish than compiling the source
code itself on decent machines with multiple cpu cores.

having an option like --assume-c99 could provide a shortcut so all
checks like

- have stdint.h
- have snprintf()
- etc

and for posix
- sizeof int == 4
- have select()
- etc

could be skipped and assumed to be always true.

only functions which are known to be non-conforming on some platforms
(like the broken strerror_r() semantics of GLIBC which conflict with
POSIX when certain CPPFLAGS are used), and special platform specific
things like linux extensions needed to be checked separately using this
mechanism.

to me, this sounds like a simple way to make configure runs at least 50%
faster on most modern platforms, while not impacting legacy systems
negatively at all.

having configure run much faster could also lead to a better public
perception of autoconf in general.

as for how to implement, maybe these flags could just set the
appropriate autoconf cache variables to true.

regards,
--JS

[John Spencer]

there are many configure scripts out there that still check for things
that are standard since at least 10 years, and doing this extensively
and over and over (people building software themselves usually build
more than one package) consumes a lot of time (especially due to the
non-parallel nature of configure scripts).
often configure scripts take longer to finish than compiling the source
code itself on decent machines with multiple cpu cores.

having an option like --assume-c99 could provide a shortcut so all
checks like

- have stdint.h
- have snprintf()
- etc

and for posix
- sizeof int == 4
- have select()
- etc

could be skipped and assumed to be always true.

only functions which are known to be non-conforming on some platforms
(like the broken strerror_r() semantics of GLIBC which conflict with
POSIX when certain CPPFLAGS are used), and special platform specific
things like linux extensions needed to be checked separately using this
mechanism.

to me, this sounds like a simple way to make configure runs at least 50%
faster on most modern platforms, while not impacting legacy systems
negatively at all.

having configure run much faster could also lead to a better public
perception of autoconf in general.

as for how to implement, maybe these flags could just set the
appropriate autoconf cache variables to true.

regards,
--JS

[Bob Friesenhahn]

there are many configure scripts out there that still check for things that
are standard since at least 10 years, and doing this extensively and over and
over (people building software themselves usually build more than one
package) consumes a lot of time (especially due to the non-parallel nature of
configure scripts).
often configure scripts take longer to finish than compiling the source code
itself on decent machines with multiple cpu cores.

Agreed.

having an option like --assume-c99 could provide a shortcut so all checks

I don't like any approach which hard-codes assumptions in Autoconf.
Hard-coded assumptions can easily be wrong and configure scripts could
be used ten years later without being re-generated with the latest
rules.

The Autoconf configuration cache mechanism (e.g. --config-cache) is
very effective. In my test here, using --config-cache reduced
repeated configure run time from 20 seconds to 5 seconds. By
transcribing selected lines from config.cache to config.site, these
cached values can be made the default for a system. Modern autoconf
searches for both $prefix/share/config.site and
$prefix/etc/config.site and will use whichever ones are found.

E.g.

egrep '^ac_cv_func_select' config.cache >> /usr/local/share/config.cache

and in /usr/local/share/config.site

. /usr/local/share/config.cache

In fact, one could take a boilerplate configure.ac, use it to create a
bare configure, and then use most of its config.cache to form
config.site.

With this approach, Autoconf can be optimized for a system with known
behavior without placing brittle assumptions within Autoconf itself.

A separate script could be used to create a reusable subset of
config.cache based on a specification.

BTFW that some software might not configure correctly given the
presence of a config.site file. In my experience, a config.site file
can cause harm when configuring GCC.

Bob

[John Spencer]

Post by Bob Friesenhahn

Post by John Spencer
there are many configure scripts out there that still check for things
that are standard since at least 10 years, and doing this extensively
and over and over (people building software themselves usually build
more than one package) consumes a lot of time (especially due to the
non-parallel nature of configure scripts).
often configure scripts take longer to finish than compiling the
source code itself on decent machines with multiple cpu cores.

Agreed.

Post by John Spencer
having an option like --assume-c99 could provide a shortcut so all checks

I don't like any approach which hard-codes assumptions in Autoconf.
Hard-coded assumptions can easily be wrong and configure scripts could
be used ten years later without being re-generated with the latest rules.
The Autoconf configuration cache mechanism (e.g. --config-cache) is very
effective. In my test here, using --config-cache reduced repeated
configure run time from 20 seconds to 5 seconds. By transcribing
selected lines from config.cache to config.site, these cached values can
be made the default for a system. Modern autoconf searches for both
$prefix/share/config.site and $prefix/etc/config.site and will use
whichever ones are found.

yes, i currently use an approach that copies a custom config.cache into
the builddir. however this implicates a maintenance burden, and i heard
some voices criticizing that approach as to unreliable.
using a config.site in place of my manual approach is interesting and
may make it look a little bit more legitimate, however it's essentially
the same.

what i'm trying to achieve is that there's some kind of official
upstream stategy (like currently the config.site usage) that could be
used/consumed easily (i.e. without having to manually maintain the
cachefile).

Post by Bob Friesenhahn
E.g.
egrep '^ac_cv_func_select' config.cache >> /usr/local/share/config.cache
and in /usr/local/share/config.site
. /usr/local/share/config.cache
In fact, one could take a boilerplate configure.ac, use it to create a
bare configure, and then use most of its config.cache to form config.site.

this is a good idea.
maybe autoconf could ship a config.site generator script which tests the
libc capabilities in regard to c99 and posix 2008, and generates the
cache for usage in a systemwide install?
testing anything else like for example whether ncurses.h is installed is
imo bogus, as the user could easily remove ncurses-dev from his box
after having run through the .site-generator script.
otoh it's unlikely that his libc implementation removes functionality
that it already had.
despite that, the script should definitely display a warning that it
should be rerun whenever the libc was updated.

having this script available as part of autoconf would make it easily
accessible to distribution maintainers, so everyone could benefit.

Post by Bob Friesenhahn
With this approach, Autoconf can be optimized for a system with known
behavior without placing brittle assumptions within Autoconf itself.
A separate script could be used to create a reusable subset of
config.cache based on a specification.
BTFW that some software might not configure correctly given the presence
of a config.site file. In my experience, a config.site file can cause
harm when configuring GCC.

i didnt experience this sort of problem with gcc, but i had some cases
where the information in my handtuned config.cache was outdated and
caused build failures.

--JS

[Tim Rice]

having an option like --assume-c99 could provide a shortcut so all checks

I don't like any approach which hard-codes assumptions in Autoconf. Hard-coded
assumptions can easily be wrong and configure scripts could be used ten years
later without being re-generated with the latest rules.

Agreed.
An example is xz-5.0.5. Configure errors out complaining of no c99 compiler
but after fooling configure the package builds fine using a c89 compiler.

[Russ Allbery]

Post by John Spencer
having an option like --assume-c99 could provide a shortcut so all
checks like
- have stdint.h
- have snprintf()
- etc

These are un-alike, just to mention. A surprising number of platforms
have an snprintf function that's broken. To test it properly, you need
something akin to the gnulib snprintf check, and it's broken in enough
places that you may not want to assume the result.

Some of the problems with snprintf are also quite serious. For example,
Solaris 9 (which I believe is otherwise C99-conforming) would return -1
when given NULL, 0 as the first arguments to snprintf, which makes it
impossible to use snprintf safely in many programs.

See:

https://www.gnu.org/software/gnulib/manual/html_node/snprintf.html

for more information about the portability mess.

[John Spencer]

Post by Russ Allbery

Post by John Spencer
having an option like --assume-c99 could provide a shortcut so all
checks like
- have stdint.h
- have snprintf()
- etc

These are un-alike, just to mention. A surprising number of platforms
have an snprintf function that's broken. To test it properly, you need
something akin to the gnulib snprintf check, and it's broken in enough
places that you may not want to assume the result.
Some of the problems with snprintf are also quite serious. For example,
Solaris 9 (which I believe is otherwise C99-conforming) would return -1
when given NULL, 0 as the first arguments to snprintf, which makes it
impossible to use snprintf safely in many programs.
https://www.gnu.org/software/gnulib/manual/html_node/snprintf.html
for more information about the portability mess.

point taken, although all the platforms described there are not exactly
what i would call modern, my leaning would be more towards calling them
broken, obsolete or simply undead.

<thinking_loud>

i even ask myself sometimes if it is still worth plastering software and
configure scripts with checks and ifdefs for these broken platforms that
probably cant even build the dependencies of modern software (these
ifdefs are almost always completely unmaintained due to lack of testing
and very likely to fail), and even if they still compile *and* work
there, are probably not even used.
because the handful of people using these platforms stick to their
current versions anyway...

speaking of it, it may be a good test to revive one of these zombie OS's
trying to do a full build of the 500 most popular non-linux-specific
packages in debian to proof that probably less than 10% will compile
despite extensive platform-specific hacks and workarounds.
carrying around all this legacy baggage is kinda like porting a 16bit
DOS program to win 3.11, then to win95, 98, xp, 7, 8 and still keeping
all the old 16bit code in #ifdefs without testing it ever.

btw, to this day i've yet to see a library or program, that for example
checks for unistd.h, and then uses HAVE_UNISTD_H consequently in all
TUs, including optional usage of any of its exported functions.
this probably worked once, when the initial autoconf support was written
and tested, but the next developer that came along just used unistd.h
without any conditionals...

</thinking_loud>

note that my suggestion is not to default to this shortcut, it's meant
to be used by users that know that their platform is fully compliant,
so it wouldnt affect users of these systems.

but maybe to make it more clear, the option should be called
--assume-c99-compliance instead.

--JS

[Bob Friesenhahn]

Post by John Spencer
i even ask myself sometimes if it is still worth plastering software and
configure scripts with checks and ifdefs for these broken platforms that
probably cant even build the dependencies of modern software (these ifdefs
are almost always completely unmaintained due to lack of testing and very
likely to fail), and even if they still compile *and* work there, are
probably not even used.
because the handful of people using these platforms stick to their current
versions anyway...

It seems likely that you are not a user of one of these platforms and
therefore not really qualified to characterize them. Most of them are
not "broken platforms" since they work just fine.

People who still use old systems do so because newer operating system
versions don't run on (or run well on) their hardware. It is up to
package developers if they care to ensure that their software runs on
something other than a very recent OS version.

Post by John Spencer
speaking of it, it may be a good test to revive one of these zombie OS's
trying to do a full build of the 500 most popular non-linux-specific packages
in debian to proof that probably less than 10% will compile despite extensive
platform-specific hacks and workarounds.
carrying around all this legacy baggage is kinda like porting a 16bit DOS
program to win 3.11, then to win95, 98, xp, 7, 8 and still keeping all the
old 16bit code in #ifdefs without testing it ever.

You are blowing the magnitude of the "legacy baggage" way out of
proportion. In fact, there is really not much left compared to what
there used to be.

Please take care because most of the software you are using was
developed on one of these legacy systems and worked just fine when it
was developed (and most still does now). Many people here developed
those systems that you describe as "broken" now.

Bob

[Thomas Jahns]

Post by Russ Allbery
Some of the problems with snprintf are also quite serious. For example,
Solaris 9 (which I believe is otherwise C99-conforming) would return -1
when given NULL, 0 as the first arguments to snprintf, which makes it
impossible to use snprintf safely in many programs.

IIRC Solaris 9 also doesn't have stdint.h (but a sufficiently good inttypes.h
for most purposes).

Thomas
--
Thomas Jahns
DKRZ GmbH, Department: Application software

Deutsches Klimarechenzentrum
Bundesstraße 45a
D-20146 Hamburg

Phone: [phone]
Fax: [phone]
Email: Thomas Jahns <***@dkrz.de>

[Gavin Smith]

On Sun, Mar 23, 2014 at 12:24 AM, John Spencer

there are many configure scripts out there that still check for things that
are standard since at least 10 years, and doing this extensively and over
and over (people building software themselves usually build more than one
package) consumes a lot of time (especially due to the non-parallel nature
of configure scripts).
often configure scripts take longer to finish than compiling the source code
itself on decent machines with multiple cpu cores.
having an option like --assume-c99 could provide a shortcut so all checks
like
- have stdint.h
- have snprintf()
- etc
and for posix
- sizeof int == 4
- have select()
- etc
could be skipped and assumed to be always true.

If changing the configure scripts in question is an option, developers
could simply require C99 with AC_PROG_CC_C99 if they rely on many
characteristics of C99. autoscan suggests tests; some of these might
be unnecessary or included in broader tests.

[Ralf Corsepius]

Post by John Spencer
there are many configure scripts out there that still check for things
that are standard since at least 10 years, and doing this extensively
and over and over (people building software themselves usually build
more than one package) consumes a lot of time (especially due to the
non-parallel nature of configure scripts).
often configure scripts take longer to finish than compiling the source
code itself on decent machines with multiple cpu cores.
having an option like --assume-c99 could provide a shortcut so all
checks like
- have stdint.h
- have snprintf()
- etc
and for posix
- sizeof int == 4
- have select()
- etc

These assumptions are unsafe, esp. on non-mainstream architectures/OSes,
which only partially implement c99.

E.g. it's not that uncommon to find toolchains for embedded platforms
which implement most of C99, but with sizeof int != 4 or without "select()".

Ralf

[John Spencer]

Post by Ralf Corsepius

Post by John Spencer
and for posix
- sizeof int == 4
- have select()
- etc

These assumptions are unsafe, esp. on non-mainstream architectures/OSes,
which only partially implement c99.
E.g. it's not that uncommon to find toolchains for embedded platforms
which implement most of C99, but with sizeof int != 4 or without "select()".

well, noone forces you to use --assume-compliant-c99.
as outlined in my previous responses, the usage is *optional* and *not
default* and you would only use it on known non-broken platforms (i.e.
those that implement C99 to the letter of the standard, for example
linux + musl libc).

[Thomas Petazzoni]

Dear John Spencer,

Post by John Spencer
there are many configure scripts out there that still check for things
that are standard since at least 10 years, and doing this extensively
and over and over (people building software themselves usually build
more than one package) consumes a lot of time (especially due to the
non-parallel nature of configure scripts).

As one of the core developer of Buildroot (http://buildroot.org), a
tool that automates the process of cross-compiling a potentially
significant number of software components to generate embedded Linux
systems, I am also very concerned about the speed of configure scripts.
On a full build, the time taken by configure scripts is very often
around 25% of the overall build time, if not more.

The suggestion of using a cache is indeed very interesting. We actually
tried it a couple of years ago, with a global cache shared by all
packages we build. However, it turned out that several packages use the
same autoconf variable name to store the result of tests that are not
exactly the same, causing a number of issues. The proposal in the
thread to use a pre-defined cache with only a selected number of
known-safe entries looks like a good idea.

However, what seems like the most important problem to me is the
non-parallel nature of autoconf tests. With CPUs going more and more
multi-core, the build part of packages (which is usually highly
parallel) is going to get faster and faster comparatively to the
configure part of packages (which is going to get stuck to a slow speed
due to the non-parallel nature of autoconf). Wouldn't it be time to
think about moving autoconf to a more parallel logic, where N tests
could run in parallel in sub-processes?

Thanks,

Thomas

[Paul Eggert]

Post by Thomas Petazzoni
Wouldn't it be time to
think about moving autoconf to a more parallel logic, where N tests
could run in parallel in sub-processes?

It's long been past time to do that. Some technical problems need to be
solved, but the main problem is that nobody has volunteered to take on
the task.

[Rhys Ulerich]

Post by Thomas Petazzoni
Wouldn't it be time to
think about moving autoconf to a more parallel logic, where N tests
could run in parallel in sub-processes?

I only find the configure process to be frustratingly slow when run
against compilers with license keys or with checks accessing network
storage. Running those in parallel would do nothing but block on the
license servers and beat up already sluggish file servers.

- Rhys

[Paul Eggert]

Post by Rhys Ulerich
I only find the configure process to be frustratingly slow when run
against compilers with license keys or with checks accessing network
storage.

That's not my experience at all. Take the bleeding-edge version of GNU
grep, say, which I just now happened to be working on. On my platform
(RHEL 6.5 x86-64, GCC 4.9.0, dual Xeon E5620), 'configure' takes 32
seconds real-time, whereas 'make -j8' takes 3 seconds.

No doubt there are environments where compilation is slow and
'configure' is not the bottleneck, but these days it's veerrry common
for 'configure' to be an unreasonably large part of the build process,
and where parallelizing would help.

[Rhys Ulerich]

veerrry common for 'configure' to be an unreasonably large part of the build
process

But is an unreasonably large part of a reasonably long build process?
You quote a from-scratch build with configure being an order of
magnitude slower than parallel compilation. But one just 35 seconds
long.

Granted, configure time does add up if one builds 10s of packages in
sequence. Is there some reason configure caches are unsuitable to
gain speedup there? I see almost a six-fold configure improvement on
grep-2.20 when its working from a config.cache. That brings configure
to 140% of parallel compilation, but we're talking an overall build
like 6 seconds.

- Rhys

[Paul Eggert]

Post by Rhys Ulerich
Is there some reason configure caches are unsuitable to
gain speedup there?

I regularly build with experimental versions in which the 'configure'
tests themselves differ from run to run, and where configure caches
would give me incorrect results. So I never use caches.

I have to wait for an unreasonably long period of time whenever I change
anything that requires reconfiguration, and this happens quite
frequently to me (often several times per day).

Post by Rhys Ulerich
one just 35 seconds long.

It's a 35-second wait in an edit-build-run cycle, and that's wayy too
long. I typically give up and work on something else for a while. The
mental context-switching overhead is extremely annoying.

I would greatly appreciate it if 'configure' ran faster.

[Harlan Stenn]

I should probably get the configure and build times re-measured for NTP.

Feel free to grab
http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-dev/ntp-dev-4.2.7p444.tar.gz
and see for yourself... it's a bit of a monster for something that's
about 200k LOC.

It would be great to see significant speedup in the configure time, as
we have a growing number of buildbot targets and we plan to do both
scratch builds as well as upgrade (with autoconf cache) builds as part
of our build/test process.

[Thomas Petazzoni]

Dear Rhys Ulerich,

Post by Rhys Ulerich
Granted, configure time does add up if one builds 10s of packages in
sequence. Is there some reason configure caches are unsuitable to
gain speedup there? I see almost a six-fold configure improvement on
grep-2.20 when its working from a config.cache. That brings configure
to 140% of parallel compilation, but we're talking an overall build
like 6 seconds.

That works when you repeatedly build the same package over and over
again with the exact same toolchain and environment.

My use case in Buildroot is very different: we build potentially
several dozens of different software components, each time with a
different toolchain/environment. So we're almost never rebuilding with
the exact same configuration, and therefore using a cache doesn't
really work for our use case.

Best regards,

Thomas

[Rhys Ulerich]

Post by Thomas Petazzoni
My use case in Buildroot is very different: we build potentially
several dozens of different software components, each time with a
different toolchain/environment. So we're almost never rebuilding with
the exact same configuration, and therefore using a cache doesn't
really work for our use case.

It sounds like there's already an appreciable amount of embarassingly
parallel work in the environment as none of those builds should depend
on any other. Non-parallel autoconf would hurt the latency of a
single build but shouldn't cause much change in the throughput of
builds one can achieve.

My apologies if my experience doesn't line up with others-- my
projects take 1-3 minutes to configure, somewhere between 10 minutes
and an hour to build, and something like a month of straight
computation on hundreds of nodes before I get an answer. Configure
times just aren't my bottleneck.

I'm not saying parallel configure wouldn't be nice, but that I keep
seeing other ways to take advantage of caching and parallelism that
don't require making a long-standing serial codebase parallel.

- Rhys