Cross-platform availability of header files

[Zack Weinberg]

I've been doing research into the cross-platform availability of
header files that are commonly probed for in Autoconf scripts.
Results so far are here:
http://www.owlfolio.org/possibly-useful/notes-on-the-cross-platform-availability-of-header-files/
Based on what I've learned, I have some questions for y'all:

1) What "interesting portability targets" have I left out? I only
went back in time as far as FreeBSD 7, and I didn't even try to get my
hands on any of the surviving proprietary Unixes; is this too
shortsighted?

2) Autoconf currently probes for several of the headers in the "safely
assumed to exist everywhere" categories, notably in
AC_INCLUDES_DEFAULT. It seems to me that this is unnecessary. Would
patches to remove under-the-hood checks for the ubiquitous headers,
and deprecate macros that do explicit checks for them, be accepted?

3) It's a little tangential, but don't you think it's about time
AC_CHECK_HEADERS stopped doing all its tests two different ways?

Thanks,
zw

[Paul Eggert]

Post by Zack Weinberg
1) What "interesting portability targets" have I left out? I only
went back in time as far as FreeBSD 7, and I didn't even try to get my
hands on any of the surviving proprietary Unixes; is this too
shortsighted?

Yes, I think so. We regularly get bug reports from people running
HP-UX, AIX, etc.

Post by Zack Weinberg
2) Autoconf currently probes for several of the headers in the "safely
assumed to exist everywhere" categories, notably in
AC_INCLUDES_DEFAULT. It seems to me that this is unnecessary. Would
patches to remove under-the-hood checks for the ubiquitous headers,
and deprecate macros that do explicit checks for them, be accepted?

If it's safe to include them now, we should stop checking for them.

Post by Zack Weinberg
3) It's a little tangential, but don't you think it's about time
AC_CHECK_HEADERS stopped doing all its tests two different ways?

I would change this, yes.

[Zack Weinberg]

Post by Paul Eggert

Post by Zack Weinberg
1) What "interesting portability targets" have I left out? I only
went back in time as far as FreeBSD 7, and I didn't even try to get my
hands on any of the surviving proprietary Unixes; is this too
shortsighted?

Yes, I think so. We regularly get bug reports from people running
HP-UX, AIX, etc.

It's not going to be easy for me to scrape up one of those; it was hard
enough putting a Solaris-descendant VM together. I wonder if people who do
have them would be willing to run "find /usr/include -name \*.h -print" and
send in the output.

I think we should try to come up with a principled cutoff for how old is
too old, though. I started this thinking POSIX.1-2001 (including XSI, but
maybe not any other options) was a reasonable place to draw the line, but
it turns out Android omits a bunch of that (and not the old junk either) so
it's not so simple.

"You can assume a C89 hosted environment" does still seem like a sound
assertion, though.

Post by Paul Eggert
2) Autoconf currently probes for several of the headers in the "safely

Post by Zack Weinberg
assumed to exist everywhere" categories, notably in
AC_INCLUDES_DEFAULT. It seems to me that this is unnecessary. Would
patches to remove under-the-hood checks for the ubiquitous headers,
and deprecate macros that do explicit checks for them, be accepted?

If it's safe to include them now, we should stop checking for them.

Post by Zack Weinberg
3) It's a little tangential, but don't you think it's about time
AC_CHECK_HEADERS stopped doing all its tests two different ways?

I would change this, yes.

I'll see if I can find time to put some patches together.

[Russ Allbery]

Post by Zack Weinberg
I think we should try to come up with a principled cutoff for how old is
too old, though. I started this thinking POSIX.1-2001 (including XSI,
but maybe not any other options) was a reasonable place to draw the
line, but it turns out Android omits a bunch of that (and not the old
junk either) so it's not so simple.
"You can assume a C89 hosted environment" does still seem like a sound
assertion, though.

It's also important not to exclude Windows, which sometimes is missing
some things that are otherwise universal. Autoconf can't probe on Windows
*directly* (without using one of the UNIX-like portability shells, which
often provide the missing bits as well), but the fact that Autoconf
generates the conditionals (in config.h for example) is still extremely
useful in combination with a hand-written config.h.win that's used by
Windows MSVC builds.

(ssize_t was the thing I ran into most recently that Windows doesn't have
but which is otherwise universal and required by POSIX.)

[Peter Rosin]

Post by Russ Allbery

Post by Zack Weinberg
I think we should try to come up with a principled cutoff for how old is
too old, though. I started this thinking POSIX.1-2001 (including XSI,
but maybe not any other options) was a reasonable place to draw the
line, but it turns out Android omits a bunch of that (and not the old
junk either) so it's not so simple.
"You can assume a C89 hosted environment" does still seem like a sound
assertion, though.

It's also important not to exclude Windows, which sometimes is missing
some things that are otherwise universal. Autoconf can't probe on Windows
*directly* (without using one of the UNIX-like portability shells, which
often provide the missing bits as well), but the fact that Autoconf
generates the conditionals (in config.h for example) is still extremely
useful in combination with a hand-written config.h.win that's used by
Windows MSVC builds.
(ssize_t was the thing I ran into most recently that Windows doesn't have
but which is otherwise universal and required by POSIX.)

I just wanted to chime in with the fact that it's working just fine to
use one of said UNIX-like portability environments and have Autoconf
probe MSVC directly. I.e. no need for any handwritten config.h.win
file. Having said that, it doesn't work for all projects of course,
but if it doesn't work and the project maintainer is willing it should
still be doable. I personally use this regularly and would hate to
see the support for it go away.

The bigger Windows problems will always be missing fork() or something
like that. The tool environment is easy in comparison (or it has been
solved by said UNIX-like environments is perhaps a truer statement).

Cheers,
Peter

[Zack Weinberg]

Post by Peter Rosin
I just wanted to chime in with the fact that it's working just fine to
use one of said UNIX-like portability environments and have Autoconf
probe MSVC directly.

This is relevant to my interests ;) How do you smooth over the
differences in command line interface between 'cc' and 'cl'? (Can you
make it work in the presence of libtool?)

zw

[Peter Rosin]

Post by Zack Weinberg

Post by Peter Rosin
I just wanted to chime in with the fact that it's working just fine to
use one of said UNIX-like portability environments and have Autoconf
probe MSVC directly.

This is relevant to my interests ;) How do you smooth over the
differences in command line interface between 'cc' and 'cl'?

The 'compile' script from (modern) Automake is good enough and 'depcomp'
has modes to support MSVC. There is also the 'ar-lib' script from the
same source that wraps lib.exe to support basic archival operations
in much the same manner.

Post by Zack Weinberg
(Can you make it work in the presence of libtool?)

Yes, libtool with MSVC in MSYS (or in Cygwin, but that is more alien)
works just fine.

I tend to use something like this:

.../configure \
CC=cl \
CFLAGS=-MD \
CXX=cl \
CXXFLAGS=-MD \
LD=link \
NM="dumpbin -symbols -headers" \
AR=lib \
STRIP=: \
RANLIB=:

It the package doesn't have AM_PROG_CC_C_O in its configure.ac it
might be needed to say CC="/path/to/compile cl", and sometimes it
might be good to say CC="cl -nologo" (since the "logo" boilerplate
is dumped on stderr and any output on stderr from the compiler is
suspect). Similarly for AR=lib vs AR="/path/to/ar-lib lib" if the
package doesn't have AM_PROG_AR.

Libtool knows about using dumpbin as symbol lister, but if the
build uses $NM directly from some Makefile (or otherwise) it gets
tougher, as the wrapper is implemented "internally" in libtool.

Cheers,
Peter

[Zack Weinberg]

Post by Russ Allbery

Post by Zack Weinberg
I think we should try to come up with a principled cutoff for how old is
too old, though. I started this thinking POSIX.1-2001 (including XSI,
but maybe not any other options) was a reasonable place to draw the
line, but it turns out Android omits a bunch of that (and not the old
junk either) so it's not so simple.
"You can assume a C89 hosted environment" does still seem like a sound
assertion, though.

It's also important not to exclude Windows, which sometimes is missing
some things that are otherwise universal.

Quite so; I have been at pains to divide everything beyond C89 into
"truly universal" and "not available on Windows". And I do find it
useful to be able to reuse configure in one of the pseudo-Unix
environments on Windows.

However, I think the right way to handle Windows/not-Windows *in the
code* is with #if(n)def _WIN32. One probably needs to do more than just
avoid unavailable headers, after all.

I should also emphasize that what I've been doing is *only* about the
presence of headers, not their contents.

zw

[Paul Eggert]

we should try to come up with a principled cutoff for how old is too old

A good rule of thumb is: if the maintainer doesn't fix bugs in
the software any more, it's too old. So, for example, in March 2012
Oracle stopped fixing bugs in Solaris 8, which means Solaris 8 is
now too old. Solaris 9 is scheduled to become too old in October 2014,
and so forth.

According to this guideline, recent releases of AIX and HP-UX are not too old:
their bugs are fixed and people are using them.

[Thomas Jahns]

Post by Zack Weinberg
I've been doing research into the cross-platform availability of
header files that are commonly probed for in Autoconf scripts.
http://www.owlfolio.org/possibly-useful/notes-on-the-cross-platform-availability-of-header-files/

although you didn't ask: aio.h does not need to be probed for: unistd.h defines
_POSIX_ASYNCHRONOUS_IO when it's available.

See <http://pubs.opengroup.org/onlinepubs/007904875/basedefs/unistd.h.html> for
some other features that don't require autoconf to check for, sys/mman.h is in
the same category.

With respect to your comment on identifiers with leading underscore: to be
conformant to e.g. C89 standards, an implementation must not define anything
without said leading underscore, but in cases where the non-standard
functionality is required to implement standard-mandated functionality, it's
easiest to have it available with a leading underscore.

In case you still need it, I could check AIX 6.1 for any header you want to know
about or provide an ls -R of /usr/include, although some of that are packaged
products and e.g. aio requires the installation of an optional kernel extension
package although I'm not sure if the header is available even when said package
is not installed.

Regards, Thomas
--
Thomas Jahns
DKRZ GmbH, Department: Application software

Deutsches Klimarechenzentrum
Bundesstraße 45a
D-20146 Hamburg

Phone: [phone]
Fax: [phone]
Email: Thomas Jahns <***@dkrz.de>

[Ralf Corsepius]

Post by Thomas Jahns

Post by Zack Weinberg
I've been doing research into the cross-platform availability of
header files that are commonly probed for in Autoconf scripts.
http://www.owlfolio.org/possibly-useful/notes-on-the-cross-platform-availability-of-header-files/

although you didn't ask: aio.h does not need to be probed for: unistd.h defines
_POSIX_ASYNCHRONOUS_IO when it's available.
See <http://pubs.opengroup.org/onlinepubs/007904875/basedefs/unistd.h.html> for
some other features that don't require autoconf to check for, sys/mman.h is in
the same category.

This is the description of an ideal world - Reality is a bit different.

aio.h and sys/mman.h are not available everywhere, nor are the _POSIX_*
defines always correct, nor are the features these are supposed to
provide available everywhere.

Ralf

[Zack Weinberg]

Post by Ralf Corsepius
aio.h and sys/mman.h are not available everywhere

aio.h certainly, but are you aware of a *specific Unix-like system*
which is in current use and does not provide <sys/mman.h>? That header
is mandatory in POSIX-2008 (although not before then) and the last time
I can remember encountering a workstation-scale Unix environment that
didn't have it was in the mid-nineties sometime. I'd believe there
might be embedded mostly-POSIX-compliant environments still that don't
do virtual memory, but I've never seen one.

zw

[Ralf Corsepius]

Post by Zack Weinberg

Post by Ralf Corsepius
aio.h and sys/mman.h are not available everywhere

aio.h certainly, but are you aware of a *specific Unix-like system*
which is in current use and does not provide <sys/mman.h>?

Unix-like! Not all OSes are Unix-like.

Most prominent one without mman.h is mingw. Others, I am aware about are
certain newlib based toolchains.

Newlib also is one of the places which has a history of struggling with
the _POSIX_*-defines (Also affects Cygwin).

Post by Zack Weinberg
That header
is mandatory in POSIX-2008 (although not before then) and the last time
I can remember encountering a workstation-scale Unix environment that
didn't have it was in the mid-nineties sometime. I'd believe there
might be embedded mostly-POSIX-compliant environments still that don't
do virtual memory, but I've never seen one.

Correct - RTEMS is one of these. We have a rudimentary implementation of
aio.h since 2010 and don't have mmap.h at all.

Ralf

[Zack Weinberg]

On Sat, Mar 16, 2013 at 1:06 AM, Ralf Corsepius

Post by Ralf Corsepius

Post by Zack Weinberg

Post by Ralf Corsepius
aio.h and sys/mman.h are not available everywhere

aio.h certainly, but are you aware of a *specific Unix-like system*
which is in current use and does not provide <sys/mman.h>?

Unix-like! Not all OSes are Unix-like.

The whole point of this project is to pin down the remaining variation
among OSes that *are* Unix-like, because IME that is the case where
AC_CHECK_HEADERS is most useful.

Post by Ralf Corsepius
Most prominent one without mman.h is mingw. Others, I am aware about are
certain newlib based toolchains.

This may be skewed by the software I've been working on lately (very
network-intensive) but portability to anything Windows-based seems
best accomplished by conditioning stuff on #ifdef _WIN32. In
particular, I don't see any real value in using AC_CHECK_HEADERS to
figure out whether you need sys/socket.h and friends or winsock2.h,
because there's going to be variation scattered through the code as
well, and _WIN32 makes a reliable master switch for all of it.
Similarly, _WIN32 will tell you whether to use sys/mman.h and mmap or
windows.h and MapViewOfFile.

That's why I wrote the original blog post the way I did: my logic is
that you can use _WIN32 to figure out if you've got Windows, and only
bother to do autoconf probes for things that are variable among
platforms that *aren't* Windows. I'd speculate that the same is true
for other decidedly non-POSIX environments; VMS, z/OS, and VxWorks
come to mind, altho I have no idea how relevant any of them still are.

I don't know which box to put newlib in, though. I know it's used in
embedded environments, but I don't know what the underlying OS(es?)
are like.

Post by Ralf Corsepius

Post by Zack Weinberg
I'd believe there
might be embedded mostly-POSIX-compliant environments still that don't
do virtual memory, but I've never seen one.

Correct - RTEMS is one of these. We have a rudimentary implementation of
aio.h since 2010 and don't have mmap.h at all.

If you send me header lists for RTEMS and any other non-Windows-based
embedded systems you think should be considered, I'll be happy to
update my tables. As I said elsethread, trailing-edge versions are
actually better than brand new ones for this analysis.

zw

[Zack Weinberg]

Post by Thomas Jahns

Post by Zack Weinberg
I've been doing research into the cross-platform availability of
header files that are commonly probed for in Autoconf scripts.
http://www.owlfolio.org/possibly-useful/notes-on-the-cross-platform-availability-of-header-files/

although you didn't ask: aio.h does not need to be probed for: unistd.h defines
_POSIX_ASYNCHRONOUS_IO when it's available.
See <http://pubs.opengroup.org/onlinepubs/007904875/basedefs/unistd.h.html> for
some other features that don't require autoconf to check for, sys/mman.h is in
the same category.

As Ralf pointed out, this mechanism is not reliable in practice.

Post by Thomas Jahns
With respect to your comment on identifiers with leading underscore: to be
conformant to e.g. C89 standards, an implementation must not define anything
without said leading underscore, but in cases where the non-standard
functionality is required to implement standard-mandated functionality, it's
easiest to have it available with a leading underscore.

The rule you cite applies only to headers that are part of C89, which
e.g. <fcntl.h> isn't. However, yes, this may have been what the folks
at MS were thinking back in the 1990s when they made their <fcntl.h> not
actually define what POSIX says it's supposed to define.

Post by Thomas Jahns
In case you still need it, I could check AIX 6.1 for any header you want to know
about or provide an ls -R of /usr/include

Yes, please. (Do I understand correctly that AIX 7 is the latest and
greatest in that line? This test is actually more interesting when
applied to older systems, as they define the baseline.)

The exact command you should run is

$ find /usr/include \( -type f -a -name \*.h \) -print |
sed s:/usr/include/:: | LC_ALL=C sort > headers-$(uname)-$(uname -r)

Don't worry about headers coming from third party libraries, I have
scripts for that.

zw

[Zack Weinberg]

Post by Zack Weinberg
I've been doing research into the cross-platform availability of
header files that are commonly probed for in Autoconf scripts.

Major update: The technique I was using for the blog post, and the
simplified technique I suggested here earlier, turned out to produce
significant errors. I've spent the past several days writing a
program that will compute header inventories _properly_. That program
is now ready for other people to poke at. Please see
<https://github.com/zackw/header-analysis>. In addition to running
scansys.py against OSes / embedded toolchains of interest, people who
remember elder days are requested to take a close look at the
'b-obsolete' and 'b-ucom' lists of obsolete and still-useful,
respectively, nonstandard but widely available header files and see if
there are any errors.

zw

[Trent Nelson]

Post by Zack Weinberg
1) What "interesting portability targets" have I left out? I only
went back in time as far as FreeBSD 7, and I didn't even try to get my
hands on any of the surviving proprietary Unixes; is this too
shortsighted?

I can set you up with access to ***@snakebite (see www.snakebite.net), which will provide you with access to the following platforms:

+-----------------------------------------------------------------------+
| Available Hosts |
| (Last Update: 2013-01-27 17:57:08Z) |
+-----------------------------------------------------------------------+
| Alias | OS | Arch | CPUs | RAM | Compilers |
+-------|-----------------|----------|--------------|-------|-----------+
| a7|AIX 7.1 | PowerPC | 2 @ 1.4GHz| 8GB|xlc121 |
| mg|Debian Wheezy | ARMv6l | 1 @ 700MHz| 256MB|gcc |
| d3x|DragonFly 3.0.2 | x86 | 1 @ 3.2GHz| 2GB|gcc |
| d3|DragonFly 3.0.2 | x64 | 1 @ 3.2GHz| 2GB|gcc |
| al|Fedora 14 | ARMv6l | 1 @ 700MHz| 256MB|gcc |
| f9|FreeBSD 9.1 | x64 | 1 @ 3.2GHz| 2GB|gcc |
| h2|HP-UX 11iv2 | PA-RISC | 2 @ 875MHz| 8GB|hpacc0395 |
| h3|HP-UX 11iv3 | Itanium2 | 4 @ 1.3GHz| 64GB|hpacc0626 |
| i6|IRIX 6.5.30 | MIPS | 4 @ 500MHz| 4GB|gcc34 |
| n51|NetBSD 5.1.2 | x64 | 1 @ 3.2GHz| 2GB|gcc |
| n51x|NetBSD 5.1.2 | x86 | 1 @ 3.2GHz| 2GB|gcc |
| x8|OS X 10.8 | x64 | 4x2 @ 3.2GHz| 32GB|clang4 |
| o51x|OpenBSD 5.1 | x86 | 1 @ 3.2GHz| 2GB|gcc |
| o51|OpenBSD 5.1 | x64 | 1 @ 3.2GHz| 2GB|gcc |
| s10|Solaris 10 | SPARC | 2 @ 1.2GHz| 8GB|suncc123 |
| s11|Solaris 11 | x64 | 2 @ 2.4GHz| 16GB|suncc123 |
| s9|Solaris 9 | SPARC | 2 @ 900MHz| 8GB|suncc123 |
| t5|Tru64 5.1B | Alpha | 4 @ 667MHz| 7GB|cc65 |
| p1|Ubuntu 12.10 | ARMv7 | 2 @ 1.2GHz| 1GB|gcc46 |
+-----------------------------------------------------------------------+
Enter alias (^C to exit):

Send me your ssh keys and we'll be good to go.

(This offer is open to any and all autoconf committers (or notable non-committer contributors, like Zack in this case). I sent an e-mail a few months ago inviting autoconf to use Snakebite but unfortunately didn't hear anything back :/)

Regards,

Trent.