Use system extensions conditionally?

[Fredrik Tolf]

Dear list,

I'm trying to write a program that can use custom stdio streams on both
Linux/glibc and FreeBSD, and to that end, I'm trying to modify to autoconf
script to properly detect these and turn on the necessary compiler flags
to support them.

On glibc, I need to define `_GNU_SOURCE' in order to gain access to
fopencookie() and cookie_io_functions_t. Autoconf neatly supports turning
this on with `AC_USE_SYSTEM_EXTENSIONS(_GNU_SOURCE)', but I can't quite
seem to figure out how to only do this when necessary. Currently, I'm
trying to do it like this:

HAS_FOPENCOOKIE=yes
AC_CHECK_FUNC(fopencookie, [AC_USE_SYSTEM_EXTENSIONS(_GNU_SOURCE)], [HAS_FOPENCOOKIE=no])
AC_CHECK_MEMBER([cookie_io_functions_t.read], [], [HAS_FOPENCOOKIE=no])

This /works/, per se, but autoconf complains loudly when I try to generate
the configure script, several times, that `AC_COMPILE_IFELSE was called
before AC_USE_SYSTEM_EXTENSIONS'. Apparently, it thinks it is really bad
practice to actually do any tests before turning on system extensions.

What should I do about this? Just ignore the warnings? Do the necessary
AC_DEFINE manually instead? (This seems ugly, since I'd need to define an
`AH_TEMPLATE` as well and all that.) Just turn on all extensions that I
may or may not use, unconditionally at the top of the file? Something
completely different?

Thanks for reading!

--
Fredrik Tolf

[Eric Blake]

Post by Fredrik Tolf
Dear list,
I'm trying to write a program that can use custom stdio streams on both
Linux/glibc and FreeBSD, and to that end, I'm trying to modify to
autoconf script to properly detect these and turn on the necessary
compiler flags to support them.
On glibc, I need to define `_GNU_SOURCE' in order to gain access to
fopencookie() and cookie_io_functions_t. Autoconf neatly supports
turning this on with `AC_USE_SYSTEM_EXTENSIONS(_GNU_SOURCE)', but I
can't quite seem to figure out how to only do this when necessary.

That's not how AC_USE_SYSTEM_EXTENSIONS works. The macro does not take
arguments. It is an all-or-none switch - if you use it, you use it
early on, to tell the configure script to enable ALL extensions on ALL
platforms at once (_GNU_SOURCE on Linux is one of these, but it also
enables extensions on other platforms).

Post by Fredrik Tolf
HAS_FOPENCOOKIE=yes
AC_CHECK_FUNC(fopencookie, [AC_USE_SYSTEM_EXTENSIONS(_GNU_SOURCE)],
[HAS_FOPENCOOKIE=no])
AC_CHECK_MEMBER([cookie_io_functions_t.read], [], [HAS_FOPENCOOKIE=no])

Rather, this should be:

AC_USE_SYSTEM_EXTENSIONS()
AC_CHECK_FUNC([fopencookie], [HAS_FOPENCOOKIE=yes], [HAS_FOPENCOOKIE=no])
AC_CHECK_MEMBER([cookie_io_functions_t.read], [], [HAS_FOPENCOOKIE=no])

Post by Fredrik Tolf
This /works/, per se, but autoconf complains loudly when I try to
generate the configure script, several times, that `AC_COMPILE_IFELSE
was called before AC_USE_SYSTEM_EXTENSIONS'. Apparently, it thinks it is
really bad practice to actually do any tests before turning on system
extensions.

Yes, it really is bad practice to mix compilation tests before turning
extensions with compilation tests done afterwards. Turning on
extensions is such a drastic change to the compilation environment that
you want ALL your tests to run under the same environment. Either you
plan on using extensions, and should enable them unconditionally up
front, or you do not need extensions, and should not use the macro.

Post by Fredrik Tolf
What should I do about this? Just ignore the warnings? Do the necessary
AC_DEFINE manually instead? (This seems ugly, since I'd need to define
an `AH_TEMPLATE` as well and all that.) Just turn on all extensions that
I may or may not use, unconditionally at the top of the file? Something
completely different?

Unconditionally call AC_USE_SYSTEM_EXTENSIONS() (note: no arguments) at
the top of the file (well, after the AC_INIT, but before any compilation
tests).

[Fredrik Tolf]

Post by Eric Blake

Post by Fredrik Tolf
On glibc, I need to define `_GNU_SOURCE' in order to gain access to
fopencookie() and cookie_io_functions_t. Autoconf neatly supports
turning this on with `AC_USE_SYSTEM_EXTENSIONS(_GNU_SOURCE)', but I
can't quite seem to figure out how to only do this when necessary.

That's not how AC_USE_SYSTEM_EXTENSIONS works. The macro does not take
arguments.

Oh, right. I seem to have read the docs a bit too fast. Thanks. :)

Post by Eric Blake
Yes, it really is bad practice to mix compilation tests before turning
extensions with compilation tests done afterwards. Turning on
extensions is such a drastic change to the compilation environment that
you want ALL your tests to run under the same environment. Either you
plan on using extensions, and should enable them unconditionally up
front, or you do not need extensions, and should not use the macro.

Hm. I see your reasoning, and I see where you're coming from. However, on
FreeBSD, I don't actually have to turn on any extensions to use the
corresponding functionality (funopen &c.). Isn't it ugly, then, to always
enable system extensions, regardless of environment?

--
Fredrik Tolf

[Eric Blake]

Post by Fredrik Tolf

Post by Eric Blake
Yes, it really is bad practice to mix compilation tests before turning
extensions with compilation tests done afterwards. Turning on
extensions is such a drastic change to the compilation environment that
you want ALL your tests to run under the same environment. Either you
plan on using extensions, and should enable them unconditionally up
front, or you do not need extensions, and should not use the macro.

Hm. I see your reasoning, and I see where you're coming from. However,
on FreeBSD, I don't actually have to turn on any extensions to use the
corresponding functionality (funopen &c.). Isn't it ugly, then, to
always enable system extensions, regardless of environment?

No. Most programs WANT to blindly turn on all extensions, regardless of
environment, because it is easier to write programs that can take
advantage of extensions, regardless of how cavalier the system is about
namespace pollution even when extensions are not requested. I could
argue that FreeBSD has a bug in that it is not strictly POSIX compliant
if funopen() is exposed by default when requesting POSIX compliant, but
when extensions are requested, the namespace pollution of exposing
funopen() is reasonable (but I'd actually have to test compilation on
FreeBSD when my C source pre-defines _POSIX_C_SOURCE prior to including
system headers, to determine whether funopen() is actually leaked when
requesting strict compliance).

There is a small set of projects that really DOES want to remain
completely POSIX-compliant (or even stricter ANSI-C compliant) - those
programs do NOT want to request extensions by default, so that they
minimize namespace pollution from system headers. But they tend to be
the minority.

In my own experience, projects that use gnulib almost always end up
using AC_USE_SYSTEM_EXTENSIONS under the hood, because gnulib is best
able to implement lots of functionality when it can rely on system
extensions.

[Fredrik Tolf]

Post by Eric Blake

Isn't it ugly, then, to always enable system extensions, regardless of
environment?

No. Most programs WANT to blindly turn on all extensions, regardless of
environment, because it is easier to write programs that can take
advantage of extensions, regardless of how cavalier the system is about
namespace pollution even when extensions are not requested.
[...]
There is a small set of projects that really DOES want to remain
completely POSIX-compliant (or even stricter ANSI-C compliant) - those
programs do NOT want to request extensions by default, so that they
minimize namespace pollution from system headers. But they tend to be
the minority.

Thanks for your reply!

I've been mulling over it for a while now, but I still don't think I quite
get it. In particular, I'm still not in the clear about what
AC_USE_SYSTEM_EXTENSIONS really does.

I read what you write kind of as if systems shouldn't expose anything
non-POSIX as long as it isn't used, but this is clearly not the case; not
only with funopen() on *BSD, but also with e.g. epoll() on GNU/Linux, or
just such things as GCC C extensions.

As such, it seems to me that there are basically three different modes of
operation: Standards-compliant mode, enabled explicitly by setting
POSIXLY_CORRECT or similar; extensions mode, enabled explicitly by using
AC_USE_SYSTEM_EXTENSIONS; and also "default" mode, which is what we have
if we don't do any of those.

Seeing as how there are a lot of extensions available in default mode, I
don't really see what the formal distinction between default mode and
extensions mode. I'm willing to buy that there is no formally defined
distinction between the two, and that the, perhaps arguably sloppy, nature
of systems development leaves things a bit unclear. But if that is so,
isn't it still ugly to have to use AC_USE_SYSTEM_EXTENSIONS always, on all
systems, rather than being left the option of using whatever the system
considers to be its "default environment"?

--
Fredrik Tolf

[Paul Eggert]

Post by Fredrik Tolf
isn't it still ugly to have to use AC_USE_SYSTEM_EXTENSIONS always

Part of the problem is that people occasionally need to set compiler
options saying "I want C99" (or whatever), and these options sometimes
disable parts of the default environment (to conform to C99 or
whatever), and so one needs to turn on extensions anyway, just to get
the default environment back (plus some more, but the differences don't
matter).

[Eric Blake]

Post by Fredrik Tolf

Post by Eric Blake

Isn't it ugly, then, to always enable system extensions, regardless
of environment?

No. Most programs WANT to blindly turn on all extensions, regardless of
environment, because it is easier to write programs that can take
advantage of extensions, regardless of how cavalier the system is about
namespace pollution even when extensions are not requested.
[...]
There is a small set of projects that really DOES want to remain
completely POSIX-compliant (or even stricter ANSI-C compliant) - those
programs do NOT want to request extensions by default, so that they
minimize namespace pollution from system headers. But they tend to be
the minority.

Thanks for your reply!
I've been mulling over it for a while now, but I still don't think I
quite get it. In particular, I'm still not in the clear about what
AC_USE_SYSTEM_EXTENSIONS really does.
I read what you write kind of as if systems shouldn't expose anything
non-POSIX as long as it isn't used, but this is clearly not the case;
not only with funopen() on *BSD, but also with e.g. epoll() on
GNU/Linux, or just such things as GCC C extensions.

How do you get epoll() on GNU/Linux? By including <sys/epoll.h>, but
that's a header that is not mentioned in POSIX. Therefore, a strictly
conforming program CANNOT use this interface, and therefore there is no
reason to gate that header by _GNU_SOURCE, therefore, there is no need
to use AC_USE_SYSTEM_EXTENSIONS to make use of the interface.

On the other hand, BSD's funopen() is exposed in <stdio.h>, which IS a
standard header; but the name funopen() is not in the list of reserved
name patterns in POSIX. Therefore, if BSD wants to comply with POSIX,
then when _POSIX_C_SOURCE is defined before including <stdio.h>, then
funopen() must NOT be visible to a user of <stdio.h>. Now, not everyone
cares about being strictly POSIX-compliant, so the default behavior,
when you don't request extensions, might have names leaked into the
environment. But on the converse side, when you KNOW you want to use
extensions, you are best off to explicitly request those extensions.
And if you are going to use non-POSIX extensions, you might as well use
all extensions on all platforms, all in one go.

Post by Fredrik Tolf
As such, it seems to me that there are basically three different modes
of operation: Standards-compliant mode, enabled explicitly by setting
POSIXLY_CORRECT or similar; extensions mode, enabled explicitly by using
AC_USE_SYSTEM_EXTENSIONS; and also "default" mode, which is what we have
if we don't do any of those.

Yes, a fairly reasonable summary (setting POSIXLY_CORRECT alone is not
enough to guarantee a strictly conforming environment, but might be part
of the steps on some platforms, confstr() can be used to determine
whether POSIXLY_CORRECT is required. Another required step in a
strictly conforming setup is to pre-define _POSIX_C_SOURCE to the
correct value prior to including any system headers).

Post by Fredrik Tolf
Seeing as how there are a lot of extensions available in default mode, I
don't really see what the formal distinction between default mode and
extensions mode. I'm willing to buy that there is no formally defined
distinction between the two, and that the, perhaps arguably sloppy,
nature of systems development leaves things a bit unclear. But if that
is so, isn't it still ugly to have to use AC_USE_SYSTEM_EXTENSIONS
always, on all systems, rather than being left the option of using
whatever the system considers to be its "default environment"?

glibc is one of the best systems out there that is actively trying to
minimize namespace pollution in default mode. BSD is slowly catching
up, but not there yet. Relying on the "default environment" is risky -
something that might be polluted into the namespace today could
disappear tomorrow if the system headers get patched to provide less
pollution. Explicitly requesting extensions is the only way to ensure
that your program that takes advantage of the extensions will not be
broken by changes to the default mode.

[Fredrik Tolf]

Post by Eric Blake

Post by Fredrik Tolf
I read what you write kind of as if systems shouldn't expose anything
non-POSIX as long as it isn't used, but this is clearly not the case;
not only with funopen() on *BSD, but also with e.g. epoll() on
GNU/Linux, or just such things as GCC C extensions.

How do you get epoll() on GNU/Linux? By including <sys/epoll.h>, but
that's a header that is not mentioned in POSIX.

True enough, my bad.

I guess my interpretation should be, then, that the default environment is
not so much about strictly hiding everything which is not in POSIX, but
rather about not breaking anything which should be valid under POSIX, for
example by not defining potentially conflicting symbols. That makes sense,
and also explains why GCC C extensions need not be explicitly hidden in
the default environment.

Post by Eric Blake
Relying on the "default environment" is risky - something that might be
polluted into the namespace today could disappear tomorrow if the system
headers get patched to provide less pollution. Explicitly requesting
extensions is the only way to ensure that your program that takes
advantage of the extensions will not be broken by changes to the default
mode.

Yes, I see your point now, and I think I finally understand why
AC_USE_SYSTEM_EXTENSIONS is all about.

Thanks a lot for the explanation!

--
Fredrik Tolf

[Russ Allbery]

Post by Eric Blake
On the other hand, BSD's funopen() is exposed in <stdio.h>, which IS a
standard header; but the name funopen() is not in the list of reserved
name patterns in POSIX. Therefore, if BSD wants to comply with POSIX,
then when _POSIX_C_SOURCE is defined before including <stdio.h>, then
funopen() must NOT be visible to a user of <stdio.h>. Now, not everyone
cares about being strictly POSIX-compliant, so the default behavior,
when you don't request extensions, might have names leaked into the
environment. But on the converse side, when you KNOW you want to use
extensions, you are best off to explicitly request those extensions.
And if you are going to use non-POSIX extensions, you might as well use
all extensions on all platforms, all in one go.

Of course, one of the drawbacks is that you have an all-or-nothing
situation. Either you have to stick to only POSIX interfaces, or you have
to give the local system carte blanche to stomp all over your namespace,
possibly resulting in code breakage down the line when some new local
extension is introduced that conflicts with your code.

This of course isn't a problem that Autoconf can solve. It's basically
inherent in how the feature test macros work. But it's worth being aware
of. All namespace bets are essentially off once you enable system
extensions.

[Nick Bowler]

Post by Russ Allbery

Post by Eric Blake
On the other hand, BSD's funopen() is exposed in <stdio.h>, which IS a
standard header; but the name funopen() is not in the list of reserved
name patterns in POSIX. Therefore, if BSD wants to comply with POSIX,
then when _POSIX_C_SOURCE is defined before including <stdio.h>, then
funopen() must NOT be visible to a user of <stdio.h>. Now, not everyone
cares about being strictly POSIX-compliant, so the default behavior,
when you don't request extensions, might have names leaked into the
environment. But on the converse side, when you KNOW you want to use
extensions, you are best off to explicitly request those extensions.
And if you are going to use non-POSIX extensions, you might as well use
all extensions on all platforms, all in one go.

Of course, one of the drawbacks is that you have an all-or-nothing
situation. Either you have to stick to only POSIX interfaces, or you have
to give the local system carte blanche to stomp all over your namespace,
possibly resulting in code breakage down the line when some new local
extension is introduced that conflicts with your code.
This of course isn't a problem that Autoconf can solve. It's basically
inherent in how the feature test macros work. But it's worth being aware
of. All namespace bets are essentially off once you enable system
extensions.

Even without the use of system extensions Autoconf makes no attempt to
enable standards-conformance mode in compilers, and most of them stomp
all over the application namespace by default. So all namespace bets
are off with or without AC_ENABLE_SYSTEM_EXTENSIONS.

For a concrete example, try to declare a variable called "unix" on a
typical GNU/Linux system...

Cheers,