Discussion:
How to change the shebang in 'configure' to require Bash
R. Diez
2018-03-18 15:39:16 UTC
Permalink
Hi all:

I find it hard to write POSIX-compliant shell scripts. The lack of the
'pipefail' flag is specially problematic. I have described this issue here:

http://rdiez.shoutwiki.com/wiki/Error_Handling_in_General_and_C%2B%2B_Exceptions_in_Particular#Set_the_pipefail_flag

Therefore, I write most of my shell scripts exclusively for Bash. I also
find some bashisms very convenient. I only test my scripts against Bash
anyway, as I haven't got the time nor the inclination for more extensive
testing. I never had a big problem in this respect, as the Bash shell is
everywhere.

I find configure.ac scripts even harder to write than normal Bash
scripts. Furthermore, ShellCheck cannot analyse them. Therefore, I want
to limit my Autoconf 'configure' scripts to Bash.

I realise that this decision goes against Autoconf's portability
principle. But even as a humble user, I am entitled to some freedom of
choice. 8-)

I could write some code in configure.ac to detect the current shell. But
is there any way to tell Autoconf to output a Bash shebang in the
generated 'configure' script, instead of the standard "#! /bin/sh".

I read about variable CONFIG_SHELL, but that looks like it's meant for
the user. Or can I set that variable inside configure.ac?

Please copy me on all the answers, as I am not subscribed to this
mailing list.

Thanks in advance,
rdiez
Eric Blake
2018-03-19 13:57:42 UTC
Permalink
Post by R. Diez
I realise that this decision goes against Autoconf's portability
principle. But even as a humble user, I am entitled to some freedom of
choice. 8-)
I could write some code in configure.ac to detect the current shell. But
is there any way to tell Autoconf to output a Bash shebang in the
generated 'configure' script, instead of the standard "#! /bin/sh".
Not any easy way that I know of, precisely because it goes against
autoconf's portability. If you're going to write in plain bash rather
than a portable shell script, then do you need the additional layer of
autoconf in the first place?
Post by R. Diez
I read about variable CONFIG_SHELL, but that looks like it's meant for
the user. Or can I set that variable inside configure.ac?
If you want to muck in the internals of autoconf, lib/m4sugar/m4sh.m4
emits the shebang lines during AS_INIT_GENERATED (for subsidiary scripts
produced by the primary, such as config.status), and during AS_INIT (for
the primary script, such as configure). Those macros are not set up to
easily hook in a replacement, so you'd have to rewrite your m4 library
to redefine those macros the way you want. If you think that other
people would commonly want to hook in a different default shell than
/bin/sh, you're welcome to turn it into a formal patch against autoconf
instead of something you just override locally, but I'm not yet
convinced that it is common to throw away portability.
Post by R. Diez
Please copy me on all the answers, as I am not subscribed to this
mailing list.
It's already list policy to reply-to-all, precisely for that reason.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3266
Virtualization: qemu.org | libvirt.org
Warren Young
2018-03-19 21:34:53 UTC
Permalink
I realise that this decision goes against Autoconf's portability principle. But even as a humble user, I am entitled to some freedom of choice. 8-)
Not any easy way that I know of, precisely because it goes against autoconf's portability.
I saw in the recent Automake release notes that there’s a tentative plan that Autoconf 2.0 will require at least a POSIX shell.

Herr Diez, is this close enough to “Bash” to suit your purposes? That is, are you just looking for things like $() instead of ``, or are you intending to commit outright bash-isms?
R. Diez
2018-03-20 10:07:49 UTC
Permalink
Post by Warren Young
I saw in the recent Automake release notes that there’s a tentative plan that
Autoconf 2.0 will require at least a POSIX shell.
Herr Diez, is this close enough to “Bash” to suit your purposes?
That is, are you just looking for things like $() instead of ``, > or are you intending to commit outright bash-isms?
First of all, thanks for your answer.

I know that Autoconf developers want to write extremely portable
scripts. But that is a huge barrier for somebody like me. I am a user,
not a shell expert.

I have heard that the POSIX standard, or maybe its implementation, is
not exactly the same across all shells. There are subtle differences.

The generated 'configure' script changes shells on start-up. The logic
is not documented. I looked inside, and it was not immediately obvious
on what grounds some shells are preferred. Depending on what is
installed, I seems possible that zsh is taken, for example.

I am unwilling to test my configure.ac on several shells. Furthermore,
the features in the POSIX standard are very limited. I am using arrays
and regular expressions, for example. An my scripts are not really that
complex.

In any case, the behaviour of my configure.ac script cannot depend on
what shells the end-user has currently installed. I do not know of any
shell linters like ShellCheck that accept configure.ac files.

I would be very happy if my software ran on 10 % of the platforms where
Bash is installed by default. Therefore, demanding Bash is for me the
best solution.

In fact, it is the only sane solution. Not even hard-core Autoconf
developers should take such risks. My guess is that Autoconf-generated
'configure' scripts usually work because they tend to switch to Bash on
start-up.

Even if you really wanted, there is no way to properly test your
configure.ac scripts. Autoconf requires a Bourne shell, but exactly what
is required, is not documented. There is no guide as to what shells and
shell versions you should be testing against.

Autoconf works because most bugs have been reported and fixed over many
years. But trying to change the shell requirement now is probably a big
nightmare.

Requiring a POSIX shell in the next version is an improvement, but POSIX
is too limiting to really help. It is 2018. No wonder so many people
want to ditch Autoconf!

Best regards,
R. Diez
Russell Shaw
2018-03-20 11:53:24 UTC
Permalink
On 20/03/18 21:07, R. Diez wrote:
...
Requiring a POSIX shell in the next version is an improvement, but POSIX is too
limiting to really help. It is 2018. No wonder so many people want to ditch
Autoconf!
Autoconf is hard to learn because becoming properly familiar with shell
programming is hard enough for new shell hackers, let alone figuring out the
several versions of Bourne-like shells around and their extensions, plus reading
all that posix standards stuff, then having to read all about and mastering M4.

An alternative way to portability would be to ditch M4 so that everything is
done directly in a shell that comes with autoconf. The shell could have inbuilt
functions for the common things needed like regular expressions etc.
Russell Shaw
2018-03-20 12:01:15 UTC
Permalink
Post by Russell Shaw
...
Requiring a POSIX shell in the next version is an improvement, but POSIX is
too limiting to really help. It is 2018. No wonder so many people want to
ditch Autoconf!
Autoconf is hard to learn because becoming properly familiar with shell
programming is hard enough for new shell hackers, let alone figuring out the
several versions of Bourne-like shells around and their extensions, plus reading
all that posix standards stuff, then having to read all about and mastering M4.
An alternative way to portability would be to ditch M4 so that everything is
done directly in a shell that comes with autoconf. The shell could have inbuilt
functions for the common things needed like regular expressions etc.
I'll need to reword that.

If autoconf'd programs required the end users to have an installation-shell that
is ported to all systems of interest, then the learning requirements of autoconf
users should be a lot less.
Bob Friesenhahn
2018-03-20 13:32:54 UTC
Permalink
Post by Russell Shaw
If autoconf'd programs required the end users to have an installation-shell
that is ported to all systems of interest, then the learning requirements of
autoconf users should be a lot less.
This has all been discussed many times before. If it had been decided
that installing some other package would be a prerequisite for
running configure, then this issue would not exist. However, that
decision was never made. The precursor to Autoconf was Perl's
Configure, which was/is shell-based.

If some other package could be a prerequisite then making the
prerequisite a Bourne like shell with a m4 pre-processor would be a
silly choice since shell scripts are a very poor choice to meet
configure's requirements.

A specially-crafted language syntax would be a better choice to meet
configure requirements in order to avoid obscure and inefficient
m4/shell/sed syntax.

Regardless, Autoconf ultimately works well and the shell script part
of the effort that a configure script implementor must create is
usually not very difficult at all.

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Warren Young
2018-03-21 00:43:56 UTC
Permalink
I know that Autoconf developers want to write extremely portable scripts. But that is a huge barrier for somebody like me. I am a user, not a shell expert.
Your argument attempts to have it both ways: Bourne and POSIX shell is too old to still be used, you say, but learning old shells is too difficult.

The thing is, these old shells were state of the art on systems so small by today’s standards that they barely even qualify as computers:

- The acme of individually-accessible computing when the Bourne shell was high technology was the PDP-11, which couldn’t even achieve 1 MIPS on benchmarks so old now that we don’t even use them any more. (“1 MIPS” was defined in terms of the PDP-11’s successor, the VAX 11/780.)

- The acme when POSIX was first ratified was the SPARCstation 1, which was outdone by Palm PDAs at their height of popularity. A smartwatch runs circles around the old PDAs now.

So, could software meant for those machines really have been all that complex?
I have heard that the POSIX standard, or maybe its implementation, is not exactly the same across all shells. There are subtle differences.
Doubtless one could find multiple odd corner cases still in existence, yet we find that shell scripts are broadly portable in practice. How do you square that with the FUD line?

All of the shell incompatibilities I’ve come across have been in the other direction: someone raised on nothing but Bash-based systems writes some Bash-ism into a script, puts a #!/bin/sh at the top, then releases it to the world, where it breaks everywhere that /bin/sh is not Bash.
The generated 'configure' script changes shells on start-up. The logic is not documented. I looked inside, and it was not immediately obvious on what grounds some shells are preferred. Depending on what is installed, I seems possible that zsh is taken, for example.
What’s wrong with that? It’s upgrading from worse shells to be better shells for you, where possible.

Were you thinking that zsh is not compatible with the POSIX and Bourne shell languages, perhaps?
I am unwilling to test my configure.ac on several shells.
Are you willing to accept bug reports from your software’s users, then, when it won’t configure on their non-Bash based OS?
Furthermore, the features in the POSIX standard are very limited.
I remember when POSIX features were unattainable, being either difficult or effectively impossible to find everywhere I wanted my software to run.

I could get behind your argument if you were fighting for a *good* language, but Bash? What weak sauce! Bash is wonderful for what it is, but it’s horribly hampered by compatibility and nailed-in-place standards. If we’re going to throw out the requirement for least-common-demoninator standards, our options open up to many far better scripting languages.

Instead of fighting for Autoconf-with-Bash, consider Autosetup:

https://msteveb.github.io/autosetup/ <https://msteveb.github.io/autosetup/>

It ships with a minimal implementation of the Tcl programming language and enough smarts to bootstrap it on systems where a local Tcl doesn’t exist. Because it uses either a) your local Tcl, which is likely to be recent, or b) Jim Tcl <http://jim.tcl.tk/index.html/doc/www/www/index.html>, which understands a modern Tcl dialect, you aren’t tied to a decades-old version of the scripting language, as with Autoconf. Yet, the language is far cleaner than Bash 4, because it has a much simpler development history.

Now you’ve got some real power, using a regular, improving syntax.

Just to pick one example, modern Tcl has associative arrays, and has had them since 2003. Bash didn’t get them until 2009, but that’s not the salient point here. The important thing is that the POSIX shell doesn’t have them, so when it comes to portability, you can’t count on them. Autosetup sidesteps that problem by depending on a language that wasn’t nailed in place in 1989 in an effort to stop the Unix Wars, resulting in it not moving forward in a compatible way from that point.
I am using arrays and regular expressions, for example. An my scripts are not really that complex.
Tcl has those, too. :)
I do not know of any shell linters like ShellCheck that accept configure.ac files.
Analysis tools should accept configure scripts.

I wouldn’t expect an Autoconf-produced configure script to pass any reasonably strong linter, though. It’s machine-generated code, so it’s probably “horrible” from a bondage-and-discipline standpoint. It’s meant to be understood by shells, not by other humans.
I would be very happy if my software ran on 10 % of the platforms where Bash is installed by default.
There’s a wonderful idea in philosophy, which is that laws should be written as if legislators did not know their eventual place in society when they left the legislature. Think how different our laws would be if that were the case.

I think you’re suffering from the same problem as our legislators here, where you believe you get to choose both the rules and your software’s place in the world governed by those rules.

What if *I* get to choose the 10% of platforms your software will run on?

I wave my Wand of Will, and tell you that your software will run only on Slackware Linux, WSL, and pre-macOS versions of OS X. If you are not using one of these today, you must either switch platforms if you want to run your software.

Are you still happy with 10%?
In fact, it is the only sane solution.
Either you just labeled all of us successful Autotools users insane, or your logic has a hole in it.
My guess is that Autoconf-generated 'configure' scripts usually work because they tend to switch to Bash on start-up.
Your guess is uninformed, else no Autotools-based software would port to any system not based on Bash, yet this regularly does occur. In fact, that’s Autotools’ original purpose.
Even if you really wanted, there is no way to properly test your configure.ac scripts.
Sure there is: try it everywhere. In today’s worlds of near-free VMs and OS images, you have less excuse than ever before not to do that.

Before cheap VMs existed, we’d just put our software out on the Net and wait for the complaints to roll in, which we’d fix one by one, until our software was deemed “portable.”

Portability is not a static state of being.
Autoconf requires a Bourne shell, but exactly what is required, is not documented.
Sure it is. We have the source code to the 1979 Bourne shell. It’s right here:

https://github.com/dspinellis/unix-history-repo/tree/Research-V7-Snapshot-Development/usr/src/cmd/sh <https://github.com/dspinellis/unix-history-repo/tree/Research-V7-Snapshot-Development/usr/src/cmd/sh>

That software saw very wide distribution, so if you want “documents” instead of source code, I could bury you in documents, if I wanted to.

If you want software that runs on a modern system, I’d recommend “osh” from the Schily Tools, or as a fallback, the Heirloom Bourne Shell:

http://schilytools.sourceforge.net/bosh.html <http://schilytools.sourceforge.net/bosh.html>
http://heirloom.sourceforge.net/sh.html <http://heirloom.sourceforge.net/sh.html>
There is no guide as to what shells and shell versions you should be testing against.
I wrote an article that should get you oriented:

https://unix.stackexchange.com/a/145524
It is 2018. No wonder so many people want to ditch Autoconf!
Name me an alternative, and I can give you a list of problems with it, too.

As you can tell from my Autosetup advocacy above, I also have substantial interest in getting away from the Autotools, but even my latest fancy has its problems, the biggest being that it doesn’t work on Windows, short of being hosted in a Linux emulation environment like Cygwin or WSL.

Nothing’s perfect.

Bob Friesenhahn
2018-03-19 16:33:40 UTC
Permalink
I realise that this decision goes against Autoconf's portability principle.
But even as a humble user, I am entitled to some freedom of choice. 8-)
I could write some code in configure.ac to detect the current shell. But is
there any way to tell Autoconf to output a Bash shebang in the generated
'configure' script, instead of the standard "#! /bin/sh".
Users also have the freedom of choice to move on (to some other
software) when configure fails.

Bash is not always /bin/bash. It may be installed in other paths.

The configure script does some checks of the current shell and it is
not uncommon for it to decide to use bash rather than the initial
shell.

Given your feelings about Autoconf's configure, perhaps you should use
a configure look-alike as GNU lzip (written by Antonio Diaz Diaz)
does.

Bob
--
Bob Friesenhahn
***@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Loading...