[GNU Autoconf 2.69] OSX autotools: `aclocal.m4' not being output by `autom4te'

[Jeffrey Sheen]

Dear lists,

I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.

Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.

If you could find the time to take a look, it'd be much appreciated.

Cheers,

Jeff.

Dear autoconf list,
I am having trouble using the GNU Autotools package under OSX Mavericks,
and was hoping you may be able to help. If this not the correct mailing
list, then please accept my apologies.
When running `aclocal', the expected `aclocal.m4' is not being output, and
subsequent `glibtoolize' and `autoconf' commands are creating unexpected
`m4trace' output at the top of my generated `configure' script.
I have opened a Stackoverflow question regarding this, so if I am
addressing the correct mailing list, please would you take a look at the
details?
http://stackoverflow.com/questions/24326419/osx-autotools-aclocal-m4-not-being-output-by-autom4te
I've been bashing my head against this for weeks, and have not managed to
find a similar issue reported with Google searching.
Any help would be greatly appreciated.
Cheers,
Jeff.

[Jeffrey Sheen]

Please also find `testsuite.log' attached.

Post by Jeffrey Sheen
Dear lists,
I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.
Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.
If you could find the time to take a look, it'd be much appreciated.
Cheers,
Jeff.

Dear autoconf list,
I am having trouble using the GNU Autotools package under OSX Mavericks,
and was hoping you may be able to help. If this not the correct mailing
list, then please accept my apologies.
When running `aclocal', the expected `aclocal.m4' is not being output,
and subsequent `glibtoolize' and `autoconf' commands are creating
unexpected `m4trace' output at the top of my generated `configure' script.
I have opened a Stackoverflow question regarding this, so if I am
addressing the correct mailing list, please would you take a look at the
details?
http://stackoverflow.com/questions/24326419/osx-autotools-aclocal-m4-not-being-output-by-autom4te
I've been bashing my head against this for weeks, and have not managed to
find a similar issue reported with Google searching.
Any help would be greatly appreciated.
Cheers,
Jeff.

[Eric Blake]

Post by Jeffrey Sheen
Dear lists,
I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.

You definitely have a broken setup, but I'm not sure what's causing it.

Post by Jeffrey Sheen
Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.
If you could find the time to take a look, it'd be much appreciated.
# -*- compilation -*-
5. tools.at:145: testing autom4te and whitespace in file names ...
./tools.at:163: mkdir "$dir" "$cachedir" "$TMPDIR" && touch "$file" || exit 77
./tools.at:174: autom4te -C "$cachedir" -B "$dir" --language=m4sugar -o "$outfile" "$file"
./tools.at:175: cat "$outfile"
--- - 2014-06-20 15:46:02.000000000 +0100
+++ /Volumes/DATA/filestore/development/gnu/autoconf/extract/autoconf-2.69/tests/testsuite.dir/at-groups/5/stdout 2014-06-20 15:46:02.000000000 +0100
@@ -1,2 +1,5 @@
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^_?m4_])
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^dnl$])
+m4trace: file with funny ' $x & #! name:1: -1- m4_include([foo.m4])
bar

We need to figure out what is causing those spurious 'm4trace: ' lines
to be output at the beginning of every run of autom4te. Do you have any
environment variables or config.site that might be interfering with
normal operation of the tools, by causing autom4te to behave as if it
were being requested to trace macros instead of do its normal job?

[Jeffrey Sheen]

Thanks Eric.

I'll check the man pages for 'autom4te' to see which environment variables can set its execution mode.

Cheers,

Jeff

Post by Eric Blake

Post by Jeffrey Sheen
Dear lists,
I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.

You definitely have a broken setup, but I'm not sure what's causing it.

Post by Jeffrey Sheen
Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.
If you could find the time to take a look, it'd be much appreciated.
# -*- compilation -*-
5. tools.at:145: testing autom4te and whitespace in file names ...
./tools.at:163: mkdir "$dir" "$cachedir" "$TMPDIR" && touch "$file" || exit 77
./tools.at:174: autom4te -C "$cachedir" -B "$dir" --language=m4sugar -o "$outfile" "$file"
./tools.at:175: cat "$outfile"
--- - 2014-06-20 15:46:02.000000000 +0100
+++ /Volumes/DATA/filestore/development/gnu/autoconf/extract/autoconf-2.69/tests/testsuite.dir/at-groups/5/stdout 2014-06-20 15:46:02.000000000 +0100
@@ -1,2 +1,5 @@
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^_?m4_])
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^dnl$])
+m4trace: file with funny ' $x & #! name:1: -1- m4_include([foo.m4])
bar

We need to figure out what is causing those spurious 'm4trace: ' lines
to be output at the beginning of every run of autom4te. Do you have any
environment variables or config.site that might be interfering with
normal operation of the tools, by causing autom4te to behave as if it
were being requested to trace macros instead of do its normal job?
--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[Jeffrey Sheen]

Hi Eric,

`printenv' yields:

TERM_PROGRAM=Apple_Terminal

TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/qy/t7gq5bws7zjd95y5p5x_sw0w0000gn/T/
Apple_PubSub_Socket_Render=/tmp/launch-lfrgeG/Render
TERM_PROGRAM_VERSION=326
TERM_SESSION_ID=2B7B0578-7BAD-4F67-BE94-158A86ED6F89
USER=jeff
SSH_AUTH_SOCK=/tmp/launch-7NOuJW/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0:2
__CHECKFIX1436934=1
PATH=/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
PWD=/Volumes/DATA/filestore/development/c/libs/freetype2/git/freetype2
DBUS_LAUNCHD_SESSION_BUS_SOCKET=/tmp/launch-d00lFv/unix_domain_listener
LANG=en_GB.UTF-8
HOME=/Users/jeff
SHLVL=1
LOGNAME=jeff
DISPLAY=/tmp/launch-jyxUVQ/org.macosforge.xquartz:0
_=/usr/bin/printenv

There is nothing Autotools specific that I can see there.

Also, there is no `./share/config.site' under the prefix directory of
`/tmp/'.

If there is no other way that the `autom4te' command is being manually told
to output `m4trace' messages, then what else could be causing it? Could the
binary be being configured incorrectly when it is built?

Cheers,

Jeff.

Thanks Eric.
I'll check the man pages for 'autom4te' to see which environment variables
can set its execution mode.
Cheers,
Jeff

Post by Eric Blake

Post by Jeffrey Sheen
Dear lists,
I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.

You definitely have a broken setup, but I'm not sure what's causing it.

Post by Jeffrey Sheen
Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.
If you could find the time to take a look, it'd be much appreciated.
# -*- compilation -*-
5. tools.at:145: testing autom4te and whitespace in file names ...
./tools.at:163: mkdir "$dir" "$cachedir" "$TMPDIR" && touch "$file" ||

exit 77

Post by Eric Blake

Post by Jeffrey Sheen
./tools.at:174: autom4te -C "$cachedir" -B "$dir" --language=m4sugar

-o "$outfile" "$file"

Post by Eric Blake

Post by Jeffrey Sheen
./tools.at:175: cat "$outfile"
--- - 2014-06-20 15:46:02.000000000 +0100
+++

/Volumes/DATA/filestore/development/gnu/autoconf/extract/autoconf-2.69/tests/testsuite.dir/at-groups/5/stdout
2014-06-20 15:46:02.000000000 +0100

Post by Eric Blake

Post by Jeffrey Sheen
@@ -1,2 +1,5 @@
+m4trace: file with funny ' $x & #! name:1: -1-

m4_pattern_forbid([^_?m4_])

Post by Eric Blake

Post by Jeffrey Sheen
+m4trace: file with funny ' $x & #! name:1: -1-

m4_pattern_forbid([^dnl$])

Post by Eric Blake

Post by Jeffrey Sheen
+m4trace: file with funny ' $x & #! name:1: -1- m4_include([foo.m4])
bar

We need to figure out what is causing those spurious 'm4trace: ' lines
to be output at the beginning of every run of autom4te. Do you have any
environment variables or config.site that might be interfering with
normal operation of the tools, by causing autom4te to behave as if it
were being requested to trace macros instead of do its normal job?
--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[Eric Blake]

Eric,
I know what is causing the issue.
I tried a fresh Mavericks install with just XCode 5.1.1 and Macports installed, and encountered the exact same problem.
I then realised that the only remaining difference between my environment and a vanilla environment, is that my development directories are on a FAT32 disk.

Thanks for the analysis. FAT is a notoriously bad file system, for how
non-POSIX compliant it is.

Indeed, the Autotools configuration worked as expected with a copy of the source tree stored on an HFS disk.
Is it expected behaviour that GNU Autotools are incompatible with some file systems natively supported by the OS?

Not entirely expected, although it may just be a bug in the testsuite
(for expecting too much from the environment) rather than a problem with
the just-built autoconf is still usable. But to avoid test failures,
we'd need patches from someone that is actively in such a setup (I don't
have access to such a system at the moment, so it would be a while to
wait if you are expecting me to try and come up with such patches).

[Jeff Sheen]

Eric,

I know what is causing the issue.

I tried a fresh Mavericks install with just XCode 5.1.1 and Macports installed, and encountered the exact same problem.

I then realised that the only remaining difference between my environment and a vanilla environment, is that my development directories are on a FAT32 disk.

Indeed, the Autotools configuration worked as expected with a copy of the source tree stored on an HFS disk.

Is it expected behaviour that GNU Autotools are incompatible with some file systems natively supported by the OS?

Cheers,

Jeff.

Post by Jeffrey Sheen
Hi Eric,

Post by Jeffrey Sheen
TERM_PROGRAM=Apple_Terminal
TERM=xterm-256color
SHELL=/bin/bash
TMPDIR=/var/folders/qy/t7gq5bws7zjd95y5p5x_sw0w0000gn/T/
Apple_PubSub_Socket_Render=/tmp/launch-lfrgeG/Render
TERM_PROGRAM_VERSION=326
TERM_SESSION_ID=2B7B0578-7BAD-4F67-BE94-158A86ED6F89
USER=jeff
SSH_AUTH_SOCK=/tmp/launch-7NOuJW/Listeners
__CF_USER_TEXT_ENCODING=0x1F5:0:2
__CHECKFIX1436934=1
PATH=/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/opt/local/bin:/opt/local/sbin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/opt/X11/bin
PWD=/Volumes/DATA/filestore/development/c/libs/freetype2/git/freetype2
DBUS_LAUNCHD_SESSION_BUS_SOCKET=/tmp/launch-d00lFv/unix_domain_listener
LANG=en_GB.UTF-8
HOME=/Users/jeff
SHLVL=1
LOGNAME=jeff
DISPLAY=/tmp/launch-jyxUVQ/org.macosforge.xquartz:0
_=/usr/bin/printenv

There is nothing Autotools specific that I can see there.
Also, there is no `./share/config.site' under the prefix directory of `/tmp/'.
If there is no other way that the `autom4te' command is being manually told to output `m4trace' messages, then what else could be causing it? Could the binary be being configured incorrectly when it is built?
Cheers,
Jeff.

Post by Jeffrey Sheen
Thanks Eric.
I'll check the man pages for 'autom4te' to see which environment variables can set its execution mode.
Cheers,
Jeff

Post by Eric Blake

Post by Jeffrey Sheen
Dear lists,
I have run `testsuite' with the latest tar of Autoconf, and have many
failures reported.

You definitely have a broken setup, but I'm not sure what's causing it.

Post by Jeffrey Sheen
Please find the output logs attached for `make check' and `make
installcheck'. The errors in both are the same according to `diff'.
If you could find the time to take a look, it'd be much appreciated.
# -*- compilation -*-
5. tools.at:145: testing autom4te and whitespace in file names ...
./tools.at:163: mkdir "$dir" "$cachedir" "$TMPDIR" && touch "$file" || exit 77
./tools.at:174: autom4te -C "$cachedir" -B "$dir" --language=m4sugar -o "$outfile" "$file"
./tools.at:175: cat "$outfile"
--- - 2014-06-20 15:46:02.000000000 +0100
+++ /Volumes/DATA/filestore/development/gnu/autoconf/extract/autoconf-2.69/tests/testsuite.dir/at-groups/5/stdout 2014-06-20 15:46:02.000000000 +0100
@@ -1,2 +1,5 @@
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^_?m4_])
+m4trace: file with funny ' $x & #! name:1: -1- m4_pattern_forbid([^dnl$])
+m4trace: file with funny ' $x & #! name:1: -1- m4_include([foo.m4])
bar

We need to figure out what is causing those spurious 'm4trace: ' lines
to be output at the beginning of every run of autom4te. Do you have any
environment variables or config.site that might be interfering with
normal operation of the tools, by causing autom4te to behave as if it
were being requested to trace macros instead of do its normal job?
--
Eric Blake eblake redhat com [phone]
Libvirt virtualization library http://libvirt.org

[Bob Friesenhahn]

Is it expected behaviour that GNU Autotools are incompatible with some file systems natively supported by the OS?

FAT32 provides only one second file timestamp resolution. That can
cause significant problems for modern computers which often perform
several steps in one second. A test to see if a file has changed may
fail to obtain the correct results.

Bob

[Jeff Sheen]

N.B. despite FAT's age, and subsequent flaws, it is still the de facto standard for portable drives.

This is because FAT is the only file system fully supported by all modern operating systems.

I run a multi-OS development environment, with a spread of tools across OSX and Windows. I wanted to migrate away from using FAT on my shared data partition, but found no other viable options. The process of testing alternative file systems was extensive, and meticulous. I spent weeks trying out NTFS and HFS partitions, with different combinations of drivers in both operating systems.

Ultimately, it is partisan nonsense that the only file system that can be agreed on is FAT, but that is the reality.

Jeff.

Is it expected behaviour that GNU Autotools are incompatible with some file systems natively supported by the OS?

FAT32 provides only one second file timestamp resolution. That can cause significant problems for modern computers which often perform several steps in one second. A test to see if a file has changed may fail to obtain the correct results.
Bob
--
Bob Friesenhahn
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/

[Gary V. Vaughan]

Post by Jeff Sheen

Is it expected behaviour that GNU Autotools are incompatible with some file systems natively supported by the OS?

FAT32 provides only one second file timestamp resolution. That can cause significant problems for modern computers which often perform several steps in one second. A test to see if a file has changed may fail to obtain the correct results.

N.B. despite FAT's age, and subsequent flaws, it is still the de facto standard for portable drives.
This is because FAT is the only file system fully supported by all modern operating systems.
I run a multi-OS development environment, with a spread of tools across OSX and Windows. I wanted to migrate away from using FAT on my shared data partition, but found no other viable options. The process of testing alternative file systems was extensive, and meticulous. I spent weeks trying out NTFS and HFS partitions, with different combinations of drivers in both operating systems.
Ultimately, it is partisan nonsense that the only file system that can be agreed on is FAT, but that is the reality.

Patient: It hurts when I do this.
Doctor: Well, then don't do that!

There really are many, many more elegant solutions than sharing files using FAT!

- use a NAS with a proper filesystem;
- mount a proper filesystem to one server and then run NFS on that server so that client machines can access it;
- use SMB on windows and Samba on Unix to cross mount a NTFS share;
- sychronize shared files using DropBox or Box.net or OwnCloud etc;
- replicate the filesystem across architectures and synchronize with rsync or git or mercurial or bzr;
- create a TrueCrypt volume if you're moving a single physical drive between machines;
- or a PKZip volume which also preserves metadata far better than FAT;

And that's just the ones I could brainstorm in the time it took me to type this up, there are surely many others.

HTH,

[Werner LEMBERG]

Post by Gary V. Vaughan

Post by Jeff Sheen
Ultimately, it is partisan nonsense that the only file system that
can be agreed on is FAT, but that is the reality.

There really are many, many more elegant solutions than sharing
files using FAT! [...]

Note, however, that failure of FAT *is not obvious* for the casual
user! You have to explicitly know this. So please add a line or two
to the autoconf documentation that there might be serious issues with
FAT due to coarse time stamps.


Werner

[Jeffrey Sheen]

Thanks for the suggestions Gary.

I do run a NAS on my LAN, and backup from my shared data partition to both
Dropbox and Copy (the only cloud storage clients that support symlinks over
multiple file system types). These both rely on network connectivity, where
the solution I require is a partition on local drive attached to the device.

Is it possible to have a local, NTFS data partition and access it with
Samba on a local boot partition? I would have thought that fundamentally
you'd need a layer converting Samba I/O calls into NTFS calls, in which
case, why not address the NTFS partition directly (functionality which does
not exist in OSX).

Synchronising data across multiple local partitions is certainly not more
elegant than a single, shared data partition, and would require multiple
times the redundancy of storage.

With regards to TrueCrypt and PKZip, are these not application layer
implementations that create files on top of a file system? If this is not
the case, and they have their own file system implementations, then let me
know and I will test them.

Without a viable alternative, I will continue to use a FAT data partition,
and temporarily copy source trees onto my boot partition when executing GNU
Autotools over them. I can then copy them back, and proceed normally.

Cheers,

Jeff.

Post by Werner LEMBERG

Post by Gary V. Vaughan

Post by Jeff Sheen
Ultimately, it is partisan nonsense that the only file system that
can be agreed on is FAT, but that is the reality.

There really are many, many more elegant solutions than sharing
files using FAT! [...]

Note, however, that failure of FAT *is not obvious* for the casual
user! You have to explicitly know this. So please add a line or two
to the autoconf documentation that there might be serious issues with
FAT due to coarse time stamps.
Werner

[Gary V. Vaughan]

Hi Jeff,

Post by Jeffrey Sheen
Thanks for the suggestions Gary.

No problem.

Post by Jeffrey Sheen
I do run a NAS on my LAN, and backup from my shared data partition to both Dropbox and Copy (the only cloud storage clients that support symlinks over multiple file system types). These both rely on network connectivity, where the solution I require is a partition on local drive attached to the device.

Assuming this means you need to unmount, detach, walk to another machine, plug, mount and then access... you still have a few better options than FAT, I think.

Post by Jeffrey Sheen
Is it possible to have a local, NTFS data partition and access it with Samba on a local boot partition? I would have thought that fundamentally you'd need a layer converting Samba I/O calls into NTFS calls, in which case, why not address the NTFS partition directly (functionality which does not exist in OSX).

You answered your own question :) OSX can mount windows shares, even though it can't read an NTFS filesystem (actually I did once find a way to mount NTFS read-only on Snow Leopard, so you might want to google that if it seems attractive in case it is still maintained and/or improved). But again, that requires at least an adhoc wifi network to connect the OSX client to the Windows SMB server, which you seem to have ruled out.

Post by Jeffrey Sheen
Synchronising data across multiple local partitions is certainly not more elegant than a single, shared data partition, and would require multiple times the redundancy of storage.

Well, that's hard to say without more context. But, I have all my open source projects checked in to local git repos on a dropbox share, and use them from dozens of heterogenous machines scattered across the globe all of which DropBox synchronizes invisibly for me... which suits me far better than FAT + sneaker net ;-) But, that only works because I know I'm only working on one machine at a time and don't have to worry about merge conflicts.

Post by Jeffrey Sheen
With regards to TrueCrypt and PKZip, are these not application layer implementations that create files on top of a file system? If this is not the case, and they have their own file system implementations, then let me know and I will test them.

They create a filesystem in a file on the host filesytem that contains contents of the packaged files + metadata (timestamps, ownership etc). You would either pkunzip into a local featureful filesystem, edit and then pkzip back into the host filesystem for sneaker net transport to the next machine, or in the case of TrueCrypt mount the embedded filesystem from either a disk image in the host fs, or from a whole disk image -- lots of details in the TrueCrypt docs. Be careful to avoid the deliberately brain-damaged most recent release though - and pick up 6.1a images for all the machines that need to read the TrueCrypt fs.

Post by Jeffrey Sheen
Without a viable alternative, I will continue to use a FAT data partition, and temporarily copy source trees onto my boot partition when executing GNU Autotools over them. I can then copy them back, and proceed normally.

In that case, what about hosting the transported files in a VM that boots with virtual box or vmware (or other multi-arch compatible VM image format) on each target machine and have the running VM export the shared files over sshfs/samba/nfs/all-3! for live mounting and editing on the target machine?

Docker also seems to run on Windows, according to its website: So a modern solution might be to wrap the whole thing up in a docker container on the external drive and deploy that on each target machine in turn for access to the files it contains. And with a little extra care you could easily incorporate snapshotting and/or backups into the process without additional tools.

Cheers,