Using 'make dist' with a 32 UID

Discussion:

Finucane, Stephen

2015-01-05 10:08:40 UTC

Autotools defaults to the 'v7' legacy tar format in GNU tar, through passing of the '-o' parameter to GNU tar. Enabling this option results in errors for users with 32 bit UIDs. For example, with the Open vSwitch package:

$ make dist
...
tardir=openvswitch-2.3.90 && ${TAR-tar} chof - "$tardir" | GZIP=--best gzip -c >openvswitch-2.3.90.tar.gz
tar: value 12345678 out of uid_t range 0..2097151
tar: Exiting with failure status due to previous errors
make[1]: Leaving directory `/development/ovs'
...

I managed to modify the Autoconf 'configure.ac' file to use the 'tar-ustar' format, which allow longer file names and other niceties. Again, with the Open vSwitch package:

diff --git a/configure.ac b/configure.ac
index ebb8b02..6505189 100644
--- a/configure.ac
+++ b/configure.ac
@@ -19,7 +19,7 @@ AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_TESTDIR([tests])
-AM_INIT_AUTOMAKE
+AM_INIT_AUTOMAKE([tar-ustar])

AC_PROG_CC_C99
AM_PROG_CC_C_O
--
1.7.4.1

This would appear to be an issue that would affect all users with 32 bit UIDs who wish to use the 'dist' target and its derivatives. Given this information, is modifying the 'configure.ac' file like above this the best approach to fix this issues? It fixes the issue but it would require modifying the 'configure.ac' file of all packages that use GNU Autotools. Perhaps there's some other way to enable this at runtime (i.e. './configure --enable-tar-ustar')?

Kind regards,

Stephen

PS: I have researched the mailing list, Google, and the source in my F20 machine for an answer to this question. Sadly, I did not find one.

Eric Blake

2015-01-05 16:27:45 UTC

Permalink

'make dist' is under the purview of automake, not autoconf. You may get
a better response by involving the automake list.

Post by Finucane, Stephen
$ make dist
...
tardir=openvswitch-2.3.90 && ${TAR-tar} chof - "$tardir" | GZIP=--best gzip -c >openvswitch-2.3.90.tar.gz
tar: value 12345678 out of uid_t range 0..2097151
tar: Exiting with failure status due to previous errors
make[1]: Leaving directory `/development/ovs'
...
diff --git a/configure.ac b/configure.ac
index ebb8b02..6505189 100644
--- a/configure.ac
+++ b/configure.ac
@@ -19,7 +19,7 @@ AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_TESTDIR([tests])
-AM_INIT_AUTOMAKE
+AM_INIT_AUTOMAKE([tar-ustar])

Are you proposing that we change the way autoconf is distributed? That
won't affect any other packages (you'd have to make the same patch for
each affected package), and so far, your code shows that you had
problems in building an openvswitch tarball, not an autoconf tarball.
Again, changing automake to do this automatically for ALL packages (once
those packages are built with a new enough automake) rather than trying
to patch one configure.ac for every affected package, seems like it
would be the better course of action.

I'm still open to be convinced that autoconf needs to alter its own
configure.ac, but I don't have enough evidence yet that it would make a
difference.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Finucane, Stephen

2015-01-05 17:30:05 UTC

Permalink

Post by Finucane, Stephen

Post by Finucane, Stephen
Autotools defaults to the 'v7' legacy tar format in GNU tar, through

passing of the '-o' parameter to GNU tar. Enabling this option results in
errors for users with 32 bit UIDs. For example, with the Open vSwitch
'make dist' is under the purview of automake, not autoconf. You may get
a better response by involving the automake list.

Sorry - I find the line between the two to be rather blurred at the best of times :) Should I forward this or CC this list?

Post by Finucane, Stephen

Post by Finucane, Stephen
$ make dist
...
tardir=openvswitch-2.3.90 && ${TAR-tar} chof - "$tardir" | GZIP=--

best gzip -c >openvswitch-2.3.90.tar.gz

Post by Finucane, Stephen
tar: value 12345678 out of uid_t range 0..2097151
tar: Exiting with failure status due to previous errors
make[1]: Leaving directory `/development/ovs'
...
I managed to modify the Autoconf 'configure.ac' file to use the 'tar-

ustar' format, which allow longer file names and other niceties. Again,

Post by Finucane, Stephen
diff --git a/configure.ac b/configure.ac
index ebb8b02..6505189 100644
--- a/configure.ac
+++ b/configure.ac
@@ -19,7 +19,7 @@ AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_HEADERS([config.h])
AC_CONFIG_TESTDIR([tests])
-AM_INIT_AUTOMAKE
+AM_INIT_AUTOMAKE([tar-ustar])

Are you proposing that we change the way autoconf is distributed? That
won't affect any other packages (you'd have to make the same patch for
each affected package), and so far, your code shows that you had
problems in building an openvswitch tarball, not an autoconf tarball.
Again, changing automake to do this automatically for ALL packages (once
those packages are built with a new enough automake) rather than trying
to patch one configure.ac for every affected package, seems like it
would be the better course of action.
I'm still open to be convinced that autoconf needs to alter its own
configure.ac, but I don't have enough evidence yet that it would make a
difference.

Not necessarily. The use of the 'v7' format means any users with a 32 bit UID will need to modify every 'configure.ac' file they come across to fix the above issue. However, it's not like I personally work with enough packages to make this much of an issue. Instead, an approach that would allow me to enable the 'tar-ustar' format dynamically (like the 'configure' option above) would be just fine, if such a thing exists. This would allow me (and all 32bit UID users) to adopt a standard approach to working with Autotools based packages (i.e. always pass an option to the 'configure' script of any Autotools-based package).

Changing the default tar format used by Autotools would result in a permanent fix for all affected users, as you say (affected users = the other developers in my team in Intel, for example). However, I don't know if this would cause compatibility issues on certain (very, very old) machines. You could see a scenario whereby the 'make dist' target of (previously working) autogenerated-Makefiles files would break when Autotools is updated, due to the ustar format not being supported. I have no data on tar ustar support, I'm afraid.

I will attempt to reproduce this issue on other Autotools-based packages and report my findings.

Post by Finucane, Stephen
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Eric Blake

2015-01-05 17:41:48 UTC

Permalink

Post by Finucane, Stephen

Post by Finucane, Stephen
Autotools defaults to the 'v7' legacy tar format in GNU tar, through

Sorry - I find the line between the two to be rather blurred at the best of times :) Should I forward this or CC this list?

I've added the automake list in cc (I'm interested enough in the outcome
that I don't mind if the autoconf list remains in the distribution).

Post by Finucane, Stephen

Post by Finucane, Stephen
$ make dist
...
tardir=openvswitch-2.3.90 && ${TAR-tar} chof - "$tardir" | GZIP=--

best gzip -c >openvswitch-2.3.90.tar.gz

ustar' format, which allow longer file names and other niceties. Again,

Are you proposing that we change the way autoconf is distributed? That
won't affect any other packages (you'd have to make the same patch for
each affected package), and so far, your code shows that you had
problems in building an openvswitch tarball, not an autoconf tarball.
Again, changing automake to do this automatically for ALL packages (once
those packages are built with a new enough automake) rather than trying
to patch one configure.ac for every affected package, seems like it
would be the better course of action.
I'm still open to be convinced that autoconf needs to alter its own
configure.ac, but I don't have enough evidence yet that it would make a
difference.

Not necessarily. The use of the 'v7' format means any users with a 32 bit UID will need to modify every 'configure.ac' file they come across to fix the above issue. However, it's not like I personally work with enough packages to make this much of an issue. Instead, an approach that would allow me to enable the 'tar-ustar' format dynamically (like the 'configure' option above) would be just fine, if such a thing exists. This would allow me (and all 32bit UID users) to adopt a standard approach to working with Autotools based packages (i.e. always pass an option to the 'configure' script of any Autotools-based package).
Changing the default tar format used by Autotools would result in a permanent fix for all affected users, as you say (affected users = the other developers in my team in Intel, for example). However, I don't know if this would cause compatibility issues on certain (very, very old) machines. You could see a scenario whereby the 'make dist' target of (previously working) autogenerated-Makefiles files would break when Autotools is updated, due to the ustar format not being supported. I have no data on tar ustar support, I'm afraid.

As for whether the upstream automake should make it easier for runtime
overrides of the format, or even dynamic changes, I am not sure. I also
know that some packages (particularly those using gnulib's GNUmakefile)
explicitly set:

export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner

to try and avoid the problem of a 32-bit uid interfering with the
creation of old-style tar files. Maybe that is an approach that should
be folded into automake?

Post by Finucane, Stephen
I will attempt to reproduce this issue on other Autotools-based packages and report my findings.

Not sure that your additional testing will change anything - again, the
issue here is that automake would be the one place to change for
everyone else to automatically benefit from, once packages are rebuilt
with newer automake.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Finucane, Stephen

2015-01-06 09:31:39 UTC

Permalink

Post by Finucane, Stephen

Post by Finucane, Stephen
Autotools defaults to the 'v7' legacy tar format in GNU tar, through

passing of the '-o' parameter to GNU tar. Enabling this option results

Post by Finucane, Stephen

Post by Finucane, Stephen
errors for users with 32 bit UIDs. For example, with the Open vSwitch
'make dist' is under the purview of automake, not autoconf. You may get
a better response by involving the automake list.

Sorry - I find the line between the two to be rather blurred at the best

of times :) Should I forward this or CC this list?
I've added the automake list in cc (I'm interested enough in the outcome
that I don't mind if the autoconf list remains in the distribution).

Post by Finucane, Stephen

Post by Finucane, Stephen
$ make dist
...
tardir=openvswitch-2.3.90 && ${TAR-tar} chof - "$tardir" | GZIP=--

best gzip -c >openvswitch-2.3.90.tar.gz

ustar' format, which allow longer file names and other niceties. Again,

Are you proposing that we change the way autoconf is distributed? That
won't affect any other packages (you'd have to make the same patch for
each affected package), and so far, your code shows that you had
problems in building an openvswitch tarball, not an autoconf tarball.
Again, changing automake to do this automatically for ALL packages (once
those packages are built with a new enough automake) rather than trying
to patch one configure.ac for every affected package, seems like it
would be the better course of action.
I'm still open to be convinced that autoconf needs to alter its own
configure.ac, but I don't have enough evidence yet that it would make a
difference.

Not necessarily. The use of the 'v7' format means any users with a 32 bit

UID will need to modify every 'configure.ac' file they come across to fix
the above issue. However, it's not like I personally work with enough
packages to make this much of an issue. Instead, an approach that would
allow me to enable the 'tar-ustar' format dynamically (like the 'configure'
option above) would be just fine, if such a thing exists. This would allow
me (and all 32bit UID users) to adopt a standard approach to working with
Autotools based packages (i.e. always pass an option to the 'configure'
script of any Autotools-based package).

Post by Finucane, Stephen
Changing the default tar format used by Autotools would result in a

permanent fix for all affected users, as you say (affected users = the
other developers in my team in Intel, for example). However, I don't know
if this would cause compatibility issues on certain (very, very old)
machines. You could see a scenario whereby the 'make dist' target of
(previously working) autogenerated-Makefiles files would break when
Autotools is updated, due to the ustar format not being supported. I have
no data on tar ustar support, I'm afraid.
As for whether the upstream automake should make it easier for runtime
overrides of the format, or even dynamic changes, I am not sure. I also
know that some packages (particularly those using gnulib's GNUmakefile)
export TAR_OPTIONS = --owner=0 --group=0 --numeric-owner
to try and avoid the problem of a 32-bit uid interfering with the
creation of old-style tar files. Maybe that is an approach that should
be folded into automake?

Ah, so there isn't currently a dynamic way to do this that you're aware of? This was my original question, such that this patch could be merged to OVS:

http://openvswitch.org/pipermail/dev/2014-December/049819.html

It is interesting to see an alternative approach to resolving the issue. That approach would prevent the need to change the default format to ustar, though there are other benefits to using the ustar format:

https://github.com/libarchive/libarchive/wiki/ManPageTar5

Post by Finucane, Stephen

Post by Finucane, Stephen
I will attempt to reproduce this issue on other Autotools-based packages

and report my findings.
Not sure that your additional testing will change anything - again, the
issue here is that automake would be the one place to change for
everyone else to automatically benefit from, once packages are rebuilt
with newer automake.

Understood. Guess we should wait for Automake folks to get back to us so.

Post by Finucane, Stephen
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org