Future autoconf package compression

[Marko Lindqvist]

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.

I were updating automake version used in OpenEmbedded, and though to
switch from tar.gz to tar.xz package while at it. That failed because
of circular dependency. In short: building xz decompressor requires
automake.


- ML

[Stefano Lattarini]

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.
I were updating automake version used in OpenEmbedded, and though to
switch from tar.gz to tar.xz package while at it. That failed because
of circular dependency. In short: building xz decompressor requires
automake.

If that is true, it must be a bug in the xz packaging process -- once
you have the distributed xz tarball, you shouldn't need the autotools
to configure and build xz from it. I think you should:

- verify the problem you're seeing is real (it seems suspicious nobody
else has spotted it so far);

- if it is, report it to the xz developers.

Regards,
Stefano

[Marko Lindqvist]

On 24 November 2012 10:58, Stefano Lattarini

Post by Stefano Lattarini

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.
I were updating automake version used in OpenEmbedded, and though to
switch from tar.gz to tar.xz package while at it. That failed because
of circular dependency. In short: building xz decompressor requires
automake.

If that is true, it must be a bug in the xz packaging process -- once
you have the distributed xz tarball, you shouldn't need the autotools
- verify the problem you're seeing is real (it seems suspicious nobody
else has spotted it so far);
- if it is, report it to the xz developers.

The problem certainly is in how OpenEmbedded handles the build. Its
metadata has the xz -> autotools dependency.

I've sent email to OpenEmbedded list too about the problem, but I
don't know how hard it would be to get rid of that dependency.
Obviously there's no such dependency problem with tar.gz autotools
packages and gzip.


- ML

[Bob Friesenhahn]

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.

There is no tangible benefit offered to the world by removing the
gzip-compressed autoconf package. Xz is excessively complex,
excessively large, and has limited portability and stability compared
with gzip.

The XZ Utils project obviously has issues. As an example, I clicked
on the NEWS link offered by its web site (http://www.tukaani.org/xz/)
to see what has changed and saw this
"http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD". Further
investigation reveals that its git repository has vaporized.

If the autoconf project desires to offer benefit to the world based on
package sizes, there is much more to be gained by reducing the bloat
that Autoconf configure and support scripts add to the thousands of
packages which depend on it.

Bob

[Jim Meyering]

Post by Bob Friesenhahn

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.

There is no tangible benefit offered to the world by removing the
gzip-compressed autoconf package. Xz is excessively complex,
excessively large, and has limited portability and stability compared
with gzip.

Hi Bob,

I don't know of significant portability problems.
In my experience, if they are reported and affect significant
(sometimes even insignificant) portability targets, they will be
addressed promptly. Can you point to reported problems that
have not been resolved?

There is no shortage of reasons to avoid gzip these days. One that
strikes home for me (as a package maintainer) is that there have
been exploitable CVEs against gzip in the recent past, and the code
is surprisingly ugly (hence hard to audit). I do not want to require
tarball consumers to use a tool that I do not feel good about, and gzip
is one of those. Just because it is still used by so many people (due
mostly to inertia) does not mean that we should ignore its faults.

Post by Bob Friesenhahn
The XZ Utils project obviously has issues. As an example, I clicked
on the NEWS link offered by its web site (http://www.tukaani.org/xz/)
to see what has changed and saw this
"http://git.tukaani.org/?p=xz.git;a=blob;f=NEWS;hb=HEAD". Further
investigation reveals that its git repository has vaporized.

While reading your message (catching up, now), I clicked on
that link, and it works fine now. Even if the site were off-line
for a few days, that is no reason to reject the technology.
If we did that across the board, very few projects would still be with us.

[Bob Friesenhahn]

Post by Jim Meyering

Post by Bob Friesenhahn

Post by Marko Lindqvist
I just encountered new argument for providing .gz of autoconf also in
the future.

There is no tangible benefit offered to the world by removing the
gzip-compressed autoconf package. Xz is excessively complex,
excessively large, and has limited portability and stability compared
with gzip.

Hi Bob,
I don't know of significant portability problems.
In my experience, if they are reported and affect significant
(sometimes even insignificant) portability targets, they will be
addressed promptly. Can you point to reported problems that
have not been resolved?

As far as I am aware, xz still fails to link successfully under x86
(x86-64 kernel) Solaris when built with GCC since xz requests to build
the objects with some special addressing model. This is a well known
issue (see
http://sourceforge.net/projects/lzmautils/forums/forum/708858/topic/3500014).
The documented workaround does not work properly. In order to
circumvent the issue, I had to use the commercial Oracle compiler. I
have not encountered any other software with this sort of issue.

Post by Jim Meyering
There is no shortage of reasons to avoid gzip these days. One that
strikes home for me (as a package maintainer) is that there have
been exploitable CVEs against gzip in the recent past, and the code
is surprisingly ugly (hence hard to audit). I do not want to require

This may indeed be true. However, if autoconf itself stops using gzip
then the effect on the world will be imperceptible except that people
who want to install autoconf and who don't have 'xz' will suffer. If
Automake (and GNU tar) was to refuse to create gzip archives, and new
gzip releases could only decompress, then there would be noticeable
impact.

Bob

[Jim Meyering]

Post by Jim Meyering

Post by Bob Friesenhahn

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.

There is no tangible benefit offered to the world by removing the
gzip-compressed autoconf package. Xz is excessively complex,
excessively large, and has limited portability and stability compared
with gzip.

Hi Bob,
I don't know of significant portability problems.
In my experience, if they are reported and affect significant
(sometimes even insignificant) portability targets, they will be
addressed promptly. Can you point to reported problems that
have not been resolved?
There is no shortage of reasons to avoid gzip these days. One that
strikes home for me (as a package maintainer) is that there have
been exploitable CVEs against gzip in the recent past, and the code
is surprisingly ugly (hence hard to audit). I do not want to require
tarball consumers to use a tool that I do not feel good about, and gzip
is one of those. Just because it is still used by so many people (due
mostly to inertia) does not mean that we should ignore its faults.

FYI, a couple of weeks ago, Aki Helin exposed still more problems in
gzip's unpacking code. Paul Eggert fixed them just a few days ago:

http://git.sv.gnu.org/cgit/gzip.git/commit/?id=f2be148c3d956c2dd19bd6fdbe6d
http://git.sv.gnu.org/cgit/gzip.git/commit/?id=16977ae732bf60f79c9a4fd6d183

[Paul Eggert]

Post by Jim Meyering
FYI, a couple of weeks ago, Aki Helin exposed still more problems in
gzip's unpacking code.

Well, to be fair, I also have a similar problem with 'tar'
in my inbox, from Aki, but I'm not inclined to suggest that
we stop using 'tar'....

[Jim Meyering]

Post by Paul Eggert

Post by Jim Meyering
FYI, a couple of weeks ago, Aki Helin exposed still more problems in
gzip's unpacking code.

Well, to be fair, I also have a similar problem with 'tar'
in my inbox, from Aki, but I'm not inclined to suggest that
we stop using 'tar'....

Of course. Nor am I.
GNU tar's overall code quality is far higher than that of gzip.

[Jeffrey Walton]

Post by Jim Meyering

Post by Paul Eggert

Post by Jim Meyering
FYI, a couple of weeks ago, Aki Helin exposed still more problems in
gzip's unpacking code.

Well, to be fair, I also have a similar problem with 'tar'
in my inbox, from Aki, but I'm not inclined to suggest that
we stop using 'tar'....

Of course. Nor am I.
GNU tar's overall code quality is far higher than that of gzip.

-Wall -Wextra -Wconversion -Wstrict-overflow go a long way in
uncovering basic developer problems, like truncations, conversions,
and 2's compliment behavior. Everyone - from GNU to private
development teams - could benefit from the illumination of issues by
the toolchain's warning system.

I understand the warning system produces false positives at times, but
I don't throw the baby out with the bath water. I work around the
warning system's short comings in an effort to keep code quality up.
Its what we have to do to separate the wheat from the chaff.

Its too bad they are not used more often.

Jeff

[Jim Meyering]

Post by Jeffrey Walton

Post by Jim Meyering

Post by Paul Eggert

Post by Jim Meyering
FYI, a couple of weeks ago, Aki Helin exposed still more problems in
gzip's unpacking code.

Well, to be fair, I also have a similar problem with 'tar'
in my inbox, from Aki, but I'm not inclined to suggest that
we stop using 'tar'....

Of course. Nor am I.
GNU tar's overall code quality is far higher than that of gzip.

-Wall -Wextra -Wconversion -Wstrict-overflow go a long way in
uncovering basic developer problems, like truncations, conversions,
and 2's compliment behavior. Everyone - from GNU to private
development teams - could benefit from the illumination of issues by
the toolchain's warning system.
I understand the warning system produces false positives at times, but
I don't throw the baby out with the bath water. I work around the
warning system's short comings in an effort to keep code quality up.
Its what we have to do to separate the wheat from the chaff.
Its too bad they are not used more often.

If you have noticed real problems in tar or gzip (or any other
GNU project) about which gcc can warn, please let us know.

It sounds like you are trying to teach grandma how to suck eggs :-)

Did you realize that several GNU projects now enable virtually
every gcc warning that is available (even including those that
are new in the upcoming gcc-4.8, for folks that use bleeding edge gcc)
via gnulib's manywarnings.m4 configure-time tests?

Of course, there is a list of warnings that we do disable,
due to their typical lack of utility and the invasiveness
of changes required do suppress them.

[Jim Meyering]

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.
I were updating automake version used in OpenEmbedded, and though to
switch from tar.gz to tar.xz package while at it. That failed because
of circular dependency. In short: building xz decompressor requires
automake.

Hi Marko,

I have simulated building xz with a broken version of automake:
(similar to when automake is not installed)

wget http://www.tukaani.org/xz/xz-5.0.4.tar.xz
tar xf xz-5.0.4.tar.xz
cd xz-5.0.4
mkdir x; printf '#!/bin/sh\nexit 1\n' > x/automake; chmod a+x x/automake
PATH=x:$PATH ./configure && make && make check

All of those commands succeeded, so it is clear to me that building
xz that way does not depend on automake.

Jim

[Marko Lindqvist]

Post by Jim Meyering

Post by Marko Lindqvist

The Autoconf team is considering releasing only .xz files for 2.69; if
this would be a hardship for you, and you need the .gz or .bz2 release,
please speak up now.

I just encountered new argument for providing .gz of autoconf also in
the future.
I were updating automake version used in OpenEmbedded, and though to
switch from tar.gz to tar.xz package while at it. That failed because
of circular dependency. In short: building xz decompressor requires
automake.

Hi Marko,
(similar to when automake is not installed)
wget http://www.tukaani.org/xz/xz-5.0.4.tar.xz
tar xf xz-5.0.4.tar.xz
cd xz-5.0.4
mkdir x; printf '#!/bin/sh\nexit 1\n' > x/automake; chmod a+x x/automake
PATH=x:$PATH ./configure && make && make check
All of those commands succeeded, so it is clear to me that building
xz that way does not depend on automake.

The fact that I were talking about OpenEmbedded build is significant.
OpenEmbedded adds the dependency. AFAIK every tool (autotools, xz,
OpenEmbedded) works correctly as designed, but there's conflict
between their designs.
I have no power to make changes to any of the projects - I can only
suggest things, but overall I think it's OpenEmbedded where changes
should be made (and are easiest to make). But until we have right
thing(tm) fix in place, simply providing also gzip compressed tarballs
of autotools would workaround the problem.


- ML