autocon and sub-packages

Post by SÃ©bastien Hinderer
The tool is distributed with the libraries it depends on
(they are provided as bundles).

This approach is generally fine in principle.

Post by SÃ©bastien Hinderer
For each dependency, coccinelle's configure script checks whether the
library is already installed. If not, the system is prepared to use the
bundled version.

The configuration interface needs also to be designed in a way so that
a software builder can select the preferred variant.

Post by SÃ©bastien Hinderer
This approach does not seem very satisfactory to me.

Interesting …

Post by SÃ©bastien Hinderer
For a start, I find it cumbersome to provide the dependencies as bundles

Is it usual that a working version of a needed software component
is distributed together with your evolving tool?

Post by SÃ©bastien Hinderer
but I'm not the person making the decisions.

Would any more maintainers like to share their experiences?

Post by SÃ©bastien Hinderer
However, any opinion or pointer about software bundling their dependencies,
pros and cons and the techniques used to do so would be warmly appreciated.

I assume that you are more looking for general solutions which can make
the involved software management a bit easier and safer.

You can call another configuration script already if a bundle provides one.

Post by SÃ©bastien Hinderer
I am aware of the AC_CONFIG_SUBDIRS macro, but this seems a bit limited to me.
For instance it means that the sub-package's configure may find a different
compiler to use than the one found by the main conigure, which is not good.

Would you like to revive discussions around an issue which is described in
a similar way in the article "Autoconf: AC_CONFIG_SUBDIRS With Custom Flags
for Subprojects" by Clemens Lang?
https://neverpanic.de/blog/2014/04/10/autoconf-ac-config-subdirs-with-custom-flags-for-subprojects/

Post by SÃ©bastien Hinderer
One other issue is that we bundle the dependencies as .tar.gz archives
and we would like to be able to extract the archives only for those
dependencies that will really be needed (because they are not already
installed on the system).

Did we stumble on similar software management challenges often enough already?

Do any "problems" come from design details like the following?
* How much control have you got on the bundle format for each item?

* Do the involved data formats support the determination of dependencies
without unpacking the available archives completely?

Post by SÃ©bastien Hinderer
Can this be achieved somehow with autoconf?

Partly, yes.

The autoconf software is reusing the programming language "M4".
https://www.gnu.org/savannah-checkouts/gnu/autoconf/manual/autoconf-2.69/html_node/Autoconf-Language.html

It can be extended with libraries as usual.
https://www.gnu.org/software/autoconf-archive/

Regards,
Markus

Eric Blake

2015-09-04 12:07:35 UTC

Post by SÃ©bastien Hinderer
Dear all,
I am one of the maintainers of Coccinelle[1], a tool written in the
Objective Caml[2] language.
The tool is distributed with the libraries it depends on (they are
provided as bundles).

At one point, this was how both libintl (gettext) and libltdl (libtool)
were primarily used - many packages would include subdirectories with
the appropriate bundled library, and then rely on macros provided by
gettext/libtool to probe whether to favor the system version if present
and capable, otherwise fall back to the bundled version. However, these
days, most people have leaned towards excluding the bundle, and instead
requiring the external library to be present. For more information,
check out:

https://www.gnu.org/software/gettext/manual/gettext.html#AM_005fGNU_005fGETTEXT
https://www.gnu.org/software/libtool/manual/libtool.html#Distributing-libltdl

Post by SÃ©bastien Hinderer
Then, assuming we continue to bundle the dependencies, it seems to me
that it would be more coherent to have the configure script of each
required bundle run by the tool's main configure script. I am aware of
the AC_CONFIG_SUBDIRS macro, but this seems a bit limited to me. For
instance it means that the sub-package's configure may find a different
compiler to use than the one found by the main conigure, which is not
good.

I'm not sure why you think a different compiler would be picked for a
sub-package. Particularly if $CC is set in the parent package before
calling out to the sub-packages, then the sub-packages should see $CC in
the environment and favor that as their compiler. But I don't have much
experience with bundled libraries (as I have moved strongly to the camp
of require the external library to be present, now that both gettext and
libtool have made that easier to be the default), to know for sure.

If you can come up with appropriate shell code to do that, then you can
embed that shell code in your configure.ac. I don't know of any autoconf
macros that would automate some of the work, but it sounds like it might
be something that could be cobbled together, if you still want to go the
route of shipping dependent library bundles.

--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org

Sébastien Hinderer

2015-09-04 12:26:20 UTC

Dear Eric,

Thanks a lot for your response.

At one point, this was how both libintl (gettext) and libltdl (libtool)
were primarily used - many packages would include subdirectories with
the appropriate bundled library, and then rely on macros provided by
gettext/libtool to probe whether to favor the system version if present
and capable, otherwise fall back to the bundled version. However, these
days, most people have leaned towards excluding the bundle, and instead
requiring the external library to be present. For more information,
https://www.gnu.org/software/gettext/manual/gettext.html#AM_005fGNU_005fGETTEXT
https://www.gnu.org/software/libtool/manual/libtool.html#Distributing-libltdl

Thank. I think the bundle approach is favoured because the Objective
Camllanguage and its libraries are not as widespread as gettext and
libtool. So the idea of the bundles is tomake life of end-users simpler,
but of course it also makes thelifeofdevelopers and maintainers a bit
harder.

I'm not sure why you think a different compiler would be picked for a
sub-package. [...]

Because that already happened. ;-)

The thing is that the macros to detect the OCaml compiler and the
associated tools are not yet included in autoconf. So we provide them in
the tool I am responsible for, and it turns out that one o the bundled
library provides them, too, but in a different version. And the
twoversions of the macro found different compilers. So things are not as
standard and straightforward for OCaml as they are for C-like languages.

Thanks. I indeed think it should be possible to achieve this.
As I said I would personally prefer not to bundle libraries but I'm not
in a position where I can take this decision.

Cheers,

Sébastien.

Earnie

2015-09-04 13:45:11 UTC

Post by SÃ©bastien Hinderer
Dear Eric,
Thanks a lot for your response.

At one point, this was how both libintl (gettext) and libltdl (libtool)
were primarily used - many packages would include subdirectories with
the appropriate bundled library, and then rely on macros provided by
gettext/libtool to probe whether to favor the system version if present
and capable, otherwise fall back to the bundled version. However, these
days, most people have leaned towards excluding the bundle, and instead
requiring the external library to be present. For more information,
https://www.gnu.org/software/gettext/manual/gettext.html#AM_005fGNU_005fGETTEXT
https://www.gnu.org/software/libtool/manual/libtool.html#Distributing-libltdl

I'd be inclined to provide binary distributions of those libraries for
the systems I wish to support and not embed bundles in my source. If
others feel it important for other systems they can provide the binary
distributions for those systems. There is also the GCC method of if the
library directory exists before configure is executed it schedules the
build of the library. In this way the build of the library is provided
for by the build system but the source package doesn't contain the
library package itself.

I'm not sure why you think a different compiler would be picked for a
sub-package. [...]

Because that already happened. ;-)
The thing is that the macros to detect the OCaml compiler and the
associated tools are not yet included in autoconf. So we provide them in
the tool I am responsible for, and it turns out that one o the bundled
library provides them, too, but in a different version. And the
twoversions of the macro found different compilers. So things are not as
standard and straightforward for OCaml as they are for C-like languages.

To use Eric's term, cobble up some m4 to detect it and provide as a
patch to autoconf. As for the "one o the bundled libraries provides
them, too" I suggest you work with that library team and correct that
issue. But since you're bundling the library then you can modify its
macro to match yours before you bundle it.

Thanks. I indeed think it should be possible to achieve this.
As I said I would personally prefer not to bundle libraries but I'm not
in a position where I can take this decision.

You could let the make do the extraction. No need to cobble something
in shell code. Make the extraction via make depend on some missing file.

--
Earnie

SF Markus Elfring

2015-09-04 14:15:32 UTC

I think the bundle approach is favoured because the Objective Camllanguage
and its libraries are not as widespread as gettext and libtool.
So the idea of the bundles is tomake life of end-users simpler,
but of course it also makes thelifeofdevelopers and maintainers
a bit harder.

The corresponding software development challenges can be manageable,
can't they?

I'm not sure why you think a different compiler would be picked for a
sub-package. [...]

This is an interesting package detail.

And the twoversions of the macro found different compilers.

Do the package requirements need another look?

So things are not as standard and straightforward for OCaml as they are
for C-like languages.

I imagine that further version surprises can happen also with more popular
programming languages if deeper component hierarchies will be considered.
How often do you need to take special care for changes around application
binary interfaces?

Post by Eric Blake
If you can come up with appropriate shell code to do that, then you can
embed that shell code in your configure.ac. I don't know of any autoconf
macros that would automate some of the work, but it sounds like it might
be something that could be cobbled together, if you still want to go the
route of shipping dependent library bundles.

Thanks. I indeed think it should be possible to achieve this.

I am curious if a bit more momentum will evolve to improve the affected
build scripts.

As I said I would personally prefer not to bundle libraries but I'm not
in a position where I can take this decision.

Are there any distribution situations where software bundling is
finally unavoidable?

Regards,
Markus

Ralf Corsepius

2015-09-04 15:38:00 UTC

Camllanguage and its libraries are not as widespread as gettext and
libtool. So the idea of the bundles is tomake life of end-users simpler,
but of course it also makes thelifeofdevelopers and maintainers a bit
harder.

Well, what I can tell you with my Fedora on is that in Fedora we
discourage bundling, because it in a nutshell raises a lot problems in
maintenance, both for system-integrators (read: distros) and upstreams.

Ralf

Sébastien Hinderer

2015-09-04 16:58:17 UTC

Dear Ralf,

Well, what I can tell you with my Fedora on is that in Fedora we discourage
bundling, because it in a nutshell raises a lot problems in maintenance,
both for system-integrators (read: distros) and upstreams.

Many thanks for having written this. It is also what I think but I'd
need so me support so people who share this point of view are very
welcome to let me know, even personally to avoid sending too many
me-mailsto the list.

Sébastien.

Wookey

2015-09-04 18:46:36 UTC

Post by Ralf Corsepius

Well, what I can tell you with my Fedora on is that in Fedora we
discourage bundling, because it in a nutshell raises a lot problems
in maintenance, both for system-integrators (read: distros) and
upstreams.

Right. Debian requires software not to use embedded libraries (unless
they are genuinely exclusive to the package) if at all possible.
Bundling libraries is bad for security (the bundled versions don't get
updated when an issue is found in the system version), it wastes
memory (unless versions are identical) and space, and conversely using
system libs encourages proper upstreaming (over random forking and
hackery), and API stabilty/management.

It's acceptable to leave such libs in the source if that's how
upstream ships it, but not to use them in the build. They are sometimes
removed from the source too in the packaging process.

Packaging (and build-systems like autotools) provides the same
functionality from the user/developer POV, of having all the
build-dependencies easily available at build-time. Distro-people
believe this is superior to every upstream trying to do it with their
own random-version, maybe-forked embedded copy :-)

Wookey

--
Principal hats: Linaro, Debian, Wookware, ARM
http://wookware.org/

Warren Young

2015-09-04 18:27:51 UTC

Post by Eric Blake
https://www.gnu.org/software/gettext/manual/gettext.html#AM_005fGNU_005fGETTEXT
https://www.gnu.org/software/libtool/manual/libtool.html#Distributing-libltdl

Thank. I think the bundle approach is favoured because the Objective
Camllanguage and its libraries are not as widespread as gettext and
libtool.

Left unsaid in Eric’s answer is that this change in distribution philosophy happened *because* capable versions started appearing everywhere, so it was no longer necessary to provide copies along with the main package.

If you’re using libraries that are also not yet ubiquitous, the alternative to providing the sub-packages with the main package is to add a hard-fail autoconf test for them.

Post by SÃ©bastien Hinderer
So the idea of the bundles is tomake life of end-users simpler,
but of course it also makes thelifeofdevelopers and maintainers a bit
harder.

Not necessarily.

Just yesterday I was building a package that had some configure-time code in it to select one of several different Tcl interpreter embedding libraries. If it found Tcl 8.6, it would simply link directly to the Tcl development libraries, but otherwise it would use an embedded fork of the Tcl 8.6 libraries with fall-backs that allowed it to link against Tcl 8.5 or 8.4.

Without that workaround embedded into the package, my only option would have been to upgrade to Tcl 8.6, or not use the feature that required Tcl 8.6.

Software is infinitely malleable, so it allows a grayscale continuum of solutions. There isn’t “good” or “bad,” only “appropriate” and “inappropriate,” or “effective” and “ineffective.”

Post by Eric Blake
I'm not sure why you think a different compiler would be picked for a
sub-package. [...]

Because that already happened. ;-)

Then it can also happen when these dependencies are external.

Post by SÃ©bastien Hinderer
I would personally prefer not to bundle libraries

I think your main mistake is bundling them as embedded tarballs instead of just shipping them as subdirectories of the lone distribution tarball. It adds complexity, and saves no space in the distribution tarball.

It does save a bit of space at build time on the distribution-user’s machine, but we’re long past the days of 5 MB disk drives the size of washing machines.

Sébastien Hinderer

2015-09-04 18:46:47 UTC

Hello Warren,

Many thanks for providing all these useul comments.

Post by Warren Young
Left unsaid in Eric’s answer is that this change in distribution
philosophy happened *because* capable versions started appearing
everywhere, so it was no longer necessary to provide copies along with
the main package.

Sure.

Post by Warren Young
If you’re using libraries that are also not yet ubiquitous, the
alternative to providing the sub-packages with the main package is to
add a hard-fail autoconf test for them.

You mean that the configure script should check for them and just report
an error if they are not found?

I think the very reason why the former developers of this project hav
chosen to use bundles is to avoid users having to install libraries for
a language they don't know and don't know the tools to install them. So
a script reporting an error and failing would probably not be
satisfactory for those persons -- it would be for me, though.

Post by SÃ©bastien Hinderer
So the idea of the bundles is tomake life of end-users simpler,
but of course it also makes thelifeofdevelopers and maintainers a bit
harder.

Not necessarily.
Just yesterday I was building a package that had some configure-time code in it to select one of several different Tcl interpreter embedding libraries. If it found Tcl 8.6, it would simply link directly to the Tcl development libraries, but otherwise it would use an embedded fork of the Tcl 8.6 libraries with fall-backs that allowed it to link against Tcl 8.5 or 8.4.
Without that workaround embedded into the package, my only option would have been to upgrade to Tcl 8.6, or not use the feature that required Tcl 8.6.
Software is infinitely malleable, so it allows a grayscale continuum
of solutions. There isn’t “good” or “bad,” only “appropriate” and
“inappropriate,” or “effective” and “ineffective.”

I understand your point and find it very true. It's just that so far my
experience of bundles is different and not as positive than the one you
describe. So far, I find that having bundles makes the build system more
complex and difficult to maintain.

Post by Eric Blake
I'm not sure why you think a different compiler would be picked for a
sub-package. [...]

Because that already happened. ;-)

Then it can also happen when these dependencies are external.

Perhaps, indeed.

So, is there a way to have all these macros that detect the OCaml-related
tools integrated to the autoconf distribution?

Would the gain in complexity really be that interesting?

Apart from the tar -xzf command that one could remove, where do you
think one could spare complexity?

Post by Warren Young
It does save a bit of space at build time on the distribution-user’s
machine, but we’re long past the days of 5 MB disk drives the size of
washing machines.

I fully agree with this.

Thanks again!

Sébastien.

Post by Warren Young
_______________________________________________
Autoconf mailing list
https://lists.gnu.org/mailman/listinfo/autoconf

SF Markus Elfring

2015-09-04 19:05:50 UTC

Post by SÃ©bastien Hinderer
So, is there a way to have all these macros that detect the OCaml-related
tools integrated to the autoconf distribution?

Yes.

Are there any more software developers who would like to contribute
their macros to the GNU Autoconf Archive so that more programming languages
can be directly supported by a safer interface?

Regards,
Markus

Warren Young

2015-09-04 19:45:49 UTC

Post by Warren Young
If you’re using libraries that are also not yet ubiquitous, the
alternative to providing the sub-packages with the main package is to
add a hard-fail autoconf test for them.

You mean that the configure script should check for them and just report
an error if they are not found?

Yes. It’s a very common thing for autoconfiscated packages to do, because it’s superior to failing at compile time.

Post by SÃ©bastien Hinderer
...to avoid users having to install libraries for
a language they don't know and don't know the tools to install them.

An inability to cope with compilers and libraries is not a good enough reason to bundle dependencies. Your users have already unpacked your package and are trying to build it, after all. If your users are so clueless that they can’t install your prereqs, they probably can’t install your package, either.

A better reason is simply to save people time chasing dependencies.

Does OCaml have a widely-used package manager, a la gem, npm, cpan, nuget…? If so, it may be sufficient simply to give the command that installs all the prereqs in your package’s build instructions. If not, then bundling them might indeed be the best plan.

Post by SÃ©bastien Hinderer
so far my
experience of bundles is different and not as positive than the one you
describe.

So, “ineffective,” then, not “bad”. :)

Remediation in place may be better than replacement.

Post by SÃ©bastien Hinderer
So, is there a way to have all these macros that detect the OCaml-related
tools integrated to the autoconf distribution?

I’m certain that patches would be thoughtfully considered. :)

If your patch is rejected, there are a couple of autoconf macro hosting sites. Merely having a canonical version of a macro may lead, over time, to less divergence in distributed macro versions.

Would the gain in complexity really be that interesting?
Apart from the tar -xzf command that one could remove, where do you
think one could spare complexity?

I’m referring to the fact that with embedded tarballs, you can’t call AC_CONFIG_SUBDIRS() to make the top-level configure script call the subpackage configure scripts. Because unpacking the tarballs is conditional, your main software package must be reinventing this wheel, and therefore probably missing out on some of the 20 years of wisdom embedded within autoconf, such as how to pass along things like --prefix, how to handle separate build and source trees, etc.

Warren Young

2015-09-04 19:52:44 UTC

Post by Warren Young
I’m referring to the fact that with embedded tarballs, you can’t call AC_CONFIG_SUBDIRS()

By the way, the how-to of conditional AC_CONFIG_SUBDIRS() is a FAQ. One set of answers is here:

http://stackoverflow.com/questions/15230696/

SF Markus Elfring

2015-09-05 10:58:42 UTC