document how to create universal binaries

[Bruno Haible]

Hi,

On MacOS X 10.5, most system-provided binaries and libraries are "fat" or
"universal": they can be executed on all architectures of MacOS X 10.5.

Given the section "Compiling For Multiple Architectures" from the INSTALL
file of so many GNU packages, I was under the impression that it was hard
to build universal binaries for packages that use autoconf. But this is
actually not the case. Among the 4 packages I tested (libiconv, libsigsegv,
gettext, gperf), all but one can be built as universal binaries out of the box,
given the appropriate instructions, and the one that did not work (libsigsegv)
could be made to work in 20 minutes.

This is clearly not optimally documented. The location where it should be
documented, IMO, is the INSTALL file. Here is a patch which adds a third
paragraph to the section "Compiling For Multiple Architectures":

On MacOS X systems, you can create libraries and executables that work
on multiple system types - known as "fat" or "universal" binaries - by
specifying multiple '-arch' options to the compiler but only a single
'-arch' option to the preprocessor. Like this:

env CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CPP="gcc -arch ppc -E" \
CXXCPP="g++ -arch ppc -E" \
./configure

Here's the proposed patch.


2008-08-17 Bruno Haible <***@clisp.org>

* doc/install.texi (Compiling For Multiple Architectures): Explain how
to create universal binaries on MacOS X.

--- doc/install.texi.bak 2008-03-14 01:46:05.000000000 +0100
+++ doc/install.texi 2008-08-17 11:33:19.000000000 +0200
@@ -6,7 +6,7 @@
@unnumbered Installation Instructions

Copyright @copyright{} 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004,
-2005, 2006, 2007 Free Software Foundation, Inc.
+2005, 2006, 2007, 2008 Free Software Foundation, Inc.

This file is free documentation; the Free Software Foundation gives
unlimited permission to copy, distribute and modify it.
@@ -120,6 +120,19 @@
installed the package for one architecture, use @samp{make distclean}
before reconfiguring for another architecture.

+On MacOS X systems, you can create libraries and executables that work
+on multiple system types -- known as "fat" or "universal" binaries --
+by specifying multiple @option{-arch} options to the compiler but only
+a single @option{-arch} option to the preprocessor. Like this:
+
+@example
+env CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CPP="gcc -arch ppc -E" \
+ CXXCPP="g++ -arch ppc -E" \
+ ./configure
+@end example
+
@node Installation Names
@section Installation Names

[Erik de Castro Lopo]

Post by Bruno Haible
On MacOS X 10.5, most system-provided binaries and libraries are "fat" or
"universal": they can be executed on all architectures of MacOS X 10.5.

This was Apple's marketing solution to a technical problem.

Post by Bruno Haible
Given the section "Compiling For Multiple Architectures" from the INSTALL
file of so many GNU packages, I was under the impression that it was hard
to build universal binaries for packages that use autoconf. But this is
actually not the case. Among the 4 packages I tested (libiconv, libsigsegv,
gettext, gperf), all but one can be built as universal binaries out of the box,
given the appropriate instructions, and the one that did not work (libsigsegv)
could be made to work in 20 minutes.

The documentation should also state that some packages can't be
compiled as universal binaries without extensive re-working.

Erik

[Andreas Schwab]

Post by Bruno Haible
env CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
CPP="gcc -arch ppc -E" \
CXXCPP="g++ -arch ppc -E" \
./configure

It is preferred to pass variable settings as arguments instead of in the
environment, for better support of rerunning configure, see
(autoconf)Defining Variables.

Andreas.

[Bruno Haible]

It is preferred to pass variable settings as arguments ...

You're right, of course. Here is a revised patch.

2008-08-17 Bruno Haible <***@clisp.org>

* doc/install.texi (Compiling For Multiple Architectures): Explain how
to create universal binaries on MacOS X.

--- doc/install.texi.bak 2008-03-14 01:46:05.000000000 +0100
+++ doc/install.texi 2008-08-17 15:02:18.000000000 +0200
@@ -6,7 +6,7 @@
@unnumbered Installation Instructions

Copyright @copyright{} 1994, 1995, 1996, 1999, 2000, 2001, 2002, 2004,
-2005, 2006, 2007 Free Software Foundation, Inc.
+2005, 2006, 2007, 2008 Free Software Foundation, Inc.

This file is free documentation; the Free Software Foundation gives
unlimited permission to copy, distribute and modify it.
@@ -120,6 +120,18 @@
installed the package for one architecture, use @samp{make distclean}
before reconfiguring for another architecture.

+On MacOS X systems, you can create libraries and executables that work
+on multiple system types -- known as "fat" or "universal" binaries --
+by specifying multiple @option{-arch} options to the compiler but only
+a single @option{-arch} option to the preprocessor. Like this:
+
+@example
+./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CPP="gcc -arch ppc -E" \
+ CXXCPP="g++ -arch ppc -E"
+@end example
+
@node Installation Names
@section Installation Names

[Peter O'Gorman]

Post by Bruno Haible
+On MacOS X systems, you can create libraries and executables that work

Only Mac OS X 10.5, on 10.4.x only x86 had fat libraries in /, it may be
best to specify that. For 10.4.x ppc you had to install an SDK and add
-isysroot flags.

Post by Bruno Haible
+on multiple system types -- known as "fat" or "universal" binaries --
+
+./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
+ CPP="gcc -arch ppc -E" \
+ CXXCPP="g++ -arch ppc -E"

For automake using projects --disable-dependency-tracking is also
necessary. Also, I am not sure if putting the -arch flags in CC/CXX and
having to specify the preprocessor, or putting them in CFLAGS/CXXFLAGS
and not specifying the preprocessor is "better".

Post by Bruno Haible
+
@node Installation Names
@section Installation Names

Because ppc and x86 are differently endian, and because 64bit and 32bit
have different sizeof(long) etc, it may be best to mention that just
because it builds does not mean that it will work. Also, I believe that
there are cases autoconf will discover different symbols/libraries when
building on i386 than when building on ppc, that can result in link
errors for one architecture.

There are a number of "gotchas" when building fat binaries, I'm not sure
it is best to put this in the INSTALL document without any warnings or
caveats.

Apple recommends building once per architecture (on a machine that can
do so natively) and using lipo to create a universal binary. That way
most of the problems are avoided.

Peter

[Bruno Haible]

Post by Bruno Haible
+On MacOS X systems, you can create libraries and executables that work

For 10.4.x ppc you had to install an SDK and add -isysroot flags.

ok, please change to doc patch to say "On MacOS X systems, version 10.5
or newer".

For automake using projects --disable-dependency-tracking is also
necessary.

What happens if it is not specified? I just compiled texinfo-4.12 without
this flag. All compilation commands included
DEPDIR=.deps depmode=gcc /bin/sh ../build-aux/depcomp ...
and there were no errors.

Also, I am not sure if putting the -arch flags in CC/CXX and
having to specify the preprocessor, or putting them in CFLAGS/CXXFLAGS
and not specifying the preprocessor is "better".

The latter does not work, because then the preprocessor cannot be run:

$ gcc -arch i386 -arch ppc -E foo.c
gcc-4.0: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags

Because ppc and x86 are differently endian, and because 64bit and 32bit
have different sizeof(long) etc, it may be best to mention that just
because it builds does not mean that it will work.

That's an issue that should be mentioned in autoconf.texi, not in install.texi,
because it's the developer, not the installer, who will have to deal with it.

Also, I believe that
there are cases autoconf will discover different symbols/libraries when
building on i386 than when building on ppc, that can result in link
errors for one architecture.

Likewise.

One could also mention that a method to detect these configuration problems
is to build in two different directories, in one with
CPP="gcc -arch i386 -E"
and in the other with
CPP="gcc -arch ppc -E"
and then look for differences between the config.h and config.status files.

Apple recommends building once per architecture (on a machine that can
do so natively) and using lipo to create a universal binary. That way
most of the problems are avoided.

But this way is a lot of hand work. It's certainly necessary for programs
like 'emacs' (due to the unexec), but can be avoided in many cases.

Bruno

[Peter O'Gorman]

Post by Bruno Haible

Post by Bruno Haible
+On MacOS X systems, you can create libraries and executables that work

For 10.4.x ppc you had to install an SDK and add -isysroot flags.

ok, please change to doc patch to say "On MacOS X systems, version 10.5
or newer".

For automake using projects --disable-dependency-tracking is also
necessary.

What happens if it is not specified? I just compiled texinfo-4.12 without
this flag. All compilation commands included
DEPDIR=.deps depmode=gcc /bin/sh ../build-aux/depcomp ...
and there were no errors.

Also, I am not sure if putting the -arch flags in CC/CXX and
having to specify the preprocessor, or putting them in CFLAGS/CXXFLAGS
and not specifying the preprocessor is "better".

$ gcc -arch i386 -arch ppc -E foo.c
gcc-4.0: -E, -S, -save-temps and -M options are not allowed with multiple -arch flags

./configure CC=gcc CFLAGS='-arch ppc -arch i386 -arch x86_64 -arch ppc64
-O2 -g' CXX='g++' CXXFLAGS='-arch ppc -arch i386 -arch x86_64 -arch
ppc64 -O2 -g'

does not require the setting of CPP, but does require that
--disable-dependency-tracking be added. Oh well. :)

Okay, I am inclined to agree with you, that the -arch flags belong in
the CC/CXX variables.

I would, however, remove the -arch ppc from the CPP and CXXCPP, i.e.
just CPP='gcc -E' CXXCPP='g++ -E' to get the native preprocessor,
whatever that may be.

In the majority of cases there are no problems building universal
binaries in this manner. But there are enough cases where it does not
build, or where it builds but has problems for some architectures at
runtime, that I still think if the instruction is added to install.texi,
a warning be added in the same spot.

Would you agree to something like this?

Peter

[Eric Blake]

Post by Peter O'Gorman
In the majority of cases there are no problems building universal
binaries in this manner. But there are enough cases where it does not
build, or where it builds but has problems for some architectures at
runtime, that I still think if the instruction is added to install.texi,
a warning be added in the same spot.
Would you agree to something like this?

I went ahead and fixed up some texinfo markup flaws, adjusted the
ChangeLog, then committed Peter's wording. As I don't currently have
access to MacOS, you'll need to double check that it looks correct. I'm
not sure if Karl's cron jobs will automatically update gnulib's copy of
INSTALL, but if that hasn't happened in another day or two, I'll ping him.

I also know that a while ago, Bruno proposed some other changes to INSTALL
wording; I'll dig through my inbox and see if I can revive that patch.

- --
Don't work too hard, make some time for fun as well!

Eric Blake ***@byu.net