help configure project for HPC - non-standard ways to run the executable

Discussion:

Anton Shterenlikht

2017-09-20 15:43:17 UTC

Hi, I'm new to this list.

I develop a Fortran 2008 coarrays project
targeting HPC systems.
I'd like to simplify my build process with autotools.

The immediate problem is that the executable is
run in different ways on different platforms.

For example, OpenCoarrays compiler is "caf", but
the executable is run with "cafrun -np .. ./a.out".
This is similar to "mpif90" and "mpirun" for MPI
programs. On Cray I compile with "ftn" and run with
"aprun" (it's also cross-compiled, but I'm not there yet).

So I get errors like:

configure:2850: caf -o conftest conftest.f >&5
configure:2854: $? = 0
configure:2861: ./conftest
[***@20474701626e.anet.bris.ac.uk] match_arg (utils/args/args.c:159): unrecognized argument pmi_
args
[***@20474701626e.anet.bris.ac.uk] HYDU_parse_array (utils/args/args.c:174): argument matching r
eturned error
[***@20474701626e.anet.bris.ac.uk] parse_args (ui/mpich/utils.c:1596): error parsing input array
[***@20474701626e.anet.bris.ac.uk] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1648): unable to
parse user arguments
[***@20474701626e.anet.bris.ac.uk] main (ui/mpich/mpiexec.c:153): error parsing parameters

because the invocation should have been
"cafrun -np .. ./conftest"
for this compiler.

Is there a standard way of implementing this
in configure.ac?

I looked at ax_prog_fc_mpi:

https://www.gnu.org/software/autoconf-archive/ax_prog_fc_mpi.html

which looks similar but not exactly what I need.

Thanks

Anton

suzuki toshiya

2017-09-21 01:40:57 UTC

Permalink

Dear Anton,

I'm afraid there is no standard mechanism to control the
execution of cross compiled binary. Yet I've not detected
which autoconf macro defines it, but I can find many
attempts trying to execute "./conftest" directly, in
generated configure script - no variable to insert the
intermediate executor.

# my experience with HPC related field is very legacy,
# but I think it was very popular situation that the
# accessible UNIX frontend provides the cross compilers
# and cannot execute the compiled binary, and the system
# executing the compiled binary do not have Unix like
# frontend.

autotools supports the situation that the cross compiler makes
a binary executable but the system cannot execute it directly.
(e.g. ./configure --host=m68k-coff --build=i386-pc-linux-gnu)

if you do not expect any results from the execution of the
compiled binary, the nearest way might be using autotools as
if you work for cross build.

However, in my impression, current autotools are optimized for
GNU-based cross development toolchain which has well-structured
prefix for each tools (cc, as, ar, ld, etc). The proprietary
toolchain for HPC systems might be something what the original
designers of autotool expected.

What kind of the automation you want to realize by autotool?

Regards,
mpsuzuki

Post by Anton Shterenlikht
Hi, I'm new to this list.
I develop a Fortran 2008 coarrays project
targeting HPC systems.
I'd like to simplify my build process with autotools.
The immediate problem is that the executable is
run in different ways on different platforms.
For example, OpenCoarrays compiler is "caf", but
the executable is run with "cafrun -np .. ./a.out".
This is similar to "mpif90" and "mpirun" for MPI
programs. On Cray I compile with "ftn" and run with
"aprun" (it's also cross-compiled, but I'm not there yet).
configure:2850: caf -o conftest conftest.f >&5
configure:2854: $? = 0
configure:2861: ./conftest
args
eturned error
parse user arguments
because the invocation should have been
"cafrun -np .. ./conftest"
for this compiler.
Is there a standard way of implementing this
in configure.ac?
https://www.gnu.org/software/autoconf-archive/ax_prog_fc_mpi.html
which looks similar but not exactly what I need.
Thanks
Anton
_______________________________________________
Autoconf mailing list
https://lists.gnu.org/mailman/listinfo/autoconf

Christian Feld

2017-09-21 06:43:26 UTC

Permalink

Post by suzuki toshiya
Dear Anton,
I'm afraid there is no standard mechanism to control the
execution of cross compiled binary. Yet I've not detected
which autoconf macro defines it, but I can find many
attempts trying to execute "./conftest" directly, in
generated configure script - no variable to insert the
intermediate executor.
# my experience with HPC related field is very legacy,
# but I think it was very popular situation that the
# accessible UNIX frontend provides the cross compilers
# and cannot execute the compiled binary, and the system
# executing the compiled binary do not have Unix like
# frontend.
autotools supports the situation that the cross compiler makes
a binary executable but the system cannot execute it directly.
(e.g. ./configure --host=m68k-coff --build=i386-pc-linux-gnu)
if you do not expect any results from the execution of the
compiled binary, the nearest way might be using autotools as
if you work for cross build.
However, in my impression, current autotools are optimized for
GNU-based cross development toolchain which has well-structured
prefix for each tools (cc, as, ar, ld, etc). The proprietary
toolchain for HPC systems might be something what the original
designers of autotool expected.

I agree that there is and IMO there will be no standard way to control
the execution of cross compiled binaries. In our project, we therefore
explicitly switch to autoconf's cross-compile mode when we are on a
(HPC) system that is known to be a cross-compile system (Blue Gene,
Fujitsu, Cray). I.e., we don't 'run' conftests during configure, we just
compile and link. Also, we don't specify --host and --build for these
systems but provide the native (MPI) compilers to CC, CXX, F77. FC.

This is how to switch to cross-compile mode:

# in configure.ac
cross_compiling=yes

In cross-compile mode the macros
AC_CHECK_FILE
AC_CHECK_FILES
AC_RUN_IFELSE
will die. For the latter one you can define an
'action-if-cross-compiling' though.

Post by suzuki toshiya
What kind of the automation you want to realize by autotool?
Regards,
mpsuzuki

Post by Anton Shterenlikht
Hi, I'm new to this list.
I develop a Fortran 2008 coarrays project
targeting HPC systems.
I'd like to simplify my build process with autotools.

My Fortran experience is limited. But I seem to remember that automatic
dependency generation and Fortran modules don't play well together. This
is an automake issue though.

Christian

Post by suzuki toshiya

Post by Anton Shterenlikht
The immediate problem is that the executable is
run in different ways on different platforms.
For example, OpenCoarrays compiler is "caf", but
the executable is run with "cafrun -np .. ./a.out".
This is similar to "mpif90" and "mpirun" for MPI
programs. On Cray I compile with "ftn" and run with
"aprun" (it's also cross-compiled, but I'm not there yet).
configure:2850: caf -o conftest conftest.f >&5
configure:2854: $? = 0
configure:2861: ./conftest
args
eturned error
parse user arguments
because the invocation should have been
"cafrun -np .. ./conftest"
for this compiler.
Is there a standard way of implementing this
in configure.ac?
https://www.gnu.org/software/autoconf-archive/ax_prog_fc_mpi.html
which looks similar but not exactly what I need.
Thanks
Anton
_______________________________________________
Autoconf mailing list
https://lists.gnu.org/mailman/listinfo/autoconf

_______________________________________________
Autoconf mailing list
https://lists.gnu.org/mailman/listinfo/autoconf

suzuki toshiya

2017-09-21 01:56:17 UTC

Permalink

Oops, sorry for my poor English.

Post by suzuki toshiya
The proprietary
toolchain for HPC systems might be something what the original
designers of autotool expected.

I meant "might be something *different from* what the original
designers of autotool expected".

Regards,
mpsuzuki

_______________________________________________
Autoconf mailing list
https://lists.gnu.org/mailman/listinfo/autoconf

Anton Shterenlikht

2017-09-21 07:51:07 UTC

Permalink

Post by suzuki toshiya
What kind of the automation you want to realize by autotool?

At present I have a collection of Makefiles,
one for each platform (HPC system, compiler, libraries, etc.).
This works ok, but sometimes the users
are not sure which libraries are available
or which they should use, i.e. they are not
sure which Makefile to use.
I was wondering if I can automate this choice
with autotools.

You mention cross-compiling.
However, my examples with mpif90 and mpirun,
or caf and cafrun, are native, not cross-compiling.
The executables are run on the same platform,
it's just the invocation of the
executable is not simply ./a.out,
but "mpirun -np <numproc> ./a.out", or
"cafrun -np <numproc> ./a.out".

Anyway, it sounds that autotools will give me more
headache than help.

Thanks!

Anton

suzuki toshiya

2017-09-21 08:06:35 UTC

Permalink

Dear Anton,

Post by Anton Shterenlikht
At present I have a collection of Makefiles,
one for each platform (HPC system, compiler, libraries, etc.).
This works ok, but sometimes the users
are not sure which libraries are available
or which they should use, i.e. they are not
sure which Makefile to use.

It's very typical cases that people try autoconf :-)

Post by Anton Shterenlikht
I was wondering if I can automate this choice
with autotools.

Good! To check the availabilities of the libraries, autoconf
is still useful. As far as the compiler use standard syntax
(like -I, -L) to specify the directories for header & libraries,
configure script does not need the execution of the compiled
binary. It would be possible for configure script to check
the availabilities of the libraries, and override sometimes
(e.g. ignore the libraries even if it was found).

Post by Anton Shterenlikht
You mention cross-compiling.
However, my examples with mpif90 and mpirun,
or caf and cafrun, are native, not cross-compiling.
The executables are run on the same platform,
it's just the invocation of the
executable is not simply ./a.out,
but "mpirun -np <numproc> ./a.out", or
"cafrun -np <numproc> ./a.out".

Ahh... sorry for my overlooking on the errors you pasted
in your first post. I can see that the program starts but
aborts immediately. Maybe "cafrun" prepares the appropriate
environment and commandline options, even if the original
source is very small bit like "hello world", they are needed.

I think, Christian's idea to insert "cross_compiling=yes"
to your skeleton could make the first issue passed.

Post by Anton Shterenlikht
Anyway, it sounds that autotools will give me more
headache than help.

Maybe I made you misunderstood, I'm quite sorry.

Regards,
mpsuzuki