How to deal with test data in ouf of source tree builds?

[Paul Menzel]

Dear Autoconf folks,


When doing an out of source build of Unbound, the test suite fails with
the error below.

```
test signature verify functions
testpkts error: could not open file testdata/test_signatures.1: No such
file or directory
Makefile:304: recipe for target 'test' failed
make: *** [test] Error 1
```

In the source directory there is a directory `testdata`, and that gets
referenced incorrectly in `testcode/unitverify.c`.

```
void
verify_test(void)
{
unit_show_feature("signature verify");
#ifdef USE_SHA1
verifytest_file("testdata/test_signatures.1", "20070818005004");
#endif
#if defined(USE_DSA) && defined(USE_SHA1)
verifytest_file("testdata/test_signatures.2", "20080414005004");
verifytest_file("testdata/test_signatures.3", "20080416005004");
[…]
```

Unfortunately, I couldn’t find any documentation how to deal with test
data when building out of the source tree. Should the test data be
copied to the build directory, despite it being not changed, should it
be linked, or does the code need to reference only absolute paths, and
therefore has to be preprocessed(?)?


Kind regards,

Paul

[Nick Bowler]

Hello Paul,

Post by Paul Menzel
When doing an out of source build of Unbound, the test suite fails

[...]

Post by Paul Menzel
In the source directory there is a directory `testdata`, and that gets
referenced incorrectly in `testcode/unitverify.c`.

[...]

Post by Paul Menzel
verifytest_file("testdata/test_signatures.1", "20070818005004");

[...]

Post by Paul Menzel
Unfortunately, I couldn’t find any documentation how to deal with test
data when building out of the source tree. Should the test data be
copied to the build directory, despite it being not changed, should it
be linked, or does the code need to reference only absolute paths, and
therefore has to be preprocessed(?)?

There are many possible solutions. The simplest is probably to cd into
the source directory before running the test programs. But this may have
other undesired effects.

I suggest eliminating the dependency on the working directory completely.

For example, you can pass the location of necessary data files to the
test programs via environment variable(s) or command-line arguments. If
all your data files are in one place this might be something simple like:

UNBOUND_TEST_DATA=${srcdir}/testdata ./the_test_program

and the test program can construct filenames based on that environment
variable.

Cheers,
Nick

[Bob Friesenhahn]

Post by Nick Bowler
I suggest eliminating the dependency on the working directory completely.
For example, you can pass the location of necessary data files to the
test programs via environment variable(s) or command-line arguments. If
UNBOUND_TEST_DATA=${srcdir}/testdata ./the_test_program
and the test program can construct filenames based on that environment
variable.

Environment variables are convenient. Command line arguments to the
test program are also convenient. If the test program is also the
installed program, take care that security or reliability issues are
not introduced due to responding to environment variables.

Microsoft Windows has very limited command line plus environment space
(come out of same allocation) so take care not to use so much
resources that it is not possible to run the tests under Windows (if
Windows is supported).

Bob

[Peter Johansson]

Hi Paul,

Post by Paul Menzel
Dear Autoconf folks,
When doing an out of source build of Unbound, the test suite fails
with the error below.
```
test signature verify functions
testpkts error: could not open file testdata/test_signatures.1: No
such file or directory
Makefile:304: recipe for target 'test' failed
make: *** [test] Error 1
```
In the source directory there is a directory `testdata`, and that gets
referenced incorrectly in `testcode/unitverify.c`.

One solution I've used is to export ${abs_top_srcdir} to config.h and
then use that value as prefix in filenames.

Another solution is to create a link to directory testdata when $srcdir
!= $builddir. There is a macro AC_CONFIG_LNKS that might be useful.

Cheers,
Peter