Background processes in GNU Autotest

[Olaf Mandel]

Hello,

I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
end-to-end tests on a network server. My current test script looks
somewhat like this:

AT_CHECK([server&], , [ignore])
AT_CHECK([client --cmd]), , [expected-output])
AT_CHECK([killall server], , [ignore])

This is far from perfect, however: there is no cleanup if the client
check fails or if the executable is called differently (see below).

Now I want to combine this with valgrind checking (from
AX_VALGRIND_CHECK()): I defined the check-TESTS target in the Makefile
so it calls "make check-autotest" and renames the log files to the
expected names and I added a macro to the test script to use an
environment variable RUNNER:

AC_DEFUN([WRAP_CMD], [${RUNNER} $(which $1) $2])
AT_CHECK(WRAP_CMD([server], [&]), , [ignore])
AT_CHECK(WRAP_CMD([client], [--cmd]), , [expected-output])
AT_CHECK(WRAP_CMD([killall], [server]), , [ignore])

But this fails to kill the server simply because the actual command
being executed by the first AT_CHECK with make check-valgrind is:

valgrind --error-exitcode=1 --quiet ./server &

So the simple killall call is no longer feasible.

Now my question: is there a standard and approved way to start a
background process in a GNU Autotest script and to kill it at or before
the AT_CLEANUP ?

Thank you for any suggestions,
Olaf Mandel

[Mike Frysinger]

Post by Olaf Mandel
I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
end-to-end tests on a network server. My current test script looks
AT_CHECK([server&], , [ignore])
AT_CHECK([client --cmd]), , [expected-output])
AT_CHECK([killall server], , [ignore])
This is far from perfect, however: there is no cleanup if the client
check fails or if the executable is called differently (see below).
Now I want to combine this with valgrind checking (from
AX_VALGRIND_CHECK()): I defined the check-TESTS target in the Makefile
so it calls "make check-autotest" and renames the log files to the
expected names and I added a macro to the test script to use an
AC_DEFUN([WRAP_CMD], [${RUNNER} $(which $1) $2])
AT_CHECK(WRAP_CMD([server], [&]), , [ignore])
AT_CHECK(WRAP_CMD([client], [--cmd]), , [expected-output])
AT_CHECK(WRAP_CMD([killall], [server]), , [ignore])
But this fails to kill the server simply because the actual command
valgrind --error-exitcode=1 --quiet ./server &
So the simple killall call is no longer feasible.
Now my question: is there a standard and approved way to start a
background process in a GNU Autotest script and to kill it at or before
the AT_CLEANUP ?

wouldn't you want the test itself to spin up/down the server as need
be ? that way you can write multiple end-to-end tests and have them
all run in parallel.

i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.
-mike

[Olaf Mandel]

Hello Mike,

Post by Mike Frysinger

Post by Olaf Mandel
I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
end-to-end tests on a network server. [...]
AT_CHECK([server&], , [ignore])
AT_CHECK([client --cmd]), , [expected-output])
AT_CHECK([killall server], , [ignore])

-Snipp-

Post by Mike Frysinger

Post by Olaf Mandel
Now I want to combine this with valgrind checking [...]
valgrind --error-exitcode=1 --quiet ./server &

-Snipp-

Post by Mike Frysinger
wouldn't you want the test itself to spin up/down the server as need
be ? that way you can write multiple end-to-end tests and have them
all run in parallel.

You mean my "client" program starting the server itself? Didn't think of
that... it would solve the cleanup question during testing. I see that
valgrind provides the --trace-children=yes option: I will have to try if
this gives suitable debugging information (I am mostly interested in
memory debugging the real server, not the client that was only written
for testing).

Post by Mike Frysinger
i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.

Right. This would already fail because the TCP port would be blocked and
there is no reliable way to feed back the port number from the server to
the client (and no: I can't use Unix sockets instead of TCP ports
without extending some external library). But here the suggestion of
forking the server from the client may help as well: the client can try
starting the server on different ports until one works.

Thank you for the suggestion,
Olaf

[Mike Frysinger]

Post by Olaf Mandel

Post by Mike Frysinger

Post by Olaf Mandel
I am trying to use GNU Autotest (via AX_GNU_AUTOTEST()) to run
end-to-end tests on a network server. [...]
AT_CHECK([server&], , [ignore])
AT_CHECK([client --cmd]), , [expected-output])
AT_CHECK([killall server], , [ignore])

-Snipp-

Post by Mike Frysinger

Post by Olaf Mandel
Now I want to combine this with valgrind checking [...]
valgrind --error-exitcode=1 --quiet ./server &

-Snipp-

Post by Mike Frysinger
wouldn't you want the test itself to spin up/down the server as need
be ? that way you can write multiple end-to-end tests and have them
all run in parallel.

You mean my "client" program starting the server itself? Didn't think of
that... it would solve the cleanup question during testing. I see that
valgrind provides the --trace-children=yes option: I will have to try if
this gives suitable debugging information (I am mostly interested in
memory debugging the real server, not the client that was only written
for testing).

Post by Mike Frysinger
i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.

Right. This would already fail because the TCP port would be blocked and
there is no reliable way to feed back the port number from the server to
the client (and no: I can't use Unix sockets instead of TCP ports
without extending some external library). But here the suggestion of
forking the server from the client may help as well: the client can try
starting the server on different ports until one works.

probing for an open port would be one way to handle it. if you can
conditionally rely on Linux namespace features, you could create a
new network namespace for each test which would give you a completely
blank slate with localhost available.
$ unshare -Urn
# ifconfig lo up
# ip a s
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group
default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: ***@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1
link/sit 0.0.0.0 brd 0.0.0.0
3: ***@NONE: <NOARP> mtu 1452 qdisc noop state DOWN group default qlen 1
link/tunnel6 :: brd ::
# netstat -nat
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address State
-mike

[Olaf Mandel]

Hello Mike,
-Snipp-

Post by Mike Frysinger

Post by Mike Frysinger
wouldn't you want the test itself to spin up/down the server as need
be ? that way you can write multiple end-to-end tests and have them
all run in parallel.

You mean my "client" program starting the server itself? [...]
I will have to try if
this gives suitable debugging information (I am mostly interested in
memory debugging the real server, not the client that was only written
for testing).

I finally got it working exactly as I want: thank you for the solution.

Post by Mike Frysinger

Post by Mike Frysinger
i'm guessing your method described above also doesn't work when you
try to run all the tests in parallel.

-Snipp-

Post by Mike Frysinger
probing for an open port would be one way to handle it. if you can
conditionally rely on Linux namespace features, you could create a
new network namespace for each test which would give you a completely
blank slate with localhost available.
$ unshare -Urn

-Snipp-

Decided not to do that because I didn't need the parallel testing on one
machine and because my default user does not have the required
permissions. But now that I learned of this I will keep it in mind
should the need arise: thanks for pointing it out.

Best regards,
Olaf