a debugging trick: how to peek into m4's processing

[Bruno Haible]

Hi Eric, Ralf,

For about 15 years, writing and debugging autoconf macros has been very hard
for me. I've come to partially understand how m4 internally works, thanks to
Eric's new documentation in the m4 manual, and due to the ability of "seeing"
what m4 does when I call it directly, on the command line.

But inside autoconf, it's more complicated, because when I call 'aclocal'
or 'autoconf', additional processing is done outside m4, which obstructs
the view on what happens inside m4.

- Before AC_INIT, I don't get any output at all.

- Trying to inspect the definition of a macro, through e.g.
m4_defn([AM_INIT_AUTOMAKE])
leads to tons of errors
configure.ac:19: error: possibly undefined macro: AC_PREREQ
configure.ac:20: error: possibly undefined macro: AC_INIT
configure.ac:25: error: possibly undefined macro: AM_INIT_AUTOMAKE
configure.ac:32: error: possibly undefined macro: AC_PROG_CC
configure.ac:33: error: possibly undefined macro: AC_PROG_INSTALL
configure.ac:45: error: possibly undefined macro: AC_SUBST
configure.ac:81: error: possibly undefined macro: AM_CONDITIONAL
configure:2533: error: possibly undefined macro: _m4_defun_pro
and produces no configure file that I could look at.

I figure that what I need is a most basic and robust 'println' device.

- autoconf has a --trace option but no example how to use it.
Actually what I want is to see some particular expansions, not
the arguments and results of macro calls.

- The output of m4_syscmd does not appear on standard output.

- m4 has no primitives for creating external files.

Here comes the trick: If, say, I want to see the definition of AM_INIT_AUTOMAKE
at a particular point, I do:

------------------------------------------------------------------
m4_syscmd([cat 1>&2 <<\EOT

Here comes the definition of AM_INIT_AUTOMAKE:]
m4_defn([AM_INIT_AUTOMAKE])
[EOT])
------------------------------------------------------------------

Benefits:
- Offers an unobstructed view to the m4 state.
- Any m4 expression, any number of m4 expressions can be printed.
- The results appear on the screen immediately. No need to open a file
to view them.

Could something like this be documented in the autoconf manual?

Or, alternatively, document how to use --trace for the same purpose?

Bruno


PS: The concrete problem for which I needed this, is to find out where to
use [...], m4_quote, m4_dquote in an expression with m4_bpatsubst and
m4_defn. It is fine if Eric the wizard can explain to me which of the 10
possibilities is the right one. But it's even better if the autoconf manual
would give me the tools to find out about it myself.


m4_syscmd([cat 1>&2 <<\EOT

[Here comes the definition of _AC_INIT_PACKAGE 1:]

m4_bpatsubst(m4_defn([_AC_INIT_PACKAGE]), [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 2:]

m4_bpatsubst(m4_quote(m4_defn([_AC_INIT_PACKAGE])), [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 3:]

m4_bpatsubst(m4_quote([m4_defn([_AC_INIT_PACKAGE])]), [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 4:]

m4_bpatsubst(m4_dquote(m4_defn([_AC_INIT_PACKAGE])), [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 5:]

m4_bpatsubst(m4_dquote([m4_defn([_AC_INIT_PACKAGE])]), [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 6:]

m4_bpatsubst([m4_defn([_AC_INIT_PACKAGE])], [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 7:]

m4_bpatsubst([m4_quote(m4_defn([_AC_INIT_PACKAGE]))], [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 8:]

m4_bpatsubst([m4_quote([m4_defn([_AC_INIT_PACKAGE])])], [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 9:]

m4_bpatsubst([m4_dquote(m4_defn([_AC_INIT_PACKAGE]))], [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[Here comes the definition of _AC_INIT_PACKAGE 10:]

m4_bpatsubst([m4_dquote([m4_defn([_AC_INIT_PACKAGE])])], [AC_PACKAGE_NAME], [gl_INIT_DUMMY])

[EOT])

[Eric Blake]

Post by Bruno Haible
Hi Eric, Ralf,
For about 15 years, writing and debugging autoconf macros has been very hard
for me. I've come to partially understand how m4 internally works, thanks to
Eric's new documentation in the m4 manual, and due to the ability of "seeing"
what m4 does when I call it directly, on the command line.
- m4 has no primitives for creating external files.

True, but m4sugar has m4_file_append to append to an arbitrary file (oh,
but that macro is still undocumented).

Post by Bruno Haible
Here comes the trick: If, say, I want to see the definition of AM_INIT_AUTOMAKE
------------------------------------------------------------------
m4_syscmd([cat 1>&2 <<\EOT
Here comes the definition of AM_INIT_AUTOMAKE:]
m4_defn([AM_INIT_AUTOMAKE])
[EOT])
------------------------------------------------------------------

Very similar to m4_file_append, except that it outputs to stderr rather
than appending to a file. For that matter, m4_errprint does that, without
having to spawn a subprocess.

Post by Bruno Haible
- Offers an unobstructed view to the m4 state.
- Any m4 expression, any number of m4 expressions can be printed.
- The results appear on the screen immediately. No need to open a file
to view them.
Could something like this be documented in the autoconf manual?

I think so. Done with the patch below.

Post by Bruno Haible
Or, alternatively, document how to use --trace for the same purpose?

Hmm, --trace shows arguments, not definition, but it also has a place.

- --
Don't work too hard, make some time for fun as well!

Eric Blake ***@byu.net

[Bruno Haible]

Hi Eric,

Post by Eric Blake

Post by Bruno Haible
------------------------------------------------------------------
m4_syscmd([cat 1>&2 <<\EOT
Here comes the definition of AM_INIT_AUTOMAKE:]
m4_defn([AM_INIT_AUTOMAKE])
[EOT])
------------------------------------------------------------------

Very similar to m4_file_append, except that it outputs to stderr rather
than appending to a file. For that matter, m4_errprint does that, without
having to spawn a subprocess.

Glad to see that m4_file_append and m4_errprintn already exist. I did not
know about them.

Post by Eric Blake
Hmm, --trace shows arguments, not definition, but it also has a place.

Yes, I agree documenting --trace by example is good as well.

Post by Eric Blake
+m4_init
+m4_errprintn([try one: m4_dquote is ]m4_defn([m4_dquote]))
+m4_divert([0])dnl
+m4_errprintn([try two: m4_dquote is ]m4_defn([m4_dquote]))dnl
+m4_dquote([hi])
+EOF
+[hi]

In this second example, the file 'foo' is never being created, since autom4te
is not being passed the options "-o foo".

More importantly, your example shows m4_errprintn only in the context of
"autom4te --language=m4sugar". But it's more powerful than that: it also
works from inside an 'autoconf' run (with is more often what the user wants
to debug).

Here is a proposed patch to
- show m4_errprintn in the context of autoconf,
- separate the two techniques (m4_errprintn and diversions) into separate
examples,
- Tweak the output so that it fits in 79 columns (in 'info' output).


2009-07-25 Bruno Haible <***@clisp.org>

Clarify autom4te debugging tips.
* doc/autoconf.texi (Debugging via autom4te): Clarify that the two
techniques are independent.

*** doc/autoconf.texi.orig 2009-07-25 16:39:05.000000000 +0200
--- doc/autoconf.texi 2009-07-25 16:36:28.000000000 +0200
***************
*** 12454,12463 ****
version is 2.63b.95-3963
@end example

! Another trick is using @code{m4_errprintn} to output debugging messages
! to standard error with no further m4 expansion, and without interfering
! with the post-processing done to standard output. For example, contrast
! these two attempts to learn how @code{m4_dquote} is implemented:

@smallexample
$ @kbd{cat <<\EOF > foo.m4}
--- 12454,12485 ----
version is 2.63b.95-3963
@end example

! Another trick is to print out the expansion of various m4 expressions
! to standard error or to a file, with no further m4 expansion, and without
! interfering with the post-processing done to standard output. There are
! two ways to do this. One is through @code{m4_errprintn}, the other one
! through diversions in @command{autom4te}.
!
! @code{m4_errprintn} shows a given expression on standard error. For
! example, if you want to see the expansion of an autoconf primitive or
! of one of your autoconf macros, you can do it like this:
!
! @smallexample
! $ @kbd{cat <<\EOF > configure.ac}
! AC_INIT
! m4_errprintn([Here comes the definition of AC_DEFINE_UNQUOTED:])
! m4_errprintn(m4_defn([AC_DEFINE_UNQUOTED]))
! AC_OUTPUT
! EOF
! $ @kbd{autoconf}
! Here comes the definition of AC_DEFINE_UNQUOTED:
! _AC_DEFINE_Q([], $@@)
! @end smallexample
!
! You can also use diversions. For example, the following is an attempt
! to learn how @code{m4_dquote} is implemented. Note that diversion 0
! is standard output, and standard output is being redirected to a file
! through the @samp{-o output} option.

@smallexample
$ @kbd{cat <<\EOF > foo.m4}
***************
*** 12466,12491 ****
m4_divert([0])dnl
try two: [m4_dquote is ]m4_defn([m4_dquote])
m4_dquote([hi])
EOF
! $ @kbd{autom4te --language=m4sugar -o foo foo.m4}
foo.m4:2: error: possibly undefined macro: m4_dquote
! If this token and others are legitimate, please use m4_pattern_allow.
! See the Autoconf documentation.
! $ @kbd{cat foo}
! try two: m4_dquote is [$@@]
! [hi]
! $ @kbd{cat <<\EOF > foo.m4}
! m4_init
! m4_errprintn([try one: m4_dquote is ]m4_defn([m4_dquote]))
! m4_divert([0])dnl
! m4_errprintn([try two: m4_dquote is ]m4_defn([m4_dquote]))dnl
! m4_dquote([hi])
! EOF
! $ @kbd{autom4te --language=m4sugar foo.m4}
! try one: m4_dquote is [$@@]
try two: m4_dquote is [$@@]
- $ @kbd{cat foo}
[hi]
@end smallexample

@node Programming in M4sh
--- 12488,12503 ----
m4_divert([0])dnl
try two: [m4_dquote is ]m4_defn([m4_dquote])
m4_dquote([hi])
+ m4_dquote(tic, tac, toe)
EOF
! $ @kbd{autom4te --language=m4sugar -o output foo.m4}
foo.m4:2: error: possibly undefined macro: m4_dquote
! If this token and others are legitimate, please use m4_pattern_allow.
! See the Autoconf documentation.
! $ @kbd{cat output}
try two: m4_dquote is [$@@]
[hi]
+ [tic],[tac],[toe]
@end smallexample

@node Programming in M4sh

[Eric Blake]

Post by Bruno Haible

Post by Eric Blake
+m4_init
+m4_errprintn([try one: m4_dquote is ]m4_defn([m4_dquote]))
+m4_divert([0])dnl
+m4_errprintn([try two: m4_dquote is ]m4_defn([m4_dquote]))dnl
+m4_dquote([hi])
+EOF
+[hi]

In this second example, the file 'foo' is never being created, since autom4te
is not being passed the options "-o foo".

Oops. Definitely needs fixing (or deletion - read on).

Post by Bruno Haible
More importantly, your example shows m4_errprintn only in the context of
"autom4te --language=m4sugar". But it's more powerful than that: it also
works from inside an 'autoconf' run (with is more often what the user wants
to debug).
Here is a proposed patch to
- show m4_errprintn in the context of autoconf,
- separate the two techniques (m4_errprintn and diversions) into separate
examples,

Using diversions is not a technique. Rather, it is an additional reason
(beyond postprocessing) why outputting to anything except stderr or
separate files is problematic, because diversions can be discarded or
rearranged - ultimately, all diversions except for KILL (-1) go to
standard out. So I'm just dropping the autom4te --language=m4sugar
example altogether.

Post by Bruno Haible
- Tweak the output so that it fits in 79 columns (in 'info' output).
Clarify autom4te debugging tips.
* doc/autoconf.texi (Debugging via autom4te): Clarify that the two
techniques are independent.

Git did not like your patch - I could not apply it with either 'git am' or
'git apply', so I had to do more work to get it into good shape to play
with it. Here's the followup I'm using:

- --
Don't work too hard, make some time for fun as well!

Eric Blake ***@byu.net

[Bruno Haible]

Hi Eric,

Post by Eric Blake
Git did not like your patch - I could not apply it with either 'git am' or
'git apply'

Patches that I send should be applied with "patch -p0". git probably did
not like my patch because it was a context diff. I send context diffs or
unified diffs, depending on which of the two is more readable: One-liner
fixes are more readable as unified diffs, whereas large block changes
(such as indentation changes) are more readable as context diffs.

Post by Eric Blake
I had to do more work to get it into good shape to play with it.

I'm sorry about that. Rule of thumb: If the mail has the form of a git
patch, use 'git am', otherwise use 'patch -p0' or 'patch -p1'.

Bruno

[Bruno Haible]

Post by Eric Blake
Using diversions is not a technique. Rather, it is an additional reason
(beyond postprocessing) why outputting to anything except stderr or
separate files is problematic, because diversions can be discarded or
rearranged

Yes, that's why I moved this example to be the last one.

Post by Eric Blake
So I'm just dropping the autom4te --language=m4sugar example altogether.

Fine with me as well :-)

Bruno