WAD (Wrapped Application Debugger)

Author(s):
    David M. Beazley (beazley@cs.uchicago.edu)

Copyright (C) 2001
University of Chicago

This library is free software; you can redistribute it and/or
modify it under the terms of the GNU Lesser General Public
License as published by the Free Software Foundation; either
version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public
License along with this library; if not, write to the Free Software
Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

See the file COPYING for a complete copy of the LGPL.

$Id$

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!                     DISCLAIMER                         !!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

THIS IS EXPERIMENTAL UNMAINTAINED RESEARCH SOFTWARE THAT REPRESENTS
WORK IN PROGRESS.  IT IS NOT PORTABLE, IT HAS NOT BEEN EXHAUSTIVELY
TESTED, AND IT MIGHT NOT WORK AT ALL.  PLEASE KEEP AWAY FROM SMALL
CHILDREN, PETS, NUCLEAR REACTORS, AIR-TRAFFIC CONTROL, AND VOTING
MACHINES.

0. Supported Platforms 

This software is currently only known to work with 32-bit applications
on Sun Sparc Solaris 2.8 and recent i386-Linux systems. In addition,
there are numerous issues concerning the interaction of this software
with signal handling, thread libraries, and compilers.  Please read
this entire document before proceeding.

1. Introduction

WAD is an embedded error-recovery mechanism that attempts to convert
fatal errors such as SIGSEGV, SIGBUS, and SIGFPE into sensible error
messages and exceptions.  It is primarily designed to support
scripting language extension programming although it can also be used
with stand-alone C programs.

The primary goal of this system is to explore an alternative approach
to mixed scripting-compiled debugging.  It requires no modifications
or recompilation of existing software. Therefore, it should be
relatively easy to try out.  Feedback is welcome. Contributions and
modifications are even more welcome.

2. Compilation and Installation

WAD is not particularly portable and at this time, only two platforms
are supported: Sun Sparc Solaris and i386-Linux.

Installation is as follows:

      ./configure
      make
      make install

The build process creates the following shared libraries:

    libwad.so        -  Standalone WAD.  Can be linked with C/C++
                        programs.

    libwadpy.so      -  Python WAD.  Can be linked to Python extension
                        modules or imported on its own as 'import libwadpy'

    libwadtcl.so     -  Tcl WAD.  Can be linked to Tcl extension
                        modules or loaded as 'load libwadtcl.so'.

    libwadpl.so      -  Perl WAD.  Can be linked to Perl extension
                        modules or loaded as 'libwadpl'.

To install the libraries, simply type 'make install'.  This copies the libraries
to /usr/local/lib (unless you modify the makefile).

Notes:

   -  The Sun version of WAD has only been tested when compiled with the
      Sun Workshop C/C++ compilers.   However WAD works with other programs
      that have been compiled with gcc.  If gcc is installed on your
      machine, you may want to set the following environment variables
      before running configure:

           setenv CC cc
           setenv CXX CC
           ./configure
 
   -  You may need to modify the Makefile to point to the installed locations
      of various scripting language libraries if you have installed
      them in non-traditional locations. 

   -  The Linux version has only been tested with 2.2-12 and 2.2-14 kernels
      and the RedHat 6.x distribution.  Your mileage may vary.   There
      may be some compatibility issues related to glibc and other parts
      of the system as well.
     
3. Using WAD

WAD has no functional API nor does it have any command line options so
it's pretty easy to describe---simply link the appropriate WAD library with
your C code.  For example:

   % cc blah.c -lwad

Once linked, fatal errors will now produce stack traces. For example:

% ./a.out seg
starting.
Segmentation fault.
#2   0x400571eb in __libc_start_main() in 'libc-start.c', line 90
#1   0x08048b39 in main(argc=0x2,argv=0xbffffce4) in 'debug.c', line 62
#0   0x080489b3 in seg_crash(n=0x0) in 'debug.c', line 9

/r0/beazley/Projects/WAD/Wad/debug.c, line 9

      int *a = 0;
      if (n > 0) seg_crash(n-1);
 =>   *a = 3;
      return 1;
    }

For scripting languages, WAD works in a similar manner--simply link
your scripting language extension module with the appropriate WAD library
(wadpy, wadtcl, wadpl).  For example:

   % ld -G $(OBJS) -lwadpy -o pymodule.so

When the scripting module is loaded into the interpreter, WAD should
automatically initialize.

4. Debugging Modes

Due to WAD's experimental nature, a number of debugging modes can be set
through the use of environment variables.   These variables control WAD's
runtime behavior and cause the system to dump debugging information for
various stages of error recovery.   A lot of this data is pretty ugly and 
probably only of interest to you if you are trying to debug WAD itself.

WAD_DEBUG_SEGMENT       -  Displays information about the virtual memory
                           map and mapping of addresses to segments.

WAD_DEBUG_SYMBOL        -  Symbol table mapping.

WAD_DEBUG_OBJECT        -  Loading/Unloading of object files.

WAD_DEBUG_FILE          -  Loading/Unloading of raw files.

WAD_DEBUG_HOLD          -  Freezes WAD before it returns from the signal handler.
                           Useful if you need to attach a debugger to WAD itself.

WAD_DEBUG_STABS         -  Display stabs data.

WAD_DEBUG_RETURN        -  Display information about WAD return points.

WAD_DEBUG_SYMBOL_SEARCH -  Display all symbols in the symbol table that are
                           searched.

WAD_DEBUG_UNWIND        -  Display information about stack unwinding.

WAD_DEBUG_SIGNAL        -  Display information about signal handling.

WAD_DEBUG_INIT          -  Print initialization information.

WAD_NOSTACK             -  Do NOT use an alternative signal handling stack.
                           This may be necessary on certain Linux systems when
                           threads are being used.

WAD_ONESHOT             -  Disable WAD signal handler after first signal has
                           been received.

WAD_DEBUG_MEMORY        -  Print information about WAD memory use.

WAD_DEBUG_STRINGS       -  Print information about WAD string manager.

5.  Platform Specific Issues

General:

  -  WAD does not gracefully recover from errors that corrupt the call
     stack (i.e., buffer overlow).

  -  Errors that destroy the process heap may or may not be recoverable
     depending on what has been destroyed.

  -  WAD does not currently support 64 bit applications on any platform.

  -  If executables have been stripped, their symbol tables might not
     have enough information to recover from errors.  Therefore, if you
     are using Python, Tcl, or Perl from a binary distribution, you
     may want to rebuild non-stripped versions of these packages yourself.

  -  WAD only works with programs that utilize the ELF32 linking format
     and stabs debugging data.  Newer formats such as DWARF2 are not
     supported at this time.

  -  WAD does not correctly report argument values for structures or
     floating point numbers yet.

  -  Overly aggressive compiler optimization may lead to very strange
     WAD output.

Solaris:

  -  WAD is extremely slow at collecting debugging information
     from large applications. 

Linux:

  -  The interaction of threads and signals are particularly problematic
     on this platform and may cause WAD not to work at all.   Here are
     some specific thread-based issues that may arise:

     1.  WAD causes the program to crash immediately upon startup. 
         This appears to be caused by a bug in in the implementation
         of sigaction() and the initialization of signals.  This 
         only occurs if WAD is directly linked to an executable
         using threads.  It does not occur when WAD is dynamically
         loaded into a threaded application.

     2.  Programs may lock up when an error occurs.  This is sometimes
         caused by an apparently broken implementation of sigaltstack().
         One solution to this is to set the following environment
         variable:

             setenv WAD_NOSTACK

         in which case the WAD signal handler will use the same
         stack as the thread/process that generates the error.

     3.  WAD just crashes altogether and doesn't seem to do anything.
         It appears that some versions of Linux threads do *not*
         pass CPU context information correctly to signal handlers
         defined in threaded programs.   There is no known fix to
         this at this time.  Upgrade your system.

  -  WAD does not work if it is compiled as PIC code.  The WAD libraries
     should be compiled *without* the -fpic option.

  -  WAD has to rely upon a heuristic register recovery scheme when it
     returns to scripting language interpreters.  It seems to
     work, but it relies upon a very specific compiler code generation
     convention for saving registers in function prologues.  It also
     relies upon the register save conventions described in the Linux
     Assembly HOWTO.

  -  If you are using WAD with pre-installed binaries for Python, Tcl,
     and other scripting languages, it may not work correctly due to
     stripped symbol tables.  Most Linux installs such as Redhat strip
     symbol tables from executables.  This makes it difficult for WAD
     to determine context correctly (although it may still work since
     the dynamic loading symbol table is still available in most cases).

6.  Language specific issues

If WAD is linked with a normal C/C++ program, errors simply produce a stack trace
that is printed on standard error.

Python:

WAD tries to raise a Python exception and return.  At this time, the exception
merely contains a traceback string.  However, in future versions, it may be
possible to access a complete exception object. 

Tcl:

WAD returns a Tcl and places the stack trace into the Tcl variable $errorInfo.
The wish shell uses this to dump error information.  

Perl:

Perl doesn't seem to have a very well-defined exception handling
mechanism.  Standard functions tend to just exit.  The WAD handler
produces a C stack trace and produces a Perl stack trace using some
code derived from the sigtrap module.

Note: 3/23/01 - Perl support is currently broken.

7. Testing and Examples

The Test directory contains some very simple code for testing WAD.  In the
most simple form, compile the stand-along test program 'debug' as follows:

% cd Test
% make

Now, running it:

% debug
WAD debug program.

Usage: debug type
   seg        - Fail with an uninitialized pointer.
   bus        - Fail with a bus error.
   abort      - Fail with an assertion error.
   math       - Fail with a math error.
   heap       - Blow the process heap.
   overflow   - Buffer overflow on the stack.

% debug seg
WAD debug program.
Segmentation fault.
#2   0x400581eb in __libc_start_main() in 'libc-start.c', line 90
#1   0x08048b61 in main(argc=0x2,argv=0xbffffc54) in 'debug.c', line 85
#0   0x080489d0 in seg_crash() in 'debug.c', line 15

/r0/beazley/Projects/WAD/Test/debug.c, line 15

    int seg_crash() {
      int *a = 0;
 =>   *a = 3;
      return 1;
    }

Additional targets 'make python', 'make tcl', and 'make perl' are also available. 
The scripts debug.py, debug.tcl, debug.pl can be used to test these extensions.

8.  Documentation

No official documentation exists at this time.  However, details
of WAD's design and implementation can be found in papers presented
at the Ninth International Python Conference and the 2000 USENIX
Technical Conference.  Both papers can be obtained at:

   http://systems.cs.uchicago.edu/wad


9. To-Do

If you find WAD to be interesting or useful, there are a variety of
ways to contribute.  Here is the short to-do list:

    -  Better register management.  Try to implement in a more portable
       way.   Add some support code for recovering local variables
       that happen to be stored in registers.

    -  Add heuristic for recovering functions called through an
       -fomit-frame-pointer compiler optimization scheme.   This
       can probably be determined by looking at the function preamble
       machine code.   Then one can back-trace to the calling function
       and look at it's preamble. 

    -  Continued clean up and modularization of the core.  Many of the
       internal APIs could be greatly improved.

    -  Support for ELF64 linking format.

    -  Support for DWARF2 debugging data.

    -  Improved support for stack-overflow and heap-corruption.  Although WAD
       probably won't be able to recover, it still might be able to produce some 
       informative diagnostics.

    -  Removal of printf() and other high-level library calls which may not
       operate with a corrupted heap.

    -  Better integration with scripting languages.

    -  Support for new platforms.

    -  Support for new scripting languages.

Please contact me if you are interested in working on any of these projects.

Dave Beazley (beazley@cs.uchicago.edu)
June 24, 2001