SWIG-1.3.12, SWIG-2.0 and Beyond

<h2>SWIG-1.3.12, SWIG 2.0, and Beyond</h2>

<p>
Author: David Beazley (beazley@cs.uchicago.edu)

<p>
June 2, 2002

<p>
With the release of SWIG-1.3.12, I thought I'd take a few moments of
everyone's time to talk about the past, the present, and the future of
SWIG development.  I'm really quite excited about the current release
because I think it represents a huge turning point in SWIG's
development.  Furthermore, it is only the beginning of bigger and
better things to come.  However, we definitely need your help.

<p>
To put a little perspective on the discussion, I'd start with a few
development statistics.  In the last 12 months, there have been over
300 entries added to the CHANGES log and over 4000 CVS commits.
Although that may not sound like a lot compared to a huge software
project, it is significant in the context of SWIG.  As a point of
comparison, there has been more SWIG development this year than in any
other year of the project and more than in the previous three years
combined.  This even includes the first few years of development in
which there was also a lot of activity.  Furthermore, many of the
recent changes have been extremely non-trivial (e.g., templates,
namespaces, type system, operators, etc.).  As a result, SWIG is more
capable than I ever imagined possible.

<p>
Regrettably, I must admit that I've been a little negligent in
discussing the roadmap for where I thought this flurry of SWIG
development was actually headed.  In part, this is because I've been
buried in work.  However, the real reason is that I didn't really know
where we were going---except that in a time far far far away in the
future, we might arrive at some kind of "stable" release with a
version number other than "1.3.x".  Needless to say, that's not a very
compelling story.
<p>
That said, I've spent a lot of time thinking about SWIG and trying to
wrap my brain around it.  Specifically, just what is (or should be)
the primary focus of this project and what are we really trying to do?
That's what the rest of this message is about.

<h3>SWIG Prehistory</h3> 

<p>
The first version of SWIG was written in 1995.  The original system
was developed to help with some software wrapping problems I
encountered while writing molecular dynamics software at Los
Alamos. Later that year, I became interested in extending the wrapper
generator to support other scripting languages so it was rewritten in
C++ and modified with multiple backends (Tcl, Perl, and Guile).  This
led to a first public release in February, 1996.  Feedback from this
release led to a series of enhancements and the release of SWIG 1.0 in
September 1996.  Throughout this process, my intent was to create a
tool that I would want to use---I never viewed the project as an
CS experiment in programming languages or software engineering.

<h3>SWIG 1.1</h3>

SWIG-1.1 (June, 1997) represented a series of enhancements that were
added in response to feedback at conferences and from users.  Shadow
classes, exception handling, typemaps, and a number of more useful
features were added.  However, the overall structure of the system was
relatively unchanged from the initial version.  Following the release
of 1.1, a series of minor patch releases were made.  This resulted in
the release of SWIG-1.1p5 in February, 1998.  Unfortunately, this
release would remain the last "stable" release for quite a long
time---in fact, it is still listed as the last "stable" release on the
SWIG web page!

<h3>SWIG Hell</h3>

Even during the development of SWIG-1.1, it was clear that the whole
design of the system was deeply flawed.  The implementation was a mess
and the C/C++ support was full of holes and nasty corner cases.
Furthermore, there was no unifying principle that tied all of the
different SWIG directives together.  Not only that, fixing these
problems appeared to be nothing short of impossible---requiring a
total ground-up rewrite at best.  The only redeeming quality was that
the system basically worked "well enough," it was extensively
documented, and its flaws mostly known.  People could use it and there
were work-arounds for most common problems.

<p>
To deal with the design problem, there were at least four attempts to
completely rewrite SWIG, some of which were attempted in parallel with
the work on SWIG-1.1.  Unfortunately, none of these were ever
completed. The primary problem was a strong "second system" effect and
a desire to make SWIG do everything that one might conceivably want to
do with a wrapper generator (somehow).  Clearly, this was a recipe for
disaster. In fact, all such attempts to rewrite SWIG were eventually
abandoned several years ago.  In hindsight, I think the real problem
was that these rewrite efforts focused far too much attention on
implementation technique rather than principles.  In short, the
failure of these efforts was due to a lack of clarity in understanding
how SWIG ought to work (regardless of how it was actually
implemented).

<h3>SWIG Restart (1.3a1-1.3a5)</h3>

<p>
Having languished for several years, the SWIG1.1p5 release had a
growing pile of maintenance issues. It didn't work for newer versions
of certain language modules and a lot of minor bug reports and feature
requests had been building up.  With a lot of help from Loic Dachary and
Thien-Thi Nguyen, we put together the 1.3a1 release (February,
2000).  This was mostly a bug fix release to 1.1p5 except that the
preprocessor module from SWIG1.2 was added and a lot of minor
enhancements were added.

<p>
For the next six months, a massive effort was made to rewrite all of
SWIG's internal data structures (strings, lists, hashes, etc.).  This
work was all going on underneath the covers while we tried to keep
SWIG in an operational state. The primary focus of this work was
really one of cleanup.  Having given up on a total rewrite, perhaps
we could settle with making the implementation incrementally better 
than before.  In addition this, Matthias Koppe jumped on board to 
reimplement the Guile module and to make other improvements to the system.

<p>
An important aspect of these releases was that many parts of the
system not directly related to wrapper code generation were removed.
This included the documentation system and Objective-C support.  These
were not removed because they weren't useful.  Instead, the
documentation system was removed because it accounted for nearly half
of the special SWIG directives, yet had no direct bearing on what SWIG
actually did.  Obective-C support was removed because it was tangled
with C++ support in a way that was very difficult to understand and
maintain.  The removal of these features was seen as a way to vastly
simplify cleanup--and to buy some time where we could rethink their
role in a future release.

<h3>SWIG Redux (1.3.6-1.3.11)</h3>

This work, started in February 2001, is the beginning of the current
SWIG implementation.  With the help of William Fulton, Matthias Koppe,
Lyle Johnson, Luigi Ballabio, Jason Stewart, Richard Palmer, Sam
Liddicot, and others, this work can best be described as the wholesale
destruction of everything remaining from SWIG-1.1.  The language
module API, type system, the parser, numerous SWIG directives, and
SWIG library files---all destroyed or rewritten.  Not only that, we
started introducing significant incompatibilities with
SWIG-1.1---mostly in an effort to correct past wrongs and get
ourselves out of the tangled mess of earlier versions.  A huge number
of long-standing bugs and difficult feature requests have also been
resolved.

<p>
The general goal of this development could best be described as an
attempt to reduce SWIG to an easily described set of general "ideas"
about how it should operate.  Special SWIG directives have been
eliminated or combined with others.  Different parts of the system have
been better integrated.  Features not directly related to wrapper code
generation have been removed and the system has become more
focused. Changes in internal data structures and APIs have allowed
SWIG to capture more information from interface files and to resolve
hard corner cases.  More often than not, these are things that you
never notice unless you are an old user and you suddenly realize that
a problem you had several years back has disappeared.

<p>
Along with the destruction of old code, this work has quietly
introduced a new core--the most significant features of which are a
new C++ type system and multi-pass compilation.  More importantly,
this work has really tried to provide a more principled foundation for
future SWIG development.  However, just what is this "more principled
foundation?"

<h3>Convergence (1.3.12)</h3>

With the upcoming 1.3.12 release, SWIG is about to take a big leap
forward. Almost all of this is due to one realization---that almost
every hard problem in past SWIG releases has been directly related to
errors and limitations in its type system.  Types are the key to
understanding the structure of C and C++ code.  They are at the heart
of understanding advanced language features like namespaces, nested
classes, and templates.  They are directly related to the data
marshalling that occurs in wrappers.  Not only that, they interact
with nearly every SWIG directive.  A proper type system *is* the
necessary foundation for moving SWIG forward.

<p>
To be honest, I don't think that the emphasis on types is entirely
obvious.  In fact, a great deal of work in past SWIG rewrites has
focused on the problem of C++ parsing.  For instance, modifying the
parser to handle more advanced C++ code or representing parse trees as
XML.  Furthermore, if one looks at the SWIG mailing list, you can find
a *lot* of messages related to issues of C++ parsing whereas almost no
one ever talks about types (well, other than typemaps).  Even other
wrapper generation tools seems to spend a lot of time dealing with the
parsing issue. Although parsing is clearly important, I don't think it
has ever been the real problem in SWIG.  This is because even though a
parser can tell you what's in a header file, it doesn't tell you
anything about how the different pieces of the system behave or how
they might interact. To do that, you need to do a lot more than just
parsing--and that's really the whole point.

<p> 
Although earlier 1.3 releases have made big improvements to the type
system, SWIG-1.3.12 is the first release that really tries to tackle
the type-system issue in a major way.  We have patched nearly all
remaining holes in the type system and we have added full support for
C++ namespaces. Not only that, we have completely reimplemented C++
template support in a way that supports templates, member templates,
and template specialization.  Luigi and I are currently using this to
work on proper SWIG library support for parts of the C++ standard
library and the Standard Template Library (STL).  Although some crusty
C programmers (present company excepted), might balk at such apparent
insanity, this work has impacted all parts of SWIG at all levels.
Even a variety of subtle errors in C support have been fixed by this
work.

<p>
In addition to the type system work, SWIG-1.3.12 contains continued
reduction in the implementation. Directives have been removed,
refined, renamed, or consolidated.  We're still narrowing the focus of
the system and working towards some kind of convergence.  "Convergence
to what?!?", you ask.

<h3>So, what is SWIG?</h3>

In a nutshell, SWIG is a C/C++ declaration compiler that generates
wrapper code (okay, so you knew that much).  However, to really
understand what SWIG is doing and where SWIG-1.3.x is headed, it is
useful to know that the whole system is essentially being built around
three extensions to the C++ type system:

<ul>

<li><p><b>Typemaps.</b>  Typemaps are rules that define the process by which
      data is converted between languages.   They are fully integrated
      with the C++ type system and they are applied using a type-based
      pattern matching mechanism.  All data conversion SWIG is
      defined by typemaps and is fully reconfigurable.

<li><p><b>Declaration annotation.</b> There are special directives that modify
      the wrapping behavior of individual declarations. Declarations
      can be selectively identified and decorated with arbitrary
      attributes that affect wrapper generation.  Like typemaps, 
      declaration matching is fully integrated with the C++ type system. 
      Almost all SWIG customization directives are a form of declaration 
      annotation.   

<li><p><b>Class extension.</b>  The ability to extend classes and structures
      with new methods/attributes when building wrappers.  Classes
      are part of the type system so class extension is naturally 
      integrated with the C++ type system as well (big surprise).

</ul>
<p>
And that's it--this is the end-game of SWIG-1.3.x development.  When
stabilized and documented it will become SWIG-2.0.

<h3>The Bigger Picture</h3>

I really want to emphasize that all of this work is part of a much
bigger picture.  SWIG is used by a surprising number of people in
industry, government, and academia.  It's used to build systems used
by millions of people every day.  It has even shown up in video games
and other unlikely settings.  Although SWIG doesn't have the same
visibility as many large software projects, over 12000 people have
downloaded SWIG-1.3.11 in the last 4 months.  Clearly someone is using
it for something!  Because of this, I think it is important for us to
work on moving SWIG towards a more solid foundation.  Doing so will
give the system more credibility and long term viability---and it will
be a lot more fun to use!

<p>
It's also worth noting that there are some rather interesting CS
connections at work here.  Extensions to the type system and typemaps
have some interesting relations to work in programming languages.  The
SWIG declaration annotation system and class extension feature seem
oddly similar to work in the emerging area of Aspect Oriented
Programming (AOP).  There are undoubtedly connections to other areas
of software engineering and architecture.

<p>
The key point is that SWIG isn't going to connect to anything if
no-one can quite describe what it is or how it works.

<h3>SWIG-2.0 and the Future</h3>

SWIG-1.3.12 still represents work in progress. There are bugs, the
documentation is still incomplete, and there are parts of the
implementation that are rather ugly.  We are also still working out a
few very hard problems like nested classes, callback functions, and
overloading.  A few old features are still missing (Objective-C,
documentation).  However, I believe the end of the 1.3.x series is
near and achievable.

<p>
Over the summer, a few more 1.3.x releases may appear but the current
plan is to aim for a SWIG-2.0 release in September.  This release is
really moving towards the design principles described above and will
be a very major enhancement over SWIG-1.1.

<p>
As for the future, a number of interesting ideas are on the table.  I
want to add support for contracts/assertions in order to solve some
reliability issues that arise when retrofitting legacy codes with a
scripting interface.  Support for an extension language has been
promoted by David Fletcher and was suggested by someone else on the
mailing list rather recently.  I have a graduate student working on
SWIG support for the Microsoft CLR and .NET languages.  Other work
might include support for alternative parsers, dynamically loadable
language modules, and so forth.  However, none of this is really going
to materialize if we can't get the 2.0 release stablized.  In fact, I
see SWIG-2.0 as a necessary step for moving forward with these ideas.

<h3>We need your help! Yes, you.</h3>

Nobody gets paid to work on SWIG.  The developers are volunteers who
work in their spare time.  Furthermore, SWIG is not supported by
investors, a large corporation, or research grants.  I work on it
because it's fun, challenging, and useful.  I presume that other
developers do the same.  However, we only have limited resources and
we need your help.

<ul>
<li><p>
If you like SWIG and find it useful, we need you to try new versions.
   We want you to torture test the releases and to break them.  We need
   bug reports. No bug is too obscure or unimportant---we *will* fix it
   if we can.  We also need feedback about things that are annoying or
   compatibility features that might help in going from 1.1 to 2.0.

<li><p> We need help with documentation, examples, testing, libraries, and all
   sorts of other aspects of the release.  Even if you have never
   written a SWIG language module or dived into its implementation,
   there are many ways that you can help.  Consider writing a case study
   about how you wrapped a big library.  Contribute tests that break the
   implementation in horrible ways.  Help with the web page or FAQ.

<li><p> Most of the SWIG-1.3.x work has focused on the SWIG core.  However, as
   the 2.0 release nears, we will be working on a variety of enhancements
   to the language modules.  If there are things you would like to see
   in any of the language modules, you need to let us know.

<li><p> There are SWIG language modules that have not made it into the
   distribution.  Examples that I know about include ITCL, Swig-Eiffel,
   and Swig-Lua.  We would gladly make these part of the standard SWIG
   distribution.  However, we also need help to do it.  Porting from
   SWIG-1.1 is no easy task, but we're more than willing to help.  It's
   not as bad as one might imagine.

<li><p> We are always looking for developers.  Subscribe to
   to the <a href="mail.html">swig-dev</a> mailing list
   or send me email to get involved.
</ul>

<h3>Acknowledgements</h3>

I'd just like to thank everyone who has submitted feedback, bugs, made
contributions, and put up with my occasional thrashing over the years. 
I welcome any comments about this document and how we can make SWIG even
better.

<p>
Dave Beazley (beazley@cs.uchicago.edu) <br>
June 2, 2002