| <html> |
| <title> |
| Using SWIG to Control, Prototype, and Debug C Programs with Python |
| </title> |
| <body BGCOLOR="ffffff"> |
| <h1> Using SWIG to Control, Prototype, and Debug C Programs with Python </h1> |
| <b> |
| (Submitted to the 4th International Python Conference) <br> <br> |
| David M. Beazley <br> |
| Department of Computer Science <br> |
| University of Utah <br> |
| Salt Lake City, Utah 84112 <br> |
| <a href="http://www.cs.utah.edu/~beazley"> beazley@cs.utah.edu </a> <br> |
| </b> |
| <h2> Abstract </h2> |
| <em> |
| I discuss early results in using SWIG to interact with C and C++ programs |
| using Python. Examples of using SWIG are provided along with a |
| discussion of the benefits and limitations of this approach. This |
| paper describes work in progress with the hope of getting feedback and |
| ideas from the Python community. |
| </em> <br> <br> |
| |
| Note : SWIG is freely available at <a href="http://www.cs.utah.edu/~beazley/SWIG"> http://www.cs.utah.edu/~beazley/SWIG </a> |
| |
| <h2> Introduction </h2> |
| |
| SWIG (Simplified Wrapper and Interface Generator) is a code |
| development |
| tool designed to make it easy for scientists and engineers to add scripting |
| language interfaces to programs and libraries written in C and C++. |
| The first version of SWIG was developed in July 1995 |
| for use with |
| very large scale scientific applications running on the Connection |
| Machine 5 and Cray T3D at Los Alamos National Laboratory. |
| Since that time, the system has been extended to support several |
| interface languages including Python, Tcl, Perl4, Perl5, and Guile [1]. |
| |
| In this paper, I will describe how SWIG can be used to interact with |
| C and C++ code from Python. I'll also discuss some |
| applications and limitations of this approach. It is important to |
| emphasize that SWIG was primarily designed for scientists and engineers |
| who would like to have a nice interface, but who would rather |
| work on more interesting problems than figuring out how to build |
| a user interface or using a complicated interface generation tool. |
| |
| <h2> Introducing SWIG </h2> |
| |
| The idea behind SWIG is really quite simple---most interface |
| languages such as Python provide some mechanism for making |
| calls to functions written in C or C++. However, this almost |
| always requires the user to write special "wrapper" functions |
| that provide the glue between the scripting language and the |
| C functions. As an example, if you wanted to add the |
| <tt> getenv() </tt> function to Python, you would need to write |
| the following wrapper code [4] : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| static PyObject *wrap_getenv(PyObject *self, PyObject *args) { |
| char * result; |
| char * arg0; |
| if(!PyArg_ParseTuple(args, "s",&arg0)) |
| return NULL; |
| result = getenv(arg0); |
| return Py_BuildValue("s", result); |
| } |
| |
| </pre> </tt> |
| </blockquote> |
| While writing a single wrapper function isn't too difficult, |
| it quickly becomes tedious and error prone as the number |
| of functions increases. |
| SWIG automates this process by generating |
| wrapper code from a list ANSI C function and variable declarations. |
| As a result, adding scripting languages to C applications can become |
| an almost trivial exercise. <br> <br> |
| |
| The core of the SWIG consists of a YACC parser and a collection |
| of general purpose utility functions. The output of the parser |
| is sent to two different modules--a language module for writing |
| wrapper functions and a documentation module for producing a simple |
| reference guide. Each of the target languages and documentation methods |
| is implemented as C++ class that can be plugged in to the system. |
| Currently, Python, Tcl, Perl4, Perl5, and Guile are supported as |
| target languages while documentation can be produced in ASCII, HTML, |
| and LaTeX. Different target languages can be implemented as a new |
| C++ class that is linked with a general purpose SWIG library file. |
| |
| <h2> Previous work </h2> |
| |
| Most Python users are probably aware that automatic wrapper generation is |
| not a new idea. In particular, packages such as ILU are capable |
| of providing bindings between C, C++, and Python using specifications |
| given in IDL (Interface Definition Language) [3]. SWIG is not really meant to compete with this |
| approach, but is designed to be a no-nonsense, easy to use tool |
| that scientists and engineers can use to build interesting interfaces |
| without having to worry about grungy programming details or learning |
| a complicated interface specification language. |
| With that said, SWIG is certainly not designed to turn Python into |
| a C or C++ clone (I'm not sure that turning Python into interpreted |
| C/C++ compiler would be a good idea anyways). However, SWIG can allow |
| Python to interface with a surprisingly wide variety of C functions. |
| |
| </body> |
| </html> |
| |
| <h2> A SWIG example </h2> |
| |
| While it is not my intent to provide a tutorial, I hope to |
| illustrate how SWIG works by building a module from part of the |
| UNIX socket library. |
| While there is probably no need to do this in practice (since Python |
| already has a socket module), it illustrates most of SWIG's capabilities |
| on a moderately complex example and provides a basis for |
| comparison between an existing module and one created by SWIG. <br> <br> |
| |
| As input, SWIG takes an input file referred to as an "interface file." |
| The file starts with a preamble containing the name of the |
| module and a code block where special header files, additional C |
| code, and other things can be added. After that, ANSI C function |
| and variable declarations are listed in any order. Since SWIG |
| uses C syntax, it's usually fairly easy to build an interface |
| from scratch or simply by copying an existing header file. |
| The following SWIG interface file will be used for our socket example : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| // socket.i |
| // SWIG Interface file to play with some sockets |
| %init sock // Name of our module |
| %{ |
| #include <sys/types.h> |
| #include <sys/socket.h> |
| #include <netinet/in.h> |
| #include <arpa/inet.h> |
| #include <netdb.h> |
| |
| /* Set some values in the sockaddr_in structure */ |
| struct sockaddr *new_sockaddr_in(short family, unsigned long hostid, int port) { |
| struct sockaddr_in *addr; |
| addr = (struct sockaddr_in *) malloc(sizeof(struct sockaddr_in)); |
| bzero((char *) addr, sizeof(struct sockaddr_in)); |
| addr->sin_family = family; |
| addr->sin_addr.s_addr = hostid; |
| addr->sin_port = htons(port); |
| return (struct sockaddr *) addr; |
| } |
| /* Get host address, but return as a string instead of hostent */ |
| char *my_gethostbyname(char *hostname) { |
| struct hostent *h; |
| h = gethostbyname(hostname); |
| if (h) return h->h_name; |
| else return ""; |
| } |
| %} |
| |
| // Add these constants |
| enum {AF_UNIX, AF_INET, SOCK_STREAM, SOCK_DGRAM, SOCK_RAW, |
| IPPROTO_UDP, IPPROTO_TCP, INADDR_ANY}; |
| |
| #define SIZEOF_SOCKADDR sizeof(struct sockaddr) |
| |
| // Wrap these functions |
| int socket(int family, int type, int protocol); |
| int bind(int sockfd, struct sockaddr *myaddr, int addrlen); |
| int connect(int sockfd, struct sockaddr *servaddr, int addrlen); |
| int listen(int sockfd, int backlog); |
| int accept(int sockfd, struct sockaddr *peer, %val int *addrlen); |
| int close(int fd); |
| struct sockaddr *new_sockaddr_in(short family, unsigned long, int port); |
| %name gethostbyname { char *my_gethostbyname(char *); } |
| unsigned long inet_addr(const char *ip); |
| |
| %include unixio.i |
| </pre> |
| </tt> |
| </blockquote> |
| |
| There are several things to notice about this file |
| |
| <ul> |
| <li> The module name is specified with the <tt> %init </tt> directive. |
| <li> Header files and supporting C code is placed in <tt> %{,%} </tt>. |
| <li> Special support functions may be necessary. For example, <tt> |
| new_sockaddr_in() </tt> creates a new <tt> struct sockaddr_in </tt> type |
| and sets some of its values. |
| <li> Constants are created with <tt> enum </tt> or <tt> #define. </tt> |
| <li> Functions can take most C basic datatypes as well as pointers |
| to complex types. |
| <li> Functions can be renamed with the <tt> %name </tt> directive. |
| <li> The <tt> %val </tt> directive forces a function argument to be called by |
| value. |
| |
| <li> The <tt> %include </tt> directive can be used to include other |
| SWIG interface files. In this case, the file <tt> unixio.i </tt> |
| might look like the following : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| // File : unixio.i |
| // Some file I/O and memory functions |
| %{ |
| %} |
| int read(int fd, void *buffer, int n); |
| int write(int fd, void *buffer, int n); |
| typedef unsigned int size_t; |
| void *malloc(size_t nbytes); |
| </pre> |
| </tt> |
| </blockquote> |
| <li> Since interface files can include other interface files, it can be very |
| easy to build up libraries and collections of useful functions. |
| <li> <tt> typedef </tt> can be used to provide mapping between datatypes. |
| <li> C and C++ comments are allowed and like C, SWIG ignores whitespace. |
| </ul> |
| |
| The key thing to notice about SWIG interface files is that they support |
| real C functions, constants, and datatypes including C pointers. As |
| a general rule these files are also independent of the target scripting |
| language--the primary reason why it's easy to support different |
| languages. |
| |
| <h2> Building a Python module </h2> |
| |
| Building a Python module with SWIG is a simple process that proceeds as |
| follows (assuming you're running Solaris 2.x) : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| unix > wrap -python socket.i |
| unix > gcc -c socket_wrap.c -I/usr/local/include/Py |
| unix > ld -G socket_wrap.o -lsocket -lnsl -o sockmodule.so |
| </pre> |
| </tt> |
| </blockquote> |
| |
| The <tt> wrap </tt> command takes the interface file and produces a C file |
| called <tt> socket_wrap.c </tt>. This file is then compiled and built into |
| a shared object file. |
| |
| <h2> A sample Python script </h2> |
| |
| Our new socket module can be used normally within a Python script. |
| For example, we could write a simple server process to echo all |
| data received back to a client : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| # Echo server program using SWIG module |
| from sock import * |
| PORT = 5000 |
| sockfd = socket(AF_INET, SOCK_STREAM, 0) |
| if sockfd < 0: |
| raise IOError, 'server : unable to open stream socket' |
| addr = new_sockaddr_in(AF_INET,INADDR_ANY,PORT) |
| if (bind(sockfd, addr, SIZEOF_SOCKADDR)) < 0 : |
| raise IOError, 'server : unable to bind local address' |
| listen(sockfd,5) |
| client_addr = new_sockaddr_in(AF_INET, 0, 0) |
| newsockfd = accept(sockfd, client_addr, SIZEOF_SOCKADDR) |
| buffer = malloc(1024) |
| while 1: |
| nbytes = read(newsockfd, buffer, 1024) |
| if nbytes > 0 : |
| write(newsockfd,buffer,nbytes) |
| else : break |
| close(newsockfd) |
| close(sockfd) |
| </pre> |
| </tt> |
| </blockquote> |
| |
| In our Python module, we are manipulating a socket, in almost |
| exactly the same way as would be done in a C program. We'll take |
| a look at a client written using the Python socket library shortly. |
| |
| <h2> Pointers and run-time type checking </h2> |
| |
| SWIG imports C pointers as ASCII strings containing both the |
| address and pointer type. Thus, a typical SWIG pointer might look |
| like the following : <br> <br> |
| |
| <center> |
| <tt> _fd2a0_struct_sockaddr_p </tt> <br> <br> |
| </center> |
| |
| Some may view the idea of importing C pointers into Python as unsafe or even |
| truly insane. However, handling pointers seems to be needed in order |
| for SWIG to handle most interesting types of C code. To provide some safety, |
| the wrapper functions generated by SWIG perform pointer-type checking |
| (since importing C pointers into |
| Python has in effect, bypassed type-checking in the C compiler). |
| When incompatible pointer types are used, this is what happens : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| unix > python |
| Python 1.3 (Apr 12 1996) [GCC 2.5.8] |
| Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam |
| >>> from sock import * |
| >>> sockfd = socket(AF_INET, SOCK_STREAM,0); |
| >>> buffer = malloc(8192); |
| >>> bind(sockfd,buffer,SIZEOF_SOCKADDR); |
| Traceback (innermost last): |
| File "<stdin>", line 1, in ? |
| TypeError: Type error in argument 2 of bind. Expected _struct_sockaddr_p. |
| >>> |
| </pre> |
| </tt> |
| </blockquote> |
| |
| |
| SWIG's run-time type checking provides a certain degree of safety from |
| using invalid parameters and making simple mistakes. Of course, any system |
| supporting pointers can be abused, but SWIG's implementation has proven |
| to be quite reliable under normal use. |
| |
| <h2> Comparison with a real Python module </h2> |
| |
| As a point of comparison, we can now write a client script using the |
| Python socket module (this example courtesy of the Python library |
| reference manual) [5]. |
| |
| <blockquote> |
| <tt> |
| <pre> |
| # Echo client program |
| from socket import * |
| HOST = 'tjaze.lanl.gov' |
| PORT = 5000 |
| s = socket(AF_INET, SOCK_STREAM) |
| s.connect(HOST,PORT) |
| s.send('Hello world') |
| data = s.recv(1024) |
| s.close() |
| print 'Received', `data` |
| </pre> |
| </tt> |
| </blockquote> |
| |
| As one might expect, the two scripts look fairly similar. However |
| there are certain obvious differences. First, the SWIG generated |
| module is clearly a quick and dirty approach. We must explicitly |
| check for errors and create a few C data structures. Furthermore, |
| while Python functions such as <tt> socket() </tt> can take optional |
| arguments, SWIG generated functions can not (since the underlying |
| C function can't). Of course, the apparent ugliness |
| of the SWIG module compared to the Python equivalent is not the |
| point here. |
| The real idea to emphasize is that SWIG |
| can take a set of real C functions with moderate complexity and |
| produce a fully functional Python module in a very short period of |
| time. In this case, about 10 minutes worth of effort. |
| (I'm |
| guessing that the Python socket module required more time than that). |
| With a bit of work, the SWIG generated module could probably be |
| used to build a more user-friendly version that hid many of the details. |
| |
| <h2> Controlling C programs </h2> |
| |
| Of course, the real goal of SWIG is not to produce quirky |
| replacements for modules in the Python library. Instead, it |
| better suited as a mechanism for |
| controlling a variety of C programs. <br> <br> |
| |
| In the SPaSM molecular dynamics code used at Los Alamos National |
| Laboratory, SWIG |
| is used to build an interface out of about 200 C functions [2]. |
| This happens at compile time so most users don't |
| notice that it has occurred. However, as a result, it is now |
| extremely easy for the physicists using the code to |
| extend it with new functions. |
| Whenever new functionality is added, the new |
| functions are put into a SWIG interface file and the functions |
| become available when the code is recompiled (a process |
| which usually only involves the new C functions and SWIG generated |
| wrapper file). Of course, when running under Python, it is possible |
| to build extensions and dynamically load them as needed. <br> <br> |
| |
| Another point, not to be overlooked, is the fact that SWIG makes it |
| very easy for a user to combine bits and pieces of completely different |
| software packages without waiting for someone to write a special |
| purpose module. For example, SWIG can be used to import the C API |
| of Matlab 4.2 into Python. This combined with a simulation module |
| can allow a scientist to perform both simulation and data analysis in |
| interactively or from a script. One could even build a Tkinter interface to control |
| the entire system. The ease of using SWIG makes it possible for |
| scientists to build applications out of components that |
| might not have been considered earlier. |
| |
| <h2> Prototyping and Debugging </h2> |
| |
| When debugging new modules or libraries, it would be nice to be able |
| to test out the functions without having to repeatedly recompile the |
| module with the rest of a large application. With SWIG, it is often |
| possible to prototype and debug modules independently. SWIG can be |
| used to build Python wrappers around the functions and scripts |
| can be used for testing. In many cases, it is possible to create |
| the C data structures and other information that would have been generated |
| by the larger application all from within a Python script. Since |
| SWIG requires no changes to the C code, integrating the new C code into |
| the large package is usually pretty easy. <br> <br> |
| |
| A related benefit of the SWIG approach is that it is now possible |
| to easily experiment with large software libraries and packages. |
| Peter-Pike Sloan at the |
| University of Utah, used SWIG to wrap the entire contents of the Open-GL |
| library (including more than 560 constants and 340 C functions). |
| This ``interpreted'' Open-GL could then be used interactively to |
| determine various image parameters and to draw simple figures. |
| Is this case, it proved to be a remarkably effective way of |
| figuring out the effects of different parameters and functions without |
| having to recompile after every modification. |
| |
| <h2> Living dangerously with datatypes </h2> |
| |
| One of the reasons SWIG is easy to use for prototyping and control |
| applications is its extremely forgiving treatment of datatypes. |
| <tt> typedef </tt> can be used to map datatypes, but whenever |
| SWIG encounters an unknown datatype, it simply assumes that it's |
| some sort of complex datatype. As a result, it's possible to |
| wrap some fairly complex C code with almost no effort. For example, |
| one could create a module allowing Python to create a Tcl interpreter |
| and execute Tcl commands as follows : |
| |
| <blockquote> |
| <tt> |
| <pre> |
| // tcl.i |
| // Wrap Tcl's C interface |
| %init tcl |
| %{ |
| #include <tcl.h> |
| %} |
| |
| Tcl_Interp *Tcl_CreateInterp(void); |
| void Tcl_DeleteInterp(Tcl_Interp *interp); |
| int Tcl_Eval(Tcl_Interp *interp, char *script); |
| int Tcl_EvalFile(Tcl_Interp *interp, char *file); |
| </pre> |
| </tt> |
| </blockquote> |
| |
| SWIG has no idea what <tt> Tcl_Interp </tt> is, but it doesn't |
| really care as long as it's used as a pointer. When used in a |
| Python script, this module would work as follows : |
| <blockquote> |
| <tt> |
| <pre> |
| unix > python |
| Python 1.3 (Mar 26 1996) [GCC 2.7.0] |
| Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam |
| >>> import tcl |
| >>> interp = tcl.Tcl_CreateInterp(); |
| >>> tcl.Tcl_Eval(interp,"for {set i 0} {$i < 5} {incr i 1} {puts $i}"); |
| 0 |
| 1 |
| 2 |
| 3 |
| 4 |
| 0 |
| >>> |
| </pre> |
| </tt> |
| </blockquote> |
| |
| <h2> Limitations </h2> |
| |
| This approach represents a balance of flexibility and ease of use. |
| I have always felt that the tool shouldn't be more complicated than |
| the original problem. Therefore, SWIG certainly won't do everything. |
| Some of SWIG's limitations include it's limited understanding of |
| complex datatypes. Often the user must write special functions |
| to extract information from structures even though passing complex |
| datatypes around by reference works remarkably well in practice. |
| SWIG's support for C++ is also quite limited. At the moment SWIG |
| can be used to pass pointers to C++ objects around, but actually |
| transforming a C++ object into some sort of Python object is far |
| beyond the capabilities of the current implementation. Finally, SWIG |
| lacks an exception model which may be of concern to some users. |
| |
| <h2> Limitations in the Python implementation </h2> |
| |
| SWIG's support for Python is in many ways, limited by the fact that |
| SWIG was originally developed for use with other languages. For example, |
| representing C pointers as a character string can be seen as a carry-over |
| from an earlier Tcl implementation. With a little work, I believe |
| that one could create a Python "pointer" datatype while still retaining |
| type-checking and other features. <br> <br> |
| |
| Another limitation in the current Python implementation is the apparent lack |
| of variable linking support. Many applications, particularly scientific ones, |
| have global variables that a user may want to access. SWIG currently |
| creates Python functions for accessing these variables, but other |
| scripting languages provide a more natural mechanism. |
| For example, in Perl, it is |
| possible to attach "set" and "get" functions to a Perl object. These |
| are evaluated whenever that variable is accessed in a script. |
| For all practical purposes this special variable looks like any other Perl |
| variable, but is really linked to an C global variable (of course, |
| maybe there is a reason why Perl calls these "magic" variables). <br> <br> |
| |
| If an equivalent variable linking mechanism is available in Python, I |
| couldn't find it [4]. |
| If not, it might be possible to create a new kind of Python datatype |
| that supports variable linking, but which interacts seamlessly |
| with existing Python integer, floating point, and character string types. |
| |
| <h2> Summary and future work </h2> |
| |
| This paper has described work in progress with the SWIG system and |
| its support for Python. I believe that this approach shows great |
| promise for future work--especially within the scientific and |
| engineering community. Most of the people introduced to SWIG and |
| Python have |
| found both systems extremely easy and straightforward to use. |
| I am currently |
| quite interested in improving SWIG's interface to Python and exploring |
| its use with numerical Python in order to build interesting |
| and flexible scientific applications [6]. |
| |
| <h2> Acknowledgments </h2> |
| |
| This project would not have been possible without the support of a number |
| of people. Peter Lomdahl, Shujia Zhou, Brad Holian, Tim Germann, and Niels |
| Jensen at Los Alamos National Laboratory were the first users and were |
| instrumental in testing out the first designs. Patrick Tullmann, |
| John Schmidt,Peter-Pike Sloan, and Kurtis Bleeker at the University of Utah have also |
| been instrumental in testing out some of the newer versions and |
| providing feedback. |
| John Buckman has suggested many interesting |
| improvements and is currently working on porting SWIG to non-Unix |
| platforms including NT, OS/2 and Macintosh. |
| Finally |
| I'd like to thank Chris Johnson and the members of the Scientific |
| Computing and Imaging group at the University of Utah for their continued |
| support. Some of this work was performed under the auspices of the |
| Department of Energy. |
| |
| <h2> References </h2> |
| |
| [1] D.M. Beazley. <em> SWIG : An Easy to Use Tool for Integrating |
| Scripting Languages with C and C++ </em>. Proceedings of the 4th |
| USENIX Tcl/Tk Workshop (1996) (to appear). <br> <br> |
| |
| [2] D.M. Beazley and P.S. Lomdahl <em> Message-Passing Multi-Cell |
| Molecular Dynamics on the Connection Machine 5 </em>, Parallel |
| Computing 20 (1994), p. 173-195. <br> <br> |
| |
| [3] Bill Janssen and Mike Spreitzer, <em> ILU : Inter-Language |
| Unification via Object Modules</em>, OOPSLA 94 Workshop on |
| Multi-Language Object Models. |
| <br> <br> |
| |
| [4] Guido van Rossum, <em> Extending and Embedding the Python Interpreter, |
| </em> October 1995. <br> <br> |
| |
| [5] Guido van Rossum, <em> Python Library Reference </em>, October |
| 1995. <br> <br> |
| |
| [6] Paul F. Dubois, Konrad Hinsen, and James Hugunin, <em> Numerical |
| Python, </em> Computers in Physics, (1996) (to appear) |
| |
| </body> |
| </html> |
