| _ _ ____ _ |
| ___| | | | _ \| | |
| / __| | | | |_) | | |
| | (__| |_| | _ <| |___ |
| \___|\___/|_| \_\_____| |
| |
| The cURL Test Suite |
| |
| Requires: |
| perl (and a unix-style shell) |
| diff (when a test fails, a diff is shown) |
| stunnel (for HTTPS and FTPS tests) |
| OpenSSH or SunSSH (for SCP, SFTP and SOCKS4/5 tests) |
| |
| Ports used by default: |
| |
| - TCP/8990 for HTTP |
| - TCP/8991 for HTTPS |
| - TCP/8992 for FTP |
| - TCP/8993 for FTPS |
| - TCP/8994 for HTTP IPv6 |
| - TCP/8995 for FTP (2) |
| - TCP/8996 for FTP IPv6 |
| - UDP/8997 for TFTP |
| - UDP/8998 for TFTP IPv6 |
| - TCP/8999 for SCP/SFTP |
| - TCP/9000 for SOCKS |
| - TCP/9001 for POP3 |
| - TCP/9002 for IMAP |
| - TCP/9003 for SMTP |
| |
| The test suite runs simple FTP, POP3, IMAP, SMTP, HTTP and TFTP stand-alone |
| servers on these ports to which it makes requests. For SSL tests, it runs |
| stunnel to handle encryption to the regular servers. For SSH, it runs a |
| standard OpenSSH server. For SOCKS4/5 tests SSH is used to perform the SOCKS |
| functionality and requires a SSH client and server. |
| |
| The base port number shown above can be changed using runtests' -b option |
| to allow running more than one instance of the test suite simultaneously |
| on one machine. |
| |
| Run: |
| 'make test'. This builds the test suite support code and invokes the |
| 'runtests.pl' perl script to run all the tests. Edit the top variables |
| of that script in case you have some specific needs, or run the script |
| manually (after the support code has been built). |
| |
| The script breaks on the first test that doesn't do OK. Use -a to prevent |
| the script from abort on the first error. Run the script with -v for more |
| verbose output. Use -d to run the test servers with debug output enabled as |
| well. Specifying -k keeps all the log files generated by the test intact. |
| |
| Use -s for shorter output, or pass test numbers to run specific tests only |
| (like "./runtests.pl 3 4" to test 3 and 4 only). It also supports test case |
| ranges with 'to', as in "./runtests 3 to 9" which runs the seven tests from |
| 3 to 9. Any test numbers starting with ! are disabled, as are any test |
| numbers found in the file data/DISABLED (one per line). |
| |
| Shell startup scripts: |
| Tests which use the ssh test server, SCP/SFTP/SOCKS tests, might be badly |
| influenced by the output of system wide or user specific shell startup |
| scripts, .bashrc, .profile, /etc/csh.cshrc, .login, /etc/bashrc, etc. which |
| output text messages or escape sequences on user login. When these shell |
| startup messages or escape sequences are output they might corrupt the |
| expected stream of data which flows to the sftp-server or from the ssh |
| client which can result in bad test behaviour or even prevent the test |
| server from running. |
| |
| If the test suite ssh or sftp server fails to start up and logs the message |
| 'Received message too long' then you are certainly suffering the unwanted |
| output of a shell startup script. Locate, cleanup or adjust the shell |
| script. |
| |
| Memory: |
| The test script will check that all allocated memory is freed properly IF |
| curl has been built with the CURLDEBUG define set. The script will |
| automatically detect if that is the case, and it will use the ../memanalyze |
| script to analyze the memory debugging output. |
| |
| The -t option will enable torture testing mode, which runs each test |
| many times but causes a different memory allocation to fail on each |
| successive run. This tests the out of memory error handling code to |
| ensure that memory leaks do not occur even in those situations. |
| |
| Debug: |
| If a test case fails, you can conveniently get the script to invoke the |
| debugger (gdb) for you with the server running and the exact same command |
| line parameters that failed. Just invoke 'runtests.pl <test number> -g' and |
| then just type 'run' in the debugger to perform the command through the |
| debugger. |
| |
| If a test case causes a core dump, analyze it by running gdb like: |
| |
| # gdb ../curl/src core |
| |
| ... and get a stack trace with the gdb command: |
| |
| (gdb) where |
| |
| Logs: |
| All logs are generated in the logs/ subdirectory (it is emptied first |
| in the runtests.pl script). Use runtests.pl -k to keep the temporary files |
| after the test run. |
| |
| Data: |
| All test cases are put in the data/ subdirectory. Each test is stored in the |
| file named according to the test number. |
| |
| See FILEFORMAT for the description of the test case files. |
| |
| Code coverage: |
| gcc provides a tool that can determine the code coverage figures for |
| the test suite. To use it, configure curl with |
| CFLAGS='-fprofile-arcs -ftest-coverage -g -O0'. Make sure you run the normal |
| and torture tests to get more full coverage, i.e. do: |
| |
| make test |
| make test-torture |
| |
| The graphical tool ggcov can be used to browse the source and create |
| coverage reports on *NIX hosts: |
| |
| ggcov -r lib src |
| |
| The text mode tool gcov may also be used, but it doesn't handle object files |
| in more than one directory very well. |
| |
| Remote testing: |
| The runtests.pl script provides some hooks to allow curl to be tested on a |
| machine where perl can not be run. The test framework in this case runs on |
| a workstation where perl is available, while curl itself is run on a remote |
| system using ssh or some other remote execution method. See the comments at |
| the beginning of runtests.pl for details. |
| |
| TEST CASE NUMBERS |
| |
| So far, we've used this system: |
| |
| 1 - 99 HTTP |
| 100 - 199 FTP* |
| 200 - 299 FILE* |
| 300 - 399 HTTPS |
| 400 - 499 FTPS |
| 500 - 599 libcurl source code tests, not using the curl command tool |
| 600 - 699 SCP/SFTP |
| 700 - 799 SOCKS4 (even numbers) and SOCK5 (odd numbers) |
| 800 - 899 POP3, IMAP, SMTP |
| 1000 - 1999 miscellaneous* |
| 2000 - x multiple sequential protocols per test case* |
| |
| Since 30-apr-2003, there's nothing in the system that requires us to keep |
| within these number series, and those sections marked with * actually |
| contain tests for a variety of protocols. Each test case now specifies |
| its own server requirements, independent of test number. |
| |
| TODO: |
| |
| * Add tests for TELNET, LDAP, DICT... |
| * SOCKS4/5 test deficiencies - no proxy authentication tests as SSH (the |
| test mechanism) doesn't support them |