| This README file describes the files and directories related -*- rst -*- |
| to the Python test suite under the current 'test' directory. |
| |
| - dotest.py |
| |
| Provides the test driver for the test suite. To invoke it, cd to the 'test' |
| directory and issue the './dotest.py' command or './dotest.py -v' for more |
| verbose output. '.dotest.py -h' prints out the help messge. |
| |
| A specific naming pattern is followed by the .py script under the 'test' |
| directory in order to be recognized by 'dotest.py' test driver as a module |
| which implements a test case, namely, Test*.py. |
| |
| Some example usages: |
| |
| 1. ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log0 |
| This runs the test suite and directs the run log to a file. |
| |
| 2. LLDB_LOG=/tmp/lldb.log GDB_REMOTE_LOG=/tmp/gdb-remote.log ./dotest.py -v . 2> ~/Developer/Log/lldbtest.log |
| This runs the test suite, with logging turned on for the lldb as well as |
| the process.gdb-remote channels and directs the run log to a file. |
| |
| - lldbtest.py |
| |
| Provides an abstract base class of lldb test case named 'TestBase', which in |
| turn inherits from Python's unittest.TestCase. The concrete subclass can |
| override lldbtest.TestBase in order to inherit the common behavior for |
| unittest.TestCase.setUp/tearDown implemented in this file. |
| |
| To provide a test case, the concrete subclass provides methods whose names |
| start with the letters test. For more details about the Python's unittest |
| framework, go to http://docs.python.org/library/unittest.html. |
| |
| ./command_source/TestCommandSource.py provides a simple example of test case |
| which overrides lldbtest.TestBase to exercise the lldb's 'command source' |
| command. The subclass should override the attribute 'mydir' in order for the |
| runtime to locate the individual test cases when running as part of a large |
| test suite or when running each test case as a separate Python invocation. |
| |
| The doc string provides more details about the setup required for running a |
| test case on its own. To run the whole test suite, 'dotest.py' is all you |
| need to do. |
| |
| - subdirectories of 'test' |
| |
| Most of them predate the introduction of the python test suite and contain |
| example C/C++/ObjC source files which get compiled into executables which are |
| to be exercised by the debugger. |
| |
| For such subdirectory which has an associated Test*.py file, it was added as |
| part of the Python-based test suite to test lldb functionality. |
| |
| Some of the subdirectories, for example, the 'help' subdirectory, do not have |
| C/C++/ObjC source files; they were created to house the Python test case which |
| does not involve lldb reading in an executable file at all. |
| |
| The sample_test directory contains examples of both a full and an "inline" |
| testcase that run a process to a breakpoint and check a local variable. These |
| are convenient starting points for adding new tests. |
| |
| - make directory |
| |
| Contains Makefile.rules, which can be utilized by test cases to write Makefile |
| based rules to build binaries for the inferiors. |
| |
| By default, the built executable name is a.out, which can be overwritten by |
| specifying your EXE make variable, via the Makefile under the specific test |
| directory or via supplying a Python dictionary to the build method in your |
| Python test script. An example of the latter can be found in |
| test/lang/objc/radar-9691614/TestObjCMethodReturningBOOL.py, where: |
| |
| def test_method_ret_BOOL_with_dsym(self): |
| """Test that objective-c method returning BOOL works correctly.""" |
| d = {'EXE': self.exe_name} |
| self.buildDsym(dictionary=d) |
| self.setTearDownCleanup(dictionary=d) |
| self.objc_method_ret_BOOL(self.exe_name) |
| |
| def test_method_ret_BOOL_with_dwarf(self): |
| """Test that objective-c method returning BOOL works correctly.""" |
| d = {'EXE': self.exe_name} |
| self.buildDwarf(dictionary=d) |
| self.setTearDownCleanup(dictionary=d) |
| self.objc_method_ret_BOOL(self.exe_name) |
| |
| def setUp(self): |
| # Call super's setUp(). |
| TestBase.setUp(self) |
| # We'll use the test method name as the exe_name. |
| self.exe_name = self.testMethodName |
| # Find the line number to break inside main(). |
| self.main_source = "main.m" |
| self.line = line_number(self.main_source, '// Set breakpoint here.') |
| |
| The exe names for the two test methods are equal to the test method names and |
| are therefore guaranteed different. |
| |
| - plugins directory |
| |
| Contains platform specific plugin to build binaries with dsym/dwarf debugging |
| info. Other platform specific functionalities may be added in the future. |
| |
| - unittest2 directory |
| |
| Many new features were added to unittest in Python 2.7, including test |
| discovery. unittest2 allows you to use these features with earlier versions of |
| Python. |
| |
| It currently has unittest2 0.5.1 from http://pypi.python.org/pypi/unittest2. |
| Version 0.5.1 of unittest2 has feature parity with unittest in Python 2.7 |
| final. If you want to ensure that your tests run identically under unittest2 |
| and unittest in Python 2.7 you should use unittest2 0.5.1. |
| |
| Later versions of unittest2 include changes in unittest made in Python 3.2 and |
| onwards after the release of Python 2.7. |
| |
| - dotest.pl |
| |
| In case you wonder, there is also a 'dotest.pl' perl script file. It was |
| created to visit each Python test case under the specified directory and |
| invoke Python's builtin unittest.main() on each test case. |
| |
| It does not take advantage of the test runner and test suite functionality |
| provided by Python's unitest framework. Its existence is because we want a |
| different way of running the whole test suite. As lldb and the Python test |
| suite become more reliable, we don't expect to be using 'dotest.pl' anymore. |
| |
| Note: dotest.pl has been moved to the attic directory. |
| |
| - Profiling dotest.py runs |
| |
| I used the following command line thingy to do the profiling on a SnowLeopard |
| machine: |
| |
| $ DOTEST_PROFILE=YES DOTEST_SCRIPT_DIR=/Volumes/data/lldb/svn/trunk/test /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/cProfile.py -o my.profile ./dotest.py -v -w 2> ~/Developer/Log/lldbtest.log |
| |
| After that, I used the pstats.py module to browse the statistics: |
| |
| $ python /System/Library/Frameworks/Python.framework/Versions/Current/lib/python2.6/pstats.py my.profile |
| |
| - Writing test cases: |
| |
| We strongly prefer writing test cases using the SB API's rather than |
| the runCmd & expect. Unless you are actually testing some feature |
| of the command line, please don't write command based tests. For |
| historical reasons there are plenty of examples of tests in the test |
| suite that use runCmd where they shouldn't, but don't copy them, |
| copy the plenty that do use the SB API's instead. |
| |
| The reason for this is that our policy is that we will maintain |
| compatibility with the SB API's. But we don't make any similar |
| guarantee about the details of command result format. If your test |
| is using the command line, it is going to have to check against the |
| command result text, and you either end up writing your check |
| pattern by checking as little as possible so you won't be exposed to |
| random changes in the text; in which case you can end up missing |
| some failure, or you test too much and it means irrelevant changes |
| break your tests. |
| |
| However, if you use the Python API's it is possible to check all the |
| results you want to check in a very explicit way, which makes the |
| tests much more robust. |
| |
| Even if you are testing that a command-line command does some |
| specific thing, it is still better in general to use the SB API's to |
| drive to the point where you want to run the test, then use |
| SBInterpreter::HandleCommand to run the command. You get the full |
| result text from the command in the command return object, and all |
| the part where you are driving the debugger to the point you want to |
| test will be more robust. |
| |
| The sample_test directory contains a standard and an "inline" test |
| that are good starting points for writing a new test. |
| |
| - Attaching in test cases: |
| |
| If you need to attach to inferiors in your tests, you must make sure |
| the inferior calls lldb_enable_attach(), before the debugger |
| attempts to attach. This function performs any platform-specific |
| processing needed to enable attaching to this process (e.g., on |
| Linux, we execute prctl(PR_SET_TRACER) syscall to disable |
| protections present in some Linux systems). |