| Reproducers |
| =========== |
| |
| As unbelievable as it may sound, the debugger has bugs. These bugs might |
| manifest themselves as errors, missing results or even a crash. Quite often |
| these bugs don't reproduce in simple, isolated scenarios. The debugger deals |
| with a lot of moving parts and subtle differences can easily add up. |
| |
| Reproducers in LLDB improve the experience for both the users encountering bugs |
| and the developers working on resolving them. The general idea consists of |
| *capturing* all the information necessary to later *replay* a debug session |
| while debugging the debugger. |
| |
| .. contents:: |
| :local: |
| |
| Usage |
| ----- |
| |
| Reproducers are a generic concept in LLDB and are not inherently coupled with |
| the command line driver. The functionality can be used for anything that uses |
| the SB API and the driver is just one example. However, because it's probably |
| the most common way users interact with lldb, that's the workflow described in |
| this section. |
| |
| Capture |
| ``````` |
| |
| Until reproducer capture is enabled by default, you need to launch LLDB in |
| capture mode. For the command line driver, this means passing ``--capture``. |
| You cannot enable reproducer capture from within LLDB, as this would be too |
| late to capture initialization of the debugger. |
| |
| .. code-block:: bash |
| |
| > lldb --capture |
| |
| In capture mode, LLDB will keep track of all the information it needs to replay |
| the current debug session. Most data is captured lazily to limit the impact on |
| performance. To create the reproducer, use the ``reproducer generate`` |
| sub-command. It's always possible to check the status of the reproducers with |
| the ``reproducer status`` sub-command. Note that generating the reproducer |
| terminates the debug session. |
| |
| .. code-block:: none |
| |
| (lldb) reproducer status |
| Reproducer is in capture mode. |
| (lldb) reproducer generate |
| Reproducer written to '/path/to/reproducer' |
| Please have a look at the directory to assess if you're willing to share the contained information. |
| |
| |
| The resulting reproducer is a directory. It was a conscious decision to not |
| compress and archive it automatically. The reproducer can contain potentially |
| sensitive information like object and symbol files, their paths on disk, debug |
| information, memory excerpts of the inferior process, etc. |
| |
| Replay |
| `````` |
| |
| It is strongly recommended to replay the reproducer locally to ensure it |
| actually reproduces the expected behavior. If the reproducer doesn't behave |
| correctly locally, it means there's a bug in the reproducer implementation that |
| should be addressed. |
| |
| To replay a reproducer, simply pass its path to LLDB through the ``--replay`` |
| flag. It is unnecessary to pass any other command line flags. The flags that |
| were passed to LLDB during capture are already part of the reproducer. |
| |
| .. code-block:: bash |
| |
| > lldb --replay /path/to/reproducer |
| |
| |
| During replay LLDB will behave similar to batch mode. The session should be |
| identical to the recorded debug session. The only expected differences are that |
| the binary being debugged doesn't actually run during replay. That means that |
| you won't see any of its side effects, like things being printed to the |
| terminal. Another expected difference is the behavior of the ``reproducer |
| generate`` command, which becomes a NOOP during replay. |
| |
| Augmenting a Bug Report with a Reproducer |
| ````````````````````````````````````````` |
| |
| A reproducer can significantly improve a bug report, but it in itself is not |
| sufficient. Always describe the expected and unexpected behavior. Just like the |
| debugger can have bugs, the reproducer can have bugs too. |
| |
| |
| Design |
| ------ |
| |
| |
| Replay |
| `````` |
| |
| Reproducers support two replay modes. The main and most common mode is active |
| replay. It's called active, because it's LLDB that is driving replay by calling |
| the captured SB API functions one after each other. The second mode is passive |
| replay. In this mode, LLDB sits idle until an SB API function is called, for |
| example from Python, and then replays just this individual call. |
| |
| Active Replay |
| ^^^^^^^^^^^^^ |
| |
| No matter how a reproducer was captured, they can always be replayed with the |
| command line driver. When a reproducer is passed with the `--replay` flag, the |
| driver short-circuits and passes off control to the reproducer infrastructure, |
| effectively bypassing its normal operation. This works because the driver is |
| implemented using the SB API and is therefore nothing more than a sequence of |
| SB API calls. |
| |
| Replay is driven by the ``Registry::Replay``. As long as there's data in the |
| buffer holding the API data, the next SB API function call is deserialized. |
| Once the function is known, the registry can retrieve its signature, and use |
| that to deserialize its arguments. The function can then be invoked, most |
| commonly through the synthesized default replayer, or potentially using a |
| custom defined replay function. This process continues, until more data is |
| available or a replay error is encountered. |
| |
| During replay only a function's side effects matter. The result returned by the |
| replayed function is ignored because it cannot be observed beyond the driver. |
| This is sound, because anything that is passed into a subsequent API call will |
| have been serialized as an input argument. This also works for SB API objects |
| because the reproducers know about every object that has crossed the API |
| boundary, which is true by definition for object return values. |
| |
| |
| Passive Replay |
| ^^^^^^^^^^^^^^ |
| |
| Passive replay exists to support running the API test suite against a |
| reproducer. The API test suite is written in Python and tests the debugger by |
| calling into its API from Python. To make this work, the API must transparently |
| replay itself when called. This is what makes passive replay different from |
| driver replay, where it is lldb itself that's driving replay. For passive |
| replay, the driving factor is external. |
| |
| In order to replay API calls, the reproducers need a way to intercept them. |
| Every API call is already instrumented with an ``LLDB_RECORD_*`` macro that |
| captures its input arguments. Furthermore, it also contains the necessary logic |
| to detect which calls cross the API boundary and should be intercepted. We were |
| able to reuse all of this to implement passive replay. |
| |
| During passive replay is enabled, nothing happens until an SB API is called. |
| Inside that API function, the macro detects whether this call should be |
| replayed (i.e. crossed the API boundary). If the answer is yes, the next |
| function is deserialized from the SB API data and compared to the current |
| function. If the signature matches, we deserialize its input arguments and |
| reinvoke the current function with the deserialized arguments. We don't need to |
| do anything special to prevent us from recursively calling the replayed version |
| again, as the API boundary crossing logic knows that we're still behind the API |
| boundary when we re-invoked the current function. |
| |
| Another big difference with driver replay is the return value. While this |
| didn't matter for driver replay, it's key for passive replay, because that's |
| what gets checked by the test suite. Luckily, the ``LLDB_RECORD_*`` macros |
| contained sufficient type information to derive the result type. |
| |
| Testing |
| ------- |
| |
| Reproducers are tested in the following ways: |
| |
| - Unit tests to cover the reproducer infrastructure. There are tests for the |
| provider, loader and for the reproducer instrumentation. |
| - Feature specific end-to-end test cases in the ``test/Shell/Reproducer`` |
| directory. These tests serve as integration and regression tests for the |
| reproducers infrastructure, as well as doing some sanity checking for basic |
| debugger functionality. |
| - The API and shell tests can be run against a replayed reproducer. The |
| ``check-lldb-reproducers`` target will run the API and shell test suite |
| twice: first running the test normally while capturing a reproducer and then |
| a second time using the replayed session as the test input. For the shell |
| tests this use a little shim (``lldb-repro``) that uses the arguments and |
| current working directory to transparently generate or replay a reproducer. |
| For the API tests an extra argument with the reproducer path is passed to |
| ``dotest.py`` which initializes the debugger in the appropriate mode. |
| Certain tests do not fit this paradigm (for example test that check the |
| output of the binary being debugged) and are skipped by marking them as |
| unsupported by adding ``UNSUPPORTED: lldb-repro`` to the top of the shell |
| test or adding the ``skipIfReproducer`` decorator for the API tests. |
| |
| Additional testing is possible: |
| |
| - It's possible to unconditionally capture reproducers while running the |
| entire test suite by setting the ``LLDB_CAPTURE_REPRODUCER`` environment |
| variable. Assuming no bugs in reproducers, this can also help to reproduce |
| and investigate test failures. |
| |
| Knows Issues |
| ------------ |
| |
| The reproducers are still a work in progress. Here's a non-exhaustive list of |
| outstanding work, limitations and known issues. |
| |
| - The VFS cannot deal with more than one current working directory. Changing |
| the current working directory during the debug session will break relative |
| paths. |
| - Not all SB APIs are properly instrumented. We need customer serialization |
| for APIs that take buffers and lengths. |
| - We leak memory during replay because the reproducer doesn't capture the end |
| of an object's life time. We need to add instrumentation to the destructor |
| of SB API objects. |
| - The reproducer includes every file opened by LLDB. This is overkill. For |
| example we do not need to capture source files for code listings. There's |
| currently no way to say that some file shouldn't be included in the |
| reproducer. |
| - We do not yet automatically generate a reproducer on a crash. The reason is |
| that generating the reproducer is too expensive to do in a signal handler. |
| We should re-invoke lldb after a crash and do the heavy lifting. |