| ========================== |
| Source-based Code Coverage |
| ========================== |
| |
| .. contents:: |
| :local: |
| |
| Introduction |
| ============ |
| |
| This document explains how to use clang's source-based code coverage feature. |
| It's called "source-based" because it operates on AST and preprocessor |
| information directly. This allows it to generate very precise coverage data. |
| |
| Clang ships two other code coverage implementations: |
| |
| * :doc:`SanitizerCoverage` - A low-overhead tool meant for use alongside the |
| various sanitizers. It can provide up to edge-level coverage. |
| |
| * gcov - A GCC-compatible coverage implementation which operates on DebugInfo. |
| |
| From this point onwards "code coverage" will refer to the source-based kind. |
| |
| The code coverage workflow |
| ========================== |
| |
| The code coverage workflow consists of three main steps: |
| |
| * Compiling with coverage enabled. |
| |
| * Running the instrumented program. |
| |
| * Creating coverage reports. |
| |
| The next few sections work through a complete, copy-'n-paste friendly example |
| based on this program: |
| |
| .. code-block:: cpp |
| |
| % cat <<EOF > foo.cc |
| #define BAR(x) ((x) || (x)) |
| template <typename T> void foo(T x) { |
| for (unsigned I = 0; I < 10; ++I) { BAR(I); } |
| } |
| int main() { |
| foo<int>(0); |
| foo<float>(0); |
| return 0; |
| } |
| EOF |
| |
| Compiling with coverage enabled |
| =============================== |
| |
| To compile code with coverage enabled, pass ``-fprofile-instr-generate |
| -fcoverage-mapping`` to the compiler: |
| |
| .. code-block:: console |
| |
| # Step 1: Compile with coverage enabled. |
| % clang++ -fprofile-instr-generate -fcoverage-mapping foo.cc -o foo |
| |
| Note that linking together code with and without coverage instrumentation is |
| supported: any uninstrumented code simply won't be accounted for. |
| |
| Running the instrumented program |
| ================================ |
| |
| The next step is to run the instrumented program. When the program exits it |
| will write a **raw profile** to the path specified by the ``LLVM_PROFILE_FILE`` |
| environment variable. If that variable does not exist, the profile is written |
| to ``default.profraw`` in the current directory of the program. If |
| ``LLVM_PROFILE_FILE`` contains a path to a non-existent directory, the missing |
| directory structure will be created. Additionally, the following special |
| **pattern strings** are rewritten: |
| |
| * "%p" expands out to the process ID. |
| |
| * "%h" expands out to the hostname of the machine running the program. |
| |
| * "%Nm" expands out to the instrumented binary's signature. When this pattern |
| is specified, the runtime creates a pool of N raw profiles which are used for |
| on-line profile merging. The runtime takes care of selecting a raw profile |
| from the pool, locking it, and updating it before the program exits. If N is |
| not specified (i.e the pattern is "%m"), it's assumed that ``N = 1``. N must |
| be between 1 and 9. The merge pool specifier can only occur once per filename |
| pattern. |
| |
| .. code-block:: console |
| |
| # Step 2: Run the program. |
| % LLVM_PROFILE_FILE="foo.profraw" ./foo |
| |
| Creating coverage reports |
| ========================= |
| |
| Raw profiles have to be **indexed** before they can be used to generate |
| coverage reports. This is done using the "merge" tool in ``llvm-profdata``, so |
| named because it can combine and index profiles at the same time: |
| |
| .. code-block:: console |
| |
| # Step 3(a): Index the raw profile. |
| % llvm-profdata merge -sparse foo.profraw -o foo.profdata |
| |
| There are multiple different ways to render coverage reports. One option is to |
| generate a line-oriented report: |
| |
| .. code-block:: console |
| |
| # Step 3(b): Create a line-oriented coverage report. |
| % llvm-cov show ./foo -instr-profile=foo.profdata |
| |
| To generate the same report in html with demangling turned on, use: |
| |
| .. code-block:: console |
| |
| % llvm-cov show ./foo -instr-profile=foo.profdata -format html -o report.dir -Xdemangler c++filt -Xdemangler -n |
| |
| This report includes a summary view as well as dedicated sub-views for |
| templated functions and their instantiations. For our example program, we get |
| distinct views for ``foo<int>(...)`` and ``foo<float>(...)``. If |
| ``-show-line-counts-or-regions`` is enabled, ``llvm-cov`` displays sub-line |
| region counts (even in macro expansions): |
| |
| .. code-block:: none |
| |
| 1| 20|#define BAR(x) ((x) || (x)) |
| ^20 ^2 |
| 2| 2|template <typename T> void foo(T x) { |
| 3| 22| for (unsigned I = 0; I < 10; ++I) { BAR(I); } |
| ^22 ^20 ^20^20 |
| 4| 2|} |
| ------------------ |
| | void foo<int>(int): |
| | 2| 1|template <typename T> void foo(T x) { |
| | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } |
| | ^11 ^10 ^10^10 |
| | 4| 1|} |
| ------------------ |
| | void foo<float>(int): |
| | 2| 1|template <typename T> void foo(T x) { |
| | 3| 11| for (unsigned I = 0; I < 10; ++I) { BAR(I); } |
| | ^11 ^10 ^10^10 |
| | 4| 1|} |
| ------------------ |
| |
| It's possible to generate a file-level summary of coverage statistics (instead |
| of a line-oriented report) with: |
| |
| .. code-block:: console |
| |
| # Step 3(c): Create a coverage summary. |
| % llvm-cov report ./foo -instr-profile=foo.profdata |
| Filename Regions Missed Regions Cover Functions Missed Functions Executed Lines Missed Lines Cover |
| -------------------------------------------------------------------------------------------------------------------------------------- |
| /tmp/foo.cc 13 0 100.00% 3 0 100.00% 13 0 100.00% |
| -------------------------------------------------------------------------------------------------------------------------------------- |
| TOTAL 13 0 100.00% 3 0 100.00% 13 0 100.00% |
| |
| A few final notes: |
| |
| * The ``-sparse`` flag is optional but can result in dramatically smaller |
| indexed profiles. This option should not be used if the indexed profile will |
| be reused for PGO. |
| |
| * Raw profiles can be discarded after they are indexed. Advanced use of the |
| profile runtime library allows an instrumented program to merge profiling |
| information directly into an existing raw profile on disk. The details are |
| out of scope. |
| |
| * The ``llvm-profdata`` tool can be used to merge together multiple raw or |
| indexed profiles. To combine profiling data from multiple runs of a program, |
| try e.g: |
| |
| .. code-block:: console |
| |
| % llvm-profdata merge -sparse foo1.profraw foo2.profdata -o foo3.profdata |
| |
| Interpreting reports |
| ==================== |
| |
| There are four statistics tracked in a coverage summary: |
| |
| * Function coverage is the percentage of functions which have been executed at |
| least once. A function is treated as having been executed if any of its |
| instantiations are executed. |
| |
| * Instantiation coverage is the percentage of function instantiations which |
| have been executed at least once. |
| |
| * Line coverage is the percentage of code lines which have been executed at |
| least once. |
| |
| * Region coverage is the percentage of code regions which have been executed at |
| least once. A code region may span multiple lines (e.g a large function with |
| no control flow). However, it's also possible for a single line to contain |
| multiple code regions (e.g some short-circuited logic). |
| |
| Format compatibility guarantees |
| =============================== |
| |
| * There are no backwards or forwards compatibility guarantees for the raw |
| profile format. Raw profiles may be dependent on the specific compiler |
| revision used to generate them. It's inadvisable to store raw profiles for |
| long periods of time. |
| |
| * Tools must retain **backwards** compatibility with indexed profile formats. |
| These formats are not forwards-compatible: i.e, a tool which uses format |
| version X will not be able to understand format version (X+k). |
| |
| * There is a third format in play: the format of the coverage mappings emitted |
| into instrumented binaries. Tools must retain **backwards** compatibility |
| with these formats. These formats are not forwards-compatible. |
| |
| Using the profiling runtime without static initializers |
| ======================================================= |
| |
| By default the compiler runtime uses a static initializer to determine the |
| profile output path and to register a writer function. To collect profiles |
| without using static initializers, do this manually: |
| |
| * Export a ``int __llvm_profile_runtime`` symbol from each instrumented shared |
| library and executable. When the linker finds a definition of this symbol, it |
| knows to skip loading the object which contains the profiling runtime's |
| static initializer. |
| |
| * Forward-declare ``void __llvm_profile_initialize_file(void)`` and call it |
| once from each instrumented executable. This function parses |
| ``LLVM_PROFILE_FILE``, sets the output path, and truncates any existing files |
| at that path. To get the same behavior without truncating existing files, |
| pass a filename pattern string to ``void __llvm_profile_set_filename(char |
| *)``. These calls can be placed anywhere so long as they precede all calls |
| to ``__llvm_profile_write_file``. |
| |
| * Forward-declare ``int __llvm_profile_write_file(void)`` and call it to write |
| out a profile. This function returns 0 when it succeeds, and a non-zero value |
| otherwise. Calling this function multiple times appends profile data to an |
| existing on-disk raw profile. |
| |
| Drawbacks and limitations |
| ========================= |
| |
| * Code coverage does not handle unpredictable changes in control flow or stack |
| unwinding in the presence of exceptions precisely. Consider the following |
| function: |
| |
| .. code-block:: cpp |
| |
| int f() { |
| may_throw(); |
| return 0; |
| } |
| |
| If the call to ``may_throw()`` propagates an exception into ``f``, the code |
| coverage tool may mark the ``return`` statement as executed even though it is |
| not. A call to ``longjmp()`` can have similar effects. |