docs/Testing.rst

:orphan:

.. @raise litre.TestsAreMissing

=============
Testing Swift
=============

This document describes how we test the Swift compiler, the Swift runtime, and
the Swift standard library.

Testing approaches
==================

We use multiple approaches to test the Swift toolchain.

* LLVM lit-based testsuites for the compiler, runtime and the standard library.

* Unit tests for sub-tools.

* A selection of open source projects written in Swift.

The LLVM lit-based testsuite
============================

**Purpose**: primary testsuites for the Swift toolchain.

**Contents**: Functional and regression tests for all toolchain components.

**Run by**:

* Engineers and contributors are expected to run tests from these testsuites
  locally before committing.  (Usually on a single platform, and not necessarily
  all tests.)

* Buildbots run all tests, on all supported platforms.

Testsuite subsets
-----------------

The testsuite is split into four subsets:

* Primary testsuite, located under ``swift/test``.

* Validation testsuite, located under ``swift/validation-test``.

* Unit tests, located under ``swift/unittests``.

* Long tests, which are marked with ``REQUIRES: long_test``.

  Unlike other tests, every long test should also include either
  ``REQUIRES: nonexecutable_test`` or ``REQUIRES: executable_test``.

Running the LLVM lit-based testsuite
------------------------------------

It is recommended that you run the Swift test suites via ``utils/build-script``.
For day-to-day work on the Swift compiler, using ``utils/build-script --test``
should be sufficient.  The buildbot runs validation tests, so if those are
accidentally broken, it should not go unnoticed.

Before committing a large change to a compiler (especially a language change),
or API changes to the standard library, it is recommended to run validation
test suite, via ``utils/build-script --validation-test``.

Although it is not recommended for day-to-day contributions, it is also
technically possible to execute the tests directly via CMake. For example, if you have
built Swift products at the directory ``build/Ninja-ReleaseAssert/swift-macosx-x86_64``,
you may run the entire test suite directly using the following command:

.. code-block:: bash

  cmake --build build/Ninja-ReleaseAssert/swift-macosx-x86_64 -- check-swift-macosx-x86_64

Note that ``check-swift`` is suffixed with a target operating system and architecture.
Besides ``check-swift``, other targets are also available. Here's the full list:

* ``check-swift``

  Runs tests from the ``${SWIFT_SOURCE_ROOT}/test`` directory.

* ``check-swift-only_validation``

  Runs tests from the ``${SWIFT_SOURCE_ROOT}/validation-test`` directory.

* ``check-swift-validation``

  Runs the primary and validation tests, without the long tests.

* ``check-swift-only_long``

  Runs long tests only.

* ``check-swift-all``

  Runs all tests (primary, validation, and long).

* ``SwiftUnitTests``

  Builds all unit tests.  Executables are located under
  ``${SWIFT_BUILD_ROOT}/unittests`` and must be run individually.

For every target above, there are variants for different optimizations:

* the target itself (e.g., ``check-swift``) -- runs all tests from the primary
  testsuite.  The execution tests are run in ``-Onone`` mode.

* the target with ``-optimize`` suffix (e.g., ``check-swift-optimize``) -- runs
  execution tests in ``-O`` mode.  This target will only run tests marked as
  ``executable_test``.

* the target with ``-optimize-unchecked`` suffix (e.g.,
  ``check-swift-optimize-unchecked``) -- runs execution tests in
  ``-Ounchecked`` mode. This target will only run tests marked as
  ``executable_test``.

* the target with ``-executable`` suffix (e.g.,
  ``check-swift-executable-iphoneos-arm64``) -- runs tests marked with
  ``executable_test`` in ``-Onone`` mode.

* the target with ``-non-executable`` suffix (e.g.,
  ``check-swift-non-executable-iphoneos-arm64``) -- runs tests not marked with
  ``executable_test`` in ``-Onone`` mode.

If you need to manually run certain tests, you can invoke LLVM's lit.py script
directly. For example:

.. code-block:: bash

    % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv ${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/Parse/

This runs the tests in the test/Parse/ directory targeting the 32-bit iOS
Simulator. The ``-sv`` options give you a nice progress bar and only show you
output from the tests that fail.

One downside of using this form is that you're appending relative paths from
the source directory to the test directory in your build directory. (That is,
there may not actually be a directory named 'Parse' in
'test-iphonesimulator-i386/'; the invocation works because there is one in the
source 'test/' directory.) There is a more verbose form that specifies the
testing configuration explicitly, which then allows you to test files
regardless of location.

.. code-block:: bash

    % ${LLVM_SOURCE_ROOT}/utils/lit/lit.py -sv --param swift_site_config=${SWIFT_BUILD_ROOT}/test-iphonesimulator-i386/lit.site.cfg ${SWIFT_SOURCE_ROOT}/test/Parse/

For more complicated configuration, copy the invocation from one of the build
targets mentioned above and modify it as necessary. lit.py also has several
useful features, like timing tests and providing a timeout. Check these features
out with ``lit.py -h``.

Extra lit.py invocation options
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

* ``--param gmalloc`` will run all tests under Guard Malloc (macOS only). See
  ``man libgmalloc`` for more information.

* ``--param swift-version=<MAJOR>`` overrides the default Swift language
  version used by swift/swiftc and swift-ide-test.

* ``--param interpret`` is an experimental option for running execution tests
  using Swift's interpreter rather than compiling them first. Note that this
  does not affect all substitutions.

* ``--param swift_test_mode=<MODE>`` drives the various suffix variations
  mentioned above. Again, it's best to get the invocation from the existing
  build system targets and modify it rather than constructing it yourself.

Writing tests
-------------

General guidelines
^^^^^^^^^^^^^^^^^^

When adding a new testcase, try to find an existing test file focused on the
same topic rather than starting a new test file.  There is a fixed runtime cost
for every test file.  On the other hand, avoid dumping new tests in a file that
is only remotely related to the purpose of the new tests.

Don't limit a test to a certain platform or hardware configuration just because
this makes the test slightly easier to write.  This sometimes means a little
bit more work when adding the test, but the payoff from the increased testing
is significant.  We heavily rely on portable tests to port Swift to other
platforms.

Avoid using unstable language features in tests which test something else (for
example, avoid using an unstable underscored attribute when another
non-underscored attribute would work).

Avoid using arbitrary implementation details of the standard library.  Always
prefer to define types locally in the test, if feasible.

Avoid purposefully shadowing names from the standard library, this makes the
test extremely confusing (if nothing else, to understand the intent --- was the
compiler bug triggered by this shadowing?)  When reducing a compiler testcase
from the standard library source, rename the types and APIs in the testcase to
differ from the standard library APIs.

In IRGen, SILGen and SIL tests, avoid using platform-dependent implementation
details of the standard library (unless doing so is point of the test).
Platform-dependent details include:

* ``Int`` (use integer types with explicit types instead).

* Layout of ``String``, ``Array``, ``Dictionary``, ``Set``.  These differ
  between platforms that have Objective-C interop and those that don't.

Unless testing the standard library, avoid using arbitrary standard library
types and APIs, even if it is very convenient for you to do so in your tests.
Using the more common APIs like ``Array`` subscript or ``+`` on ``IntXX`` is
acceptable.  This is important because you can't rely on the full standard
library being available.  The long-term plan is to introduce a mock, minimal
standard library that only has a very basic set of APIs.

If you write an executable test please add ``REQUIRES: executable_test`` to the
test.

Substitutions in lit tests
^^^^^^^^^^^^^^^^^^^^^^^^^^

Substitutions that start with ``%target`` configure the compiler for building
code for the target that is not the build machine:

* ``%target-typecheck-verify-swift``: parse and type check the current Swift file
  for the target platform and verify diagnostics, like ``swift -frontend -typecheck -verify
  %s``.

  Use this substitution for testing semantic analysis in the compiler.

* ``%target-swift-frontend``: run ``swift -frontend`` for the target.

  Use this substitution (with extra arguments) for tests that don't fit any
  other pattern.

* ``%target-swift-frontend(mock-sdk:`` *mock sdk arguments* ``)`` *other
  arguments*: like ``%target-swift-frontend``, but allows to specify command
  line parameters (typically ``-sdk`` and ``-I``) to use a mock SDK and SDK
  overlay that would take precedence over the target SDK.

* ``%target-build-swift``: compile and link a Swift program for the target.

  Use this substitution only when you intend to run the program later in the
  test.

* ``%target-run-simple-swift``: build a one-file Swift program and run it on
  the target machine.

  Use this substitution for executable tests that don't require special
  compiler arguments.

  Add ``REQUIRES: executable_test`` to the test.

* ``%target-run-stdlib-swift``: like ``%target-run-simple-swift`` with
  ``-parse-stdlib -Xfrontend -disable-access-control``.

  This is sometimes useful for testing the Swift standard library.

  Add ``REQUIRES: executable_test`` to the test.

* ``%target-repl-run-simple-swift``: run a Swift program in a REPL on the
  target machine.

* ``%target-run``: run a command on the target machine.

  Add ``REQUIRES: executable_test`` to the test.

* ``%target-jit-run``: run a Swift program on the target machine using a JIT
  compiler.

* ``%target-swiftc_driver``: run ``swiftc`` for the target.

* ``%target-sil-opt``: run ``sil-opt`` for the target.

* ``%target-sil-func-extractor``: run ``sil-func-extractor`` for the target.

* ``%target-swift-ide-test``: run ``swift-ide-test`` for the target.

* ``%target-swift-ide-test(mock-sdk:`` *mock sdk arguments* ``)`` *other
  arguments*: like ``%target-swift-ide-test``, but allows to specify command
  line parameters to use a mock SDK.

* ``%target-swift-autolink-extract``: run ``swift-autolink-extract`` for the
  target to extract its autolink flags on platforms that support them (when the
  autolink-extract feature flag is set)

* ``%target-clang``: run the system's ``clang++`` for the target.

  If you want to run the ``clang`` executable that was built alongside
  Swift, use ``%clang`` instead.

* ``%target-ld``: run ``ld`` configured with flags pointing to the standard
  library directory for the target.

* ``%target-cc-options``: the clang flags to setup the target with the right
  architecture and platform version.

* ``%target-triple``: a triple composed of the ``%target-cpu``, the vendor,
  the ``%target-os``, and the operating system version number. Possible values
  include ``i386-apple-ios7.0`` or ``armv7k-apple-watchos2.0``.

* ``%target-cpu``: the target CPU instruction set (``i386``, ``x86_64``,
  ``armv7``, ``armv7k``, ``arm64``).

* ``%target-os``: the target operating system (``macosx``, ``darwin``,
  ``linux``, ``freebsd``, ``windows-cygnus``, ``windows-gnu``).

* ``%target-object-format``: the platform's object format (``elf``, ``macho``,
  ``coff``).

* ``%target-runtime``: the platform's Swift runtime (objc, native).

* ``%target-ptrsize``: the pointer size of the target (32, 64).

* ``%target-swiftmodule-name`` and ``%target-swiftdoc-name``: the basename of
  swiftmodule and swiftdoc files for a framework compiled for the target (for
  example, ``arm64.swiftmodule`` and ``arm64.swiftdoc``).

* ``%target-sdk-name``: only for Apple platforms: ``xcrun``-style SDK name
  (``macosx``, ``iphoneos``, ``iphonesimulator``).

* ``%target-static-stdlib-path``: the path to the static standard library.

  Add ``REQUIRES: static_stdlib`` to the test.

Always use ``%target-*`` substitutions unless you have a good reason.  For
example, an exception would be a test that checks how the compiler handles
mixing module files for incompatible platforms (that test would need to compile
Swift code for two different platforms that are known to be incompatible).

When you can't use ``%target-*`` substitutions, you can use:

* ``%swift_driver_plain``: run ``swift`` for the build machine.

* ``%swift_driver``: like ``%swift_driver_plain`` with ``-module-cache-path``
  set to a temporary directory used by the test suite, and using the
  ``SWIFT_TEST_OPTIONS`` environment variable if available.

* ``%swiftc_driver``: like ``%target-swiftc_driver`` for the build machine.

* ``%swiftc_driver_plain``: like ``%swiftc_driver``, but does not set the
  ``-module-cache-path`` to a temporary directory used by the test suite,
  and does not respect the ``SWIFT_TEST_OPTIONS`` environment variable.

* ``%sil-opt``: like ``%target-sil-opt`` for the build machine.

* ``%sil-func-extractor``: run ``%target-sil-func-extractor`` for the build machine.

* ``%lldb-moduleimport-test``: run ``lldb-moduleimport-test`` for the build
  machine in order simulate importing LLDB importing modules from the
  ``__apple_ast`` section in Mach-O files. See
  ``tools/lldb-moduleimport-test/`` for details.

* ``%swift-ide-test``: like ``%target-swift-ide-test`` for the build machine.

* ``%swift-ide-test_plain``: like ``%swift-ide-test``, but does not set the
  ``-module-cache-path`` or ``-completion-cache-path`` to temporary directories
  used by the test suite.

* ``%swift``: like ``%target-swift-frontend`` for the build machine.

* ``%clang``: run the locally-built ``clang``. To run ``clang++`` for the
  target, use ``%target-clang``.

Other substitutions:

* ``%clang-include-dir``: absolute path of the directory where the Clang
  include headers are stored on Linux build machines.

* ``%clang-importer-sdk``: FIXME.

* ``%clang_apinotes``: run ``clang -cc1apinotes`` using the locally-built
  clang.

* ``%sdk``: only for Apple platforms: the ``SWIFT_HOST_VARIANT_SDK`` specified
  by tools/build-script. Possible values include ``IOS`` or ``TVOS_SIMULATOR``.

* ``%gyb``: run ``gyb``, a boilerplate generation script. For details see
  ``utils/gyb``.

* ``%platform-module-dir``: absolute path of the directory where the standard
  library module file for the target platform is stored.  For example,
  ``/.../lib/swift/macosx``.

* ``%platform-sdk-overlay-dir``: absolute path of the directory where the SDK
  overlay module files for the target platform are stored.

* ``%{python}``: run the same Python interpreter that's being used to run the
  current ``lit`` test.

* ``%FileCheck``: like the LLVM ``FileCheck`` utility, but occurrences of full
  paths to the source and build directories in the input text are replaced with
  path-independent constants.

* ``%raw-FileCheck``: the LLVM ``FileCheck`` utility.

When writing a test where output (or IR, SIL) depends on the bitness of the
target CPU, use this pattern::

  // RUN: %target-swift-frontend ... | %FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-ptrsize %s

  // CHECK: common line
  // CHECK-32: only for 32-bit
  // CHECK-64: only for 64-bit

  // FileCheck does a single pass for a combined set of CHECK lines, so you can
  // do this:
  //
  // CHECK: define @foo() {
  // CHECK-32: integer_literal $Builtin.Int32, 0
  // CHECK-64: integer_literal $Builtin.Int64, 0

When writing a test where output (or IR, SIL) depends on the target CPU itself,
use this pattern::

  // RUN: %target-swift-frontend ... | %FileCheck --check-prefix=CHECK --check-prefix=CHECK-%target-cpu %s

  // CHECK: common line
  // CHECK-i386:        only for i386
  // CHECK-x86_64:      only for x86_64
  // CHECK-armv7:       only for armv7
  // CHECK-arm64:       only for arm64
  // CHECK-powerpc64:   only for powerpc64
  // CHECK-powerpc64le: only for powerpc64le

Features for ``REQUIRES`` and ``XFAIL``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

FIXME: full list.

* ``swift_ast_verifier``: present if the AST verifier is enabled in this build.

* When writing a test specific to x86, if possible, prefer ``REQUIRES:
  CPU=i386_or_x86_64`` to ``REQUIRES: CPU=x86_64``.

* ``swift_test_mode_optimize[_unchecked|none]`` and
  ``swift_test_mode_optimize[_unchecked|none]_<CPUNAME>``: specify a test mode
  plus cpu configuration.

* ``optimized_stdlib_<CPUNAME>``: an optimized stdlib plus cpu configuration.

* ``SWIFT_VERSION=<MAJOR>``: restricts a test to Swift 3 or Swift 4. If you
  need to use this, make sure to add a test for the other version as well
  unless you are specifically testing ``-swift-version``-related functionality.

* ``XFAIL: linux``: tests that need to be adapted for Linux, for example parts
  that depend on Objective-C interop need to be split out.

Feature ``REQUIRES: executable_test``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

This feature marks an executable test. The test harness makes this feature
generally available. It can be used to restrict the set of tests to run.

StdlibUnittest
^^^^^^^^^^^^^^

Tests accept command line parameters, run StdlibUnittest-based test binary
with ``--help`` for more information.

Testing memory management in execution tests
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In execution tests, memory management testing should be performed
using local variables enclosed in a closure passed to the standard
library ``autoreleasepool`` function. For example::

  // A counter that's decremented by Canary's deinitializer.
  var CanaryCount = 0

  // A class whose instances increase a counter when they're destroyed.
  class Canary {
    deinit { ++CanaryCount }
  }

  // Test that a local variable is correctly released before it goes out of
  // scope.
  CanaryCount = 0
  autoreleasepool {
    let canary = Canary()
  }
  assert(CanaryCount == 1, "canary was not released")

Memory management tests should be performed in a local scope because Swift does
not guarantee the destruction of global variables. Code that needs to
interoperate with Objective-C may put references in the autorelease pool, so
code that uses an ``if true {}`` or similar no-op scope instead of
``autoreleasepool`` may falsely report leaks or fail to catch overrelease bugs.
If you're specifically testing the autoreleasing behavior of code, or do not
expect code to interact with the Objective-C runtime, it may be OK to use ``if
true {}``, but those assumptions should be commented in the test.