mypyc/doc/differences_from_python.rst

.. _differences-from-python:

Differences from Python
=======================

Mypyc aims to be sufficiently compatible with Python semantics so that
migrating code to mypyc often doesn't require major code
changes. There are various differences to enable performance gains
that you need to be aware of, however.

This section documents notable differences from Python. We discuss
many of them also elsewhere, but it's convenient to have them here in
one place.

Running compiled modules
------------------------

You can't use ``python3 <module>.py`` or ``python3 -m <module>``
to run compiled modules. Use ``python3 -c "import <module>"`` instead,
or write a wrapper script that imports your module.

As a side effect, you can't rely on checking the ``__name__`` attribute in compiled
code, like this::

    if __name__ == "__main__":  # Can't be used in compiled code
        main()

Type errors prevent compilation
-------------------------------

You can't compile code that generates mypy type check errors. You can
sometimes ignore these with a ``# type: ignore`` comment, but this can
result in bad code being generated, and it's considered dangerous.

.. note::

    In the future, mypyc may reject ``# type: ignore`` comments that
    may be unsafe.

Runtime type checking
---------------------

Non-erased types in annotations will be type checked at runtime. For example,
consider this function::

    def twice(x: int) -> int:
        return x * 2

If you try to call this function with a ``float`` or ``str`` argument,
you'll get a type error on the call site, even if the call site is not
being type checked::

    twice(5)  # OK
    twice(2.2)  # TypeError
    twice("blah")  # TypeError

Also, values with *inferred* types will be type checked. For example,
consider a call to the stdlib function ``socket.gethostname()`` in
compiled code. This function is not compiled (no stdlib modules are
compiled with mypyc), but mypyc uses a *library stub file* to infer
the return type as ``str``. Compiled code calling ``gethostname()``
will fail with ``TypeError`` if ``gethostname()`` would return an
incompatible value, such as ``None``::

    import socket

    # Fail if returned value is not a str
    name = socket.gethostname()

Note that ``gethostname()`` is defined like this in the stub file for
``socket`` (in typeshed)::

    def gethostname() -> str: ...

Thus mypyc verifies that library stub files and annotations in
non-compiled code match runtime values. This adds an extra layer of
type safety.

Casts such as ``cast(str, x)`` will also result in strict type
checks. Consider this example::

    from typing import cast
    ...
    x = cast(str, y)

The last line is essentially equivalent to this Python code when compiled::

    if not isinstance(y, str):
        raise TypeError(...)
    x = y

In interpreted mode ``cast`` does not perform a runtime type check.

Native classes
--------------

Native classes behave differently from Python classes.  See
:ref:`native-classes` for the details.

Primitive types
---------------

Some primitive types behave differently in compiled code to improve
performance.

``int`` objects use an unboxed (non-heap-allocated) representation for small
integer values. A side effect of this is that the exact runtime type of
``int`` values is lost. For example, consider this simple function::

    def first_int(x: list[int]) -> int:
        return x[0]

    print(first_int([True]))  # Output is 1, instead of True!

``bool`` is a subclass of ``int``, so the above code is
valid. However, when the list value is converted to ``int``, ``True``
is converted to the corresponding ``int`` value, which is ``1``.

Note that integers still have an arbitrary precision in compiled code,
similar to normal Python integers.

Fixed-length tuples are unboxed, similar to integers. The exact type
and identity of fixed-length tuples is not preserved, and you can't
reliably use ``is`` checks to compare tuples that are used in compiled
code.

.. _early-binding:

Early binding
-------------

References to functions, types, most attributes, and methods in the
same :ref:`compilation unit <compilation-units>` use *early binding*:
the target of the reference is decided at compile time, whenever
possible. This contrasts with normal Python behavior of *late
binding*, where the target is found by a namespace lookup at
runtime. Omitting these namespace lookups improves performance, but
some Python idioms don't work without changes.

Note that non-final module-level variables still use late binding.
You may want to avoid these in very performance-critical code.

Examples of early and late binding::

    from typing import Final

    import lib  # "lib" is not compiled

    x = 0
    y: Final = 1

    def func() -> None:
        pass

    class Cls:
        def __init__(self, attr: int) -> None:
            self.attr = attr

        def method(self) -> None:
            pass

    def example() -> None:
        # Early binding:
        var = y
        func()
        o = Cls()
        o.x
        o.method()

        # Late binding:
        var = x  # Module-level variable
        lib.func()  # Accessing library that is not compiled

Pickling and copying objects
----------------------------

Mypyc tries to enforce that instances native classes are properly
initialized by calling ``__init__`` implicitly when constructing
objects, even if objects are constructed through ``pickle``,
``copy.copy`` or ``copy.deepcopy``, for example.

If a native class doesn't support calling ``__init__`` without arguments,
you can't pickle or copy instances of the class. Use the
``mypy_extensions.mypyc_attr`` class decorator to override this behavior
and enable pickling through the ``serializable`` flag::

    from mypy_extensions import mypyc_attr
    import pickle

    @mypyc_attr(serializable=True)
    class Cls:
        def __init__(self, n: int) -> None:
            self.n = n

    data = pickle.dumps(Cls(5))
    obj = pickle.loads(data)  # OK

Additional notes:

* All subclasses inherit the ``serializable`` flag.
* If a class has the ``allow_interpreted_subclasses`` attribute, it
  implicitly supports serialization.
* Enabling serialization may slow down attribute access, since compiled
  code has to be always prepared to raise ``AttributeError`` in case an
  attribute is not defined at runtime.
* If you try to pickle an object without setting the ``serializable``
  flag, you'll get a ``TypeError`` about missing arguments to
  ``__init__``.


Monkey patching
---------------

Since mypyc function and class definitions are immutable, you can't
perform arbitrary monkey patching, such as replacing functions or
methods with mocks in tests.

.. note::

    Each compiled module has a Python namespace that is initialized to
    point to compiled functions and type objects. This namespace is a
    regular ``dict`` object, and it *can* be modified. However,
    compiled code generally doesn't use this namespace, so any changes
    will only be visible to non-compiled code.

Stack overflows
---------------

Compiled code currently doesn't check for stack overflows. Your
program may crash in an unrecoverable fashion if you have too many
nested function calls, typically due to out-of-control recursion.

.. note::

   This limitation will be fixed in the future.

Final values
------------

Mypy treats variables and attributes defined as ``Final`` specially.
For instance attributes of native classes, mypyc prevents reassignment
of these attributes.

Mypyc replaces references to a variable or attribute declared ``Final``
with the value of the attribute computed at compile time, when it can
be determined during compilation. Example::

    MAX: Final = 100

    def limit_to_max(x: int) -> int:
        if x > MAX:
            return MAX
        return x

The two references to ``MAX`` don't involve any module namespace lookups,
and are equivalent to this code::

    def limit_to_max(x: int) -> int:
        if x > 100:
            return 100
        return x

When run as interpreted, the first example will execute slower due to
the extra namespace lookups.

For a final class attribute or global variable whose value can't be
determined during compilation, mypyc defines a hidden native variable
that stores the value when the initialization assignment is evaluated.
Redefining the attribute using ``setattr`` or direct namespace access
has no effect for mypyc-compiled code (in the same compilation unit).

Final global variables and class attributes are faster to read in
compiled code, since they don't have to be looked up from a namespace
dictionary. Thus using a ``Final`` object that holds mutable state in an
attribute tends to be faster than a plain global variable in compiled code::

    class Counter:
        def __init__(self) -> None:
            self.value = 0

    x: Final = Counter()
    y = 0

    def inc() -> None:
        # Faster: no Python namespace lookup
        x.value += 1
        # Slower: access through globals() namespace dictionary
        global y
        y += 1

.. _free-threading:

Free threading
--------------

Mypyc supports free threading. However, optimized primitive operations in
compiled code may have different atomicity properties compared to CPython.
Use explicit synchronization if code depends on operations being atomic and
race conditions are possible. This is already the recommended approach for
normal Python code. You can often use :ref:`librt.threading.Lock <librt-threading-lock>`
(or :py:class:`threading.Lock`, which is less efficient than
``librt.threading.Lock``) to fix data races.

Since mypyc 2.3, the vast majority of operations are memory safe even if
there are race conditions (unlike earlier mypyc releases). This includes
list operations and access to native instance attributes (except for
a few less common use cases that will be fixed in future releases).

.. note::

    Operations that aren't safe under race conditions in interpreted CPython
    are not expected to be memory safe in compiled code either.
    Some :ref:`librt <librt>` features are heavily optimized for performance and
    don't guarantee memory safety when there are race conditions
    (notably the :ref:`vec <librt-vecs>` type).

It's always safe to perform read-only operations concurrently. Using
objects with final attributes and tuple objects can help prevent
race conditions without introducing extra overhead from explicit
synchronization operations.

Unsupported features
--------------------

Some Python features are not supported by mypyc (yet). They can't be
used in compiled code, or there are some limitations. You can
partially work around some of these limitations by running your code
in interpreted mode.

Nested classes
**************

Nested classes are not supported.

Conditional functions or classes
********************************

Function and class definitions guarded by an if-statement are not supported.

Dunder methods
**************

Native classes **cannot** use these dunders. If defined, they will not
work as expected.

* ``__index__``
* ``__getattribute__``
* ``__delattr__``

Generator expressions
*********************

Generator expressions are not supported. To make it easier to compile
existing code, they are implicitly replaced with list comprehensions.
*This does not always produce the same behavior.*

To work around this limitation, you can usually use a generator
function instead.  You can sometimes replace the generator expression
with an explicit list comprehension.

Descriptors
***********

Native classes can't contain arbitrary descriptors. Properties, static
methods and class methods are supported.

Introspection
*************

Various methods of introspection may break by using mypyc. Here's an
non-exhaustive list of what won't work:

- Instance ``__annotations__`` is usually not kept
- Frames of compiled functions can't be inspected using ``inspect``
- Compiled methods aren't considered methods by ``inspect.ismethod``
- ``inspect.signature`` chokes on compiled functions with default arguments that
  are not simple literals

Profiling hooks and tracing
***************************

Compiled functions don't trigger profiling and tracing hooks, such as
when using the ``profile``, ``cProfile``, or ``trace`` modules.

Debuggers
*********

You can't set breakpoints in compiled functions or step through
compiled functions using ``pdb``. Often you can debug your code in
interpreted mode instead.