[mypyc] Fix crashes and compile errors from the result of native __len__ methods (#22485)
The `sq_length`/`mp_length` slot of a native class didn't check the
value returned by `__len__` like CPython's `slot_sq_length` does, and
the compiled `len()` of a native class instance only partly did:
- A negative result from `__len__` declared to return `int` or `Any` was
returned from the slot as is, without an exception set. `len(x)` then
raised `SystemError: <built-in function len> returned NULL without
setting an exception` when called from interpreted code, and crashed the
interpreter (segfault) in some other cases, such as `len(x)` in compiled
code where `x` has type `Any`.
- `__len__` declared to return `bool` or a native int type (`i64`,
`i32`, `i16`, `i8`, `u8`, ...) generated `PyLong_AsSsize_t(retval)` with
a C integer `retval`, which is a C compile error (`-Wint-conversion`)
with current GCC and clang.
- With `-> Any`, a result that isn't an int raised `TypeError: an
integer is required`, while Python accepts any object with `__index__`.
- The compiled `len(x)`, where `x` is a native class instance, checked
that the length isn't negative, but returned a value that doesn't fit in
`Py_ssize_t` (such as `2**63`) instead of raising `OverflowError`.
```python
class C:
def __init__(self, n: int) -> None:
self.n = n
def __len__(self) -> int:
return self.n
class B:
def __len__(self) -> bool: # C compile error
return True
```
```python
>>> len(C(-1))
SystemError: <built-in function len> returned NULL without setting an exception
```
CPython's `slot_sq_length` converts the result with `__index__`, raises
`ValueError: __len__() should return >= 0` if it's negative, and
`OverflowError: cannot fit 'int' into an index-sized integer` if it
doesn't fit in `Py_ssize_t`.
Now:
- `generate_len_wrapper()` handles a short int inline. Anything else
goes through the new `CPy_LenFromResult()` in `generic_ops.c`, which
does what `slot_sq_length` does, after boxing the value if needed (like
`generate_hash_wrapper()` after #22453).
- In the compiled `len()` of a native class instance, the length check
is the new `CPyTagged_CheckLen()` in `CPy.h`. It's inline for short
ints, and for long ints it also raises `OverflowError` if the value
doesn't fit in `Py_ssize_t`. If `__len__` returns something other than
an int type (for example `Any`), the result is converted by
`CPyObject_LenFromResult()` like in the slot.
Tests: the new run test `testDundersLenResult` covers `-> int`, `->
Any`, `-> bool`, `-> i64`, `-> i8` and `-> u8`, through both the
compiled `len()` and the slot (`len()` of an `Any` value), with negative
values, values that don't fit in `Py_ssize_t`, `True`, an object with
`__index__` and results that aren't ints. I updated `testDundersLen` in
`irbuild-dunders.test` and added `testDundersLenAny`.
We are always happy to answer questions! Here are some good places to ask them:
If you're just getting started, the documentation and type hints cheat sheet can also help answer questions.
If you think you've found a bug:
To report a bug or request an enhancement:
To discuss a new type system feature:
Mypy is a static type checker for Python.
Type checkers help ensure that you're using variables and functions in your code correctly. With mypy, add type hints (PEP 484) to your Python programs, and mypy will warn you when you use those types incorrectly.
Python is a dynamic language, so usually you'll only see errors in your code when you attempt to run it. Mypy is a static checker, so it finds bugs in your programs without even running them!
Here is a small example to whet your appetite:
number = input("What is your favourite number?") print("It is", number + 1) # error: Unsupported operand types for + ("str" and "int")
Adding type hints for mypy does not interfere with the way your program would otherwise run. Think of type hints as similar to comments! You can always use the Python interpreter to run your code, even if mypy reports errors.
Mypy is designed with gradual typing in mind. This means you can add type hints to your code base slowly and that you can always fall back to dynamic typing when static typing is not convenient.
Mypy has a powerful and easy-to-use type system, supporting features such as type inference, generics, callable types, tuple types, union types, structural subtyping and more. Using mypy will make your programs easier to understand, debug, and maintain.
See the documentation for more examples and information.
In particular, see:
Mypy can be installed using pip:
python3 -m pip install -U mypy
If you want to run the latest version of the code, you can install from the repo directly:
python3 -m pip install -U git+https://github.com/python/mypy.git
Now you can type-check the statically typed parts of a program like this:
mypy PROGRAM
You can always use the Python interpreter to run your statically typed programs, even if mypy reports type errors:
python3 PROGRAM
If you are working with large code bases, you can run mypy in daemon mode, that will give much faster (often sub-second) incremental updates:
dmypy run -- PROGRAM
You can also try mypy in an online playground (developed by Yusuke Miyazaki).
Mypy can be integrated into popular IDEs:
Additional information is available at the web site:
Jump straight to the documentation:
Follow along our changelog at:
https://mypy-lang.blogspot.com/
Help in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.
To get started with developing mypy, see CONTRIBUTING.md.
Mypyc uses Python type hints to compile Python modules to faster C extensions. Mypy is itself compiled using mypyc: this makes mypy approximately 4 times faster than if interpreted!
To install an interpreted mypy instead, use:
python3 -m pip install --no-binary mypy -U mypy
To use a compiled version of a development version of mypy, directly install a binary from https://github.com/mypyc/mypy_mypyc-wheels/releases/latest.
To contribute to the mypyc project, check out the issue tracker at https://github.com/mypyc/mypyc