[mypyc] Fix attribute offsets when only a subclass defines __call__ (#22483)
The object struct of a native class that has a `__call__` method has a
`vectorcall` field (used through `tp_vectorcall_offset`). It was placed
right after `vtable`, before the attributes. A class inherits `__call__`
from its bases, so if only a subclass defines it, the subclass struct
has the extra field while the base class struct doesn't. All attributes
inherited from the base class were then at different offsets in the
subclass, and code compiled for the base class (methods, attribute
access through the base type) read and wrote the wrong fields. This is a
segfault or memory corruption:
```python
class A:
def __init__(self, x: str) -> None:
self.x = x
def get_x(self) -> str:
return self.x
class B(A):
def __call__(self) -> str:
return self.x
B("b")() # Segfault
```
Generated struct of `B`:
```c
typedef struct {
PyObject_HEAD
CPyVTableItem *vtable;
vectorcallfunc vectorcall; // not in the struct of A
PyObject *_x; // at a different offset than in A
} B;
```
`A.__init__` (compiled for the struct of `A`) stores `x` in the
`vectorcall` field of the `B` instance, so calling it jumps to the
address of a `str` object. With more attributes, base class methods and
subclass code also see each other's attributes in the wrong fields.
Now the `vectorcall` field goes after the attributes, so the attributes
of a base class are at the same offsets in all subclasses. The field is
only accessed through the struct of the class itself (in its setup
function, which isn't shared with native subclasses) and through
`tp_vectorcall_offset`, which is set for each class, so it's fine for
its offset to differ between a class and its subclasses. This also works
with separate compilation, since each module computes the layout from
the class IR in the same way.
Classes where the base class also defines `__call__` were not affected,
and neither were classes without attributes.
Tests: a run test with `__call__` defined only in a subclass, accessing
inherited attributes through base class methods, directly and through
`Any`; `__call__` overridden in a sub-subclass; an intermediate class
with attributes; a base class with weak reference support; and
interpreted subclasses (one inheriting `__call__`, one defining it). A
multi-module test has the base class in another module (another group in
separate compilation mode). The new run test fails on master.
We are always happy to answer questions! Here are some good places to ask them:
If you're just getting started, the documentation and type hints cheat sheet can also help answer questions.
If you think you've found a bug:
To report a bug or request an enhancement:
To discuss a new type system feature:
Mypy is a static type checker for Python.
Type checkers help ensure that you're using variables and functions in your code correctly. With mypy, add type hints (PEP 484) to your Python programs, and mypy will warn you when you use those types incorrectly.
Python is a dynamic language, so usually you'll only see errors in your code when you attempt to run it. Mypy is a static checker, so it finds bugs in your programs without even running them!
Here is a small example to whet your appetite:
number = input("What is your favourite number?") print("It is", number + 1) # error: Unsupported operand types for + ("str" and "int")
Adding type hints for mypy does not interfere with the way your program would otherwise run. Think of type hints as similar to comments! You can always use the Python interpreter to run your code, even if mypy reports errors.
Mypy is designed with gradual typing in mind. This means you can add type hints to your code base slowly and that you can always fall back to dynamic typing when static typing is not convenient.
Mypy has a powerful and easy-to-use type system, supporting features such as type inference, generics, callable types, tuple types, union types, structural subtyping and more. Using mypy will make your programs easier to understand, debug, and maintain.
See the documentation for more examples and information.
In particular, see:
Mypy can be installed using pip:
python3 -m pip install -U mypy
If you want to run the latest version of the code, you can install from the repo directly:
python3 -m pip install -U git+https://github.com/python/mypy.git
Now you can type-check the statically typed parts of a program like this:
mypy PROGRAM
You can always use the Python interpreter to run your statically typed programs, even if mypy reports type errors:
python3 PROGRAM
If you are working with large code bases, you can run mypy in daemon mode, that will give much faster (often sub-second) incremental updates:
dmypy run -- PROGRAM
You can also try mypy in an online playground (developed by Yusuke Miyazaki).
Mypy can be integrated into popular IDEs:
Additional information is available at the web site:
Jump straight to the documentation:
Follow along our changelog at:
https://mypy-lang.blogspot.com/
Help in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.
To get started with developing mypy, see CONTRIBUTING.md.
Mypyc uses Python type hints to compile Python modules to faster C extensions. Mypy is itself compiled using mypyc: this makes mypy approximately 4 times faster than if interpreted!
To install an interpreted mypy instead, use:
python3 -m pip install --no-binary mypy -U mypy
To use a compiled version of a development version of mypy, directly install a binary from https://github.com/mypyc/mypy_mypyc-wheels/releases/latest.
To contribute to the mypyc project, check out the issue tracker at https://github.com/mypyc/mypyc