[mypyc] Enable incremental self-compilation (#21369)
Six fixes on top of #21299, all required to self-compile mypy or to
install a `separate=True` wheel via pip.
- `mypyc/build.py`: pip invokes `setup.py` twice when building a wheel.
On the second invocation mypy's incremental cache is fully warm, so we
generate no new C source for any group; the resulting extensions ship
without their entry points and import as stubs.
- **Fix**: when a group emits no C source, reuse the .c file from the
previous pass.
- `mypyc/codegen/{emit,emitfunc}.py`: when code in one compiled group
reads an attribute on an object whose class lives in another group, the
generated cast depends on that other group's struct definitions. We
weren't recording the dependency, so the C compiler couldn't see the
layout and the build failed.
- **Fix**: register the dependency at the cast site.
- `mypyc/codegen/emitmodule.py`: when mypy compiles itself, a generated
shim file can share a basename with a runtime C file. The C compiler
resolves the runtime include relative to the shim's directory and picks
up the shim instead.
- **Fix**: search the include path explicitly so shims can't shadow
runtime files.
- `mypyc/lib-rt/misc_ops.c`: each compiled module gets its own shared
library next to it in the package tree. The runtime was computing the
module's file path as if a single shared library sat above the whole
package, which doubled the package prefix and broke submodule lookups.
- **Fix**: detect the per-module case and use only the module's leaf
name.
- `mypyc/irbuild/prepare.py`: traits and builtin-derived classes don't
get a real C constructor emitted. A clean build sidesteps that, but a
fully cached rebuild was taking the direct-call path and producing C
that referenced a constructor that doesn't exist.
- **Fix**: skip the registration the same way a clean build does.
- `mypyc/build.py`: on every build_ext, setuptools rewrites every
compiled .so in the source tree even when nothing changed. On macOS this
invalidates the OS signature cache, so every import on the next run pays
a re-verification cost.
- **Fix**: skip the copy when source and destination already match,
taking a 1-line edit rebuild from ~72s to ~6s. This is really a `setup`
tools limitation though (relevant [mypy
issue](https://github.com/mypyc/mypyc/issues/1068) ?)
I also added a `MYPYC_SEPARATE` env knob so CI can exercise the codegen
path against mypy itself.
## Benchmarks
Mypy self-compile on macOS, `MYPYC_OPT_LEVEL=0`, `-j 11`. Three
scenarios:
| | monolithic | separate=True |
|:---:|:---:|:---:|
| Clean build | 180s | 108s |
| No-op rebuild | 124s | 5s |
| 1-line edit | 106s | 6s |
We are always happy to answer questions! Here are some good places to ask them:
If you're just getting started, the documentation and type hints cheat sheet can also help answer questions.
If you think you've found a bug:
To report a bug or request an enhancement:
To discuss a new type system feature:
Mypy is a static type checker for Python.
Type checkers help ensure that you're using variables and functions in your code correctly. With mypy, add type hints (PEP 484) to your Python programs, and mypy will warn you when you use those types incorrectly.
Python is a dynamic language, so usually you'll only see errors in your code when you attempt to run it. Mypy is a static checker, so it finds bugs in your programs without even running them!
Here is a small example to whet your appetite:
number = input("What is your favourite number?") print("It is", number + 1) # error: Unsupported operand types for + ("str" and "int")
Adding type hints for mypy does not interfere with the way your program would otherwise run. Think of type hints as similar to comments! You can always use the Python interpreter to run your code, even if mypy reports errors.
Mypy is designed with gradual typing in mind. This means you can add type hints to your code base slowly and that you can always fall back to dynamic typing when static typing is not convenient.
Mypy has a powerful and easy-to-use type system, supporting features such as type inference, generics, callable types, tuple types, union types, structural subtyping and more. Using mypy will make your programs easier to understand, debug, and maintain.
See the documentation for more examples and information.
In particular, see:
Mypy can be installed using pip:
python3 -m pip install -U mypy
If you want to run the latest version of the code, you can install from the repo directly:
python3 -m pip install -U git+https://github.com/python/mypy.git
Now you can type-check the statically typed parts of a program like this:
mypy PROGRAM
You can always use the Python interpreter to run your statically typed programs, even if mypy reports type errors:
python3 PROGRAM
If you are working with large code bases, you can run mypy in daemon mode, that will give much faster (often sub-second) incremental updates:
dmypy run -- PROGRAM
You can also try mypy in an online playground (developed by Yusuke Miyazaki).
Mypy can be integrated into popular IDEs:
Additional information is available at the web site:
Jump straight to the documentation:
Follow along our changelog at:
https://mypy-lang.blogspot.com/
Help in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.
To get started with developing mypy, see CONTRIBUTING.md.
Mypyc uses Python type hints to compile Python modules to faster C extensions. Mypy is itself compiled using mypyc: this makes mypy approximately 4 times faster than if interpreted!
To install an interpreted mypy instead, use:
python3 -m pip install --no-binary mypy -U mypy
To use a compiled version of a development version of mypy, directly install a binary from https://github.com/mypyc/mypy_mypyc-wheels/releases/latest.
To contribute to the mypyc project, check out the issue tracker at https://github.com/mypyc/mypyc