Run conformance tests in GitHub CI (#4579)

Summary:

The typing conformance suite only runs inside Meta, via the `conformance_output_test` Buck target. On GitHub nothing invokes it, so a PR that changes pyrefly's conformance output goes green while leaving the checked-in `conformance.exp` stale. The mismatch only surfaces later, when the change is imported internally and fails there.

Add a step to the `pyrefly` workflow that runs `conformance_output.py --mode check` against the release binary that job already built, matching what internal CI checks. The step is restricted to Linux: the generated output does not vary by platform, so one OS suffices, and the check cannot run on Windows because it keys results by splitting paths on `/`.

The check-mode failure message told the reader to re-generate the output with `buck2`, which a GitHub contributor has no way to run. It now branches on whether `--executable` was passed, since that flag is precisely the signal that the caller is outside Buck, and prints the corresponding cargo-based command.

Reviewed By: ndmitchell, rchen152

Differential Revision: D116394357


2 files changed
tree: ec6ce63742dff1812b2cf291a77f97228d911642
  1. .cargo/
  2. .claude/
  3. .github/
  4. .llms/
  5. .vscode/
  6. conformance/
  7. crates/
  8. lsp/
  9. pyrefly/
  10. pyrefly_wasm/
  11. release_notes/
  12. schemas/
  13. scripts/
  14. tensor-shapes/
  15. test/
  16. website/
  17. .gitattributes
  18. .gitignore
  19. action.yml
  20. AGENTS.md
  21. ARCHITECTURE.md
  22. Cargo.lock
  23. Cargo.toml
  24. CODE_OF_CONDUCT.md
  25. CONTRIBUTING.md
  26. debug.html
  27. empty.py
  28. LICENSE
  29. package-lock.json
  30. README.md
  31. REVIEW.md
  32. rust-toolchain.toml
  33. rustfmt.toml
  34. TENSOR_SHAPES_CONTRIBUTING.md
  35. test.py
  36. version.bzl
  37. yarn.lock
README.md

Pyrefly: A fast type checker and language server for Python with powerful IDE features

pyrefly PyPI VS Code Open VSX Discord License: MIT

Pyrefly is a type checker and language server for Python, which provides lightning-fast type checking along with IDE features such as code navigation, semantic highlighting, and code completion. It is available as a command-line tool and an extension for popular IDEs and editors such as VSCode, Neovim, Zed, and more.

See the Pyrefly website for full documentation and how to add Pyrefly to your editor of choice.

Pyrefly's current development status is stable.

Key Features

  • Fast. Pyrefly checks over 1.85 million lines of code per second, type checking projects like PyTorch 15x faster than Mypy and Pyright. In the IDE, rechecks typically complete in under 10 milliseconds after saving a file.
  • Production-proven at scale. Pyrefly is the default type checker for Instagram's 20-million-line Python codebase at Meta, and has been adopted by large open source projects including PyTorch and JAX.
  • Full-featured language server. Code navigation, autocomplete, hover information, inlay hints, semantic highlighting, and more, with consistent results across the CLI and your editor of choice.
  • Understands real-world Python. Built-in support for frameworks and tools like Pydantic, Django, and pytest, with model validation, field types, fixture navigation, and autocomplete that work out of the box.
  • Adoption-ready. Migrate from Mypy or Pyright with pyrefly init, silence existing errors with pyrefly suppress, and generate type annotations with pyrefly infer. Start with one file and expand at your own pace.

Getting Started

Version Policy

Pyrefly releases new minor versions (1.x.0) monthly and patch versions in between as-needed for critical fixes. Pyrefly does not follow strict semantic versioning: minor versions contain more significant changes than patch versions, but any version may introduce new type errors and other breaking changes. The pyrefly suppress command can be used to easily silence errors when upgrading to a new version.

Getting Involved

If you have questions or would like to report a bug, please create an issue.

See our contributing guide and architecture overview for information on how to contribute to Pyrefly.

Join our Discord to chat about Pyrefly and types. This is also where we hold biweekly office hours.