Merge branch 'main' of https://github.com/bazel-contrib/rules_python into py-extension
diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d06a68..469e9d3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md
@@ -85,6 +85,8 @@ {#v0-0-0-added} ### Added +* (runfiles) The Python runfiles library now supports Bazel's + `--incompatible_compact_repo_mapping_manifest` flag. * (bootstrap) {obj}`--bootstrap_impl=system_python` now supports the {obj}`main_module` attribute. * (bootstrap) {obj}`--bootstrap_impl=system_python` now supports the
diff --git a/docs/howto/multi-platform-pypi-deps.md b/docs/howto/multi-platform-pypi-deps.md new file mode 100644 index 0000000..6cc7f84 --- /dev/null +++ b/docs/howto/multi-platform-pypi-deps.md
@@ -0,0 +1,194 @@ +:::{default-domain} bzl +::: + +# How-to: Multi-Platform PyPI Dependencies + +When developing applications that need to run on a wide variety of platforms, +managing PyPI dependencies can become complex. You might need different sets of +dependencies for different combinations of Python version, threading model, +operating system, CPU architecture, libc, and even hardware accelerators like +GPUs. + +This guide demonstrates how to manage this complexity using `rules_python` with +bzlmod. If you prefer to learn by example, complete example code is provided at +the end. + +In this how to guide, we configure for using 4 requirements files, each +for a different variation using Python 3.14 on Linux: + +* Regular (non-freethreaded) Python +* Freethreaded Python +* Regular Python for CUDA 12.9 +* Freethreaded Python for ARM and Musl + +## Mapping requirements files to Bazel configuration settings + +Unfortunately, a requirements file doesn't tell what it's compatible with, +so we have to manually specify the Bazel configuration settings for it. To do +that using rules_python, there are two steps: defining a platform, then +associating a requirements file with the platform. + +### Defining a platform + +First, we define a "platform" using {obj}`pip.default`. This associates an +arbitrary name with a list of Bazel {obj}`config_setting` targets. While any +name can be used for a platform (its name has no inherent semantic meaning), it +should encode all the relevant dimensions that distinguish a requirements file. +For example, if a requirements file is specifically for the combination of CUDA +12.9 and NumPy 2.0, then the platform name should represent that. + +The convention is to follow the format of `{os}_{cpu}{threading}`, where: + +* `{os}` is the operating system (`linux`, `osx`, `windows`). +* `{cpu}` is the architecture (`x86_64`, `aarch64`). +* `{threading}` is `_freethreaded` for a freethreaded Python runtime, or an + empty string for the regular runtime. + +Additional dimensions should be appended and separated with an underscore (e.g. +`linux_x86_64_musl_cuda12.9_numpy2`). + +The platform name should not include the Python version. That is handled by +`pip.parse.python_version` separately. + +:::{note} +The term _platform_ here has nothing to do with Bazel's `platform()` rule. +::: + +#### Defining custom settings + +Because {obj}`pip.parse.config_settings` is a list of arbitrary `config_setting` +targets, you can define your own flags or implement custom config matching +logic. This allows you to model settings that aren't inherently part of +rules_python. + +This is typically done using [bazel_skylib flags](https://bazel.build/extending/config), but any [Starlark +defined build setting](https://bazel.build/extending/config) can be used. Just +remember to use `config_setting()` to match a particular value of the flag. + +In our example below, we define a custom flag for CUDA version. + +#### Predefined and common build settings + +rules_python has some predefined build settings you can use. Commonly used ones +are: + +* {obj}`@rules_python//python/config_settings:py_linux_libc` +* {obj}`@rules_python//python/config_settings:py_freethreaded` + +Additionally, [Bazel @platforms](https://github.com/bazelbuild/platforms) +contains commonly used settings for OS and CPU: + +* `@platforms//os:windows` +* `@platforms//os:linux` +* `@platforms//os:osx` +* `@platforms//cpu:x86_64` +* `@platforms//cpu:aarch64` + +Note that these are the raw flag names. In order to use them with `pip.default`, +you must use {obj}`config_setting()` to match a particular value for them. + +### Associating Requirements to Platforms + +Next, we associate a requirements file with a platform using +{obj}`pip.parse.requirements_by_platform`. This is a dictionary attribute where +the keys are requirements files and the value is a platform name. The platform +value can use a trailing or leading `*` to match multiple platforms. It can also +specify multiple platform names using commas to separate them. + +Note that the Python version is _not_ part of the platform name. + +Under the hood, `pip.parse` merges all the requirements (for a `hub_name`) and +constructs `select()` expressions to route to the appropriate dependencies. + +### Using it in practice + +Finally, to make use of what we've configured, perform a build and set +command line flags to the appropriate values. + +```shell +# Build for CUDA +bazel build --//:cuda_version=12.9 //:binary + +# Build for ARM with musl +bazel build --@rules_python//python/config_settings:py_linux_libc=musl \ + --cpu=aarch64 //:binary + +# Build for freethreaded +bazel build --@rules_python//python/config_settings:py_freethreaded=yes //:binary +``` + +Note that certain combinations of flags may result in an error or undefined +behavior. For example, trying to set both freethreaded and CUDA at the same +time would result in an error because no requirements file was registered +to match that combination. + +## Multiple Python Versions + +Having multiple Python versions is fully supported. Simply add a `pip.parse()` +call and set `python_version` appropriately. + +## Multiple hubs + +Having multiple `pip.parse` calls with different `hub_name` values is fully +supported. Each hub only contains the requirements registered to it. + +## Complete Example + +Here is a complete example that puts all the pieces together. + +```starlark +# File: BUILD.bazel +load("@bazel_skylib//rules:common_settings.bzl", "string_flag") + +# A custom flag for controlling the CUDA version +string_flag( + name = "cuda_version", + build_setting_default = "none", +) + +config_setting( + name = "is_cuda_12_9", + flag_values = {":cuda_version": "12.9"}, +) + +# A config_setting that uses the built-in libc flag from rules_python +config_setting( + name = "is_musl", + flag_values = {"@rules_python//python/config_settings:py_linux_libc": "muslc"}, +) + +# File: MODULE.bazel +pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip") + +# A custom platform for CUDA on glibc linux +pip.default( + platform = "linux_x86_64_cuda12.9", + os = "linux", + cpu = "x86_64", + config_settings = ["@//:is_cuda_12_9"], +) + +# A custom platform for musl on linux +pip.default( + platform = "linux_aarch64_musl", + os = "linux", + cpu = "aarch64", + config_settings = ["@//:is_musl"], +) + +pip.parse( + hub_name = "my_deps", + python_version = "3.14", + requirements_by_platform = { + # Map to default platform names + "//:py3.14-regular-linux-x86-glibc-cpu.txt": "linux_x86_64", + "//:py3.14-freethreaded-linux-x86-glibc-cpu.txt": "linux_x86_64_freethreaded", + + # Map to our custom platform names + "//:py3.14-regular-linux-x86-glibc-cuda12.9.txt": "linux_x86_64_cuda12.9", + "//:py3.14-freethreaded-linux-arm-musl-cpu.txt": "linux_aarch64_musl", + }, +) + +use_repo(pip, "my_deps") +```
diff --git a/docs/pypi/index.md b/docs/pypi/index.md index c32bafc..1792889 100644 --- a/docs/pypi/index.md +++ b/docs/pypi/index.md
@@ -11,6 +11,7 @@ With the advanced topics covered separately: * Dealing with [circular dependencies](./circular-dependencies). +* Handling [multi-platform dependencies](../howto/multi-platform-pypi-deps). ```{toctree} lock @@ -22,6 +23,9 @@ ## Advanced topics ```{toctree} +:maxdepth: 1 + circular-dependencies patch +../howto/multi-platform-pypi-deps ```
diff --git a/docs/requirements.txt b/docs/requirements.txt index 5be6c59..e4a4365 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt
@@ -111,9 +111,9 @@ --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \ --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6 # via sphinx -docutils==0.21.2 \ - --hash=sha256:3a6b18732edf182daa3cd12775bbb338cf5691468f91eeeb109deff6ebfa986f \ - --hash=sha256:dafca5b9e384f0e419294eb4d2ff9fa826435bf15f15b7bd45723e8ad76811b2 +docutils==0.22.2 \ + --hash=sha256:9fdb771707c8784c8f2728b67cb2c691305933d68137ef95a75db5f4dfbc213d \ + --hash=sha256:b0e98d679283fc3bb0ead8a5da7f501baa632654e7056e9c5846842213d674d8 # via # myst-parser # sphinx
diff --git a/python/config_settings/BUILD.bazel b/python/config_settings/BUILD.bazel index 82a73ce..cc5c472 100644 --- a/python/config_settings/BUILD.bazel +++ b/python/config_settings/BUILD.bazel
@@ -1,4 +1,4 @@ -load("@bazel_skylib//rules:common_settings.bzl", "string_flag") +load("@bazel_skylib//rules:common_settings.bzl", "bool_flag", "string_flag") load("@pythons_hub//:versions.bzl", "DEFAULT_PYTHON_VERSION", "MINOR_MAPPING", "PYTHON_VERSIONS") load( "//python/private:flags.bzl", @@ -240,3 +240,9 @@ # NOTE: Only public because it is used in pip hub repos. visibility = ["//visibility:public"], ) + +bool_flag( + name = "experimental_python_import_all_repositories", + build_setting_default = True, + visibility = ["//visibility:public"], +)
diff --git a/python/private/flags.bzl b/python/private/flags.bzl index 710402b..82ec832 100644 --- a/python/private/flags.bzl +++ b/python/private/flags.bzl
@@ -21,6 +21,59 @@ load("@bazel_skylib//rules:common_settings.bzl", "BuildSettingInfo") load(":enum.bzl", "FlagEnum", "enum") +# Maps "--myflag" to a tuple of: +# +# - the flag's ctx.fragments native API accessor +# -"native|starlark": which definition to use if the flag is available both +# from ctx.fragments and Starlark +# +# Builds that set --incompatible_remove_ctx_py_fragment or +# --incompatible_remove_ctx_bazel_py_fragment disable ctx.fragments. These +# builds assume flags are solely defined in Starlark. +# +# The "native|starlark" override is only for devs who are testing flag +# Starlarkification. If ctx.fragments.[py|bazel_py] is available and +# a flag is set to "starlark", we exclusively read its starlark version. +# +# See https://github.com/bazel-contrib/rules_python/issues/3252. +_POSSIBLY_NATIVE_FLAGS = { + "build_python_zip": (lambda ctx: ctx.fragments.py.build_python_zip, "native"), + "default_to_explicit_init_py": (lambda ctx: ctx.fragments.py.default_to_explicit_init_py, "native"), + "disable_py2": (lambda ctx: ctx.fragments.py.disable_py2, "native"), + "python_import_all_repositories": (lambda ctx: ctx.fragments.bazel_py.python_import_all_repositories, "native"), + "python_path": (lambda ctx: ctx.fragments.bazel_py.python_path, "native"), +} + +def read_possibly_native_flag(ctx, flag_name): + """ + Canonical API for reading a Python build flag. + + Flags might be defined in Starlark or native-Bazel. This function reasd flags + from tbe correct source based on supporting Bazel version and --incompatible* + flags that disable native references. + + Args: + ctx: Rule's configuration context. + flag_name: Name of the flag to read, without preceding "--". + + Returns: + The flag's value. + """ + + # Bazel 9.0+ can disable these fragments with --incompatible_remove_ctx_py_fragment and + # --incompatible_remove_ctx_bazel_py_fragment. Disabling them means bazel expects + # Python to read Starlark flags. + use_native_def = hasattr(ctx.fragments, "py") and hasattr(ctx.fragments, "bazel_py") + + # Developer override to force the Starlark definition for testing. + if _POSSIBLY_NATIVE_FLAGS[flag_name][1] == "starlark": + use_native_def = False + if use_native_def: + return _POSSIBLY_NATIVE_FLAGS[flag_name][0](ctx) + else: + # Starlark definition of "--foo" is assumed to be a label dependency named "_foo". + return getattr(ctx.attr, "_" + flag_name)[BuildSettingInfo].value + def _AddSrcsToRunfilesFlag_is_enabled(ctx): value = ctx.attr._add_srcs_to_runfiles_flag[BuildSettingInfo].value if value == AddSrcsToRunfilesFlag.AUTO:
diff --git a/python/private/py_executable.bzl b/python/private/py_executable.bzl index 5993a4f..dd0a1a1 100644 --- a/python/private/py_executable.bzl +++ b/python/private/py_executable.bzl
@@ -53,7 +53,7 @@ "target_platform_has_any_constraint", ) load(":common_labels.bzl", "labels") -load(":flags.bzl", "BootstrapImplFlag", "VenvsUseDeclareSymlinkFlag") +load(":flags.bzl", "BootstrapImplFlag", "VenvsUseDeclareSymlinkFlag", "read_possibly_native_flag") load(":precompile.bzl", "maybe_precompile") load(":py_cc_link_params_info.bzl", "PyCcLinkParamsInfo") load(":py_executable_info.bzl", "PyExecutableInfo") @@ -293,7 +293,7 @@ def _should_create_init_files(ctx): if ctx.attr.legacy_create_init == -1: - return not ctx.fragments.py.default_to_explicit_init_py + return not read_possibly_native_flag(ctx, "default_to_explicit_init_py") else: return bool(ctx.attr.legacy_create_init) @@ -381,7 +381,7 @@ extra_files_to_build = [] # NOTE: --build_python_zip defaults to true on Windows - build_zip_enabled = ctx.fragments.py.build_python_zip + build_zip_enabled = read_possibly_native_flag(ctx, "build_python_zip") # When --build_python_zip is enabled, then the zip file becomes # one of the default outputs. @@ -587,7 +587,7 @@ output = site_init, substitutions = { "%coverage_tool%": _get_coverage_tool_runfiles_path(ctx, runtime), - "%import_all%": "True" if ctx.fragments.bazel_py.python_import_all_repositories else "False", + "%import_all%": "True" if read_possibly_native_flag(ctx, "python_import_all_repositories") else "False", "%site_init_runfiles_path%": "{}/{}".format(ctx.workspace_name, site_init.short_path), "%workspace_name%": ctx.workspace_name, }, @@ -668,7 +668,7 @@ output = output, substitutions = { "%coverage_tool%": _get_coverage_tool_runfiles_path(ctx, runtime), - "%import_all%": "True" if ctx.fragments.bazel_py.python_import_all_repositories else "False", + "%import_all%": "True" if read_possibly_native_flag(ctx, "python_import_all_repositories") else "False", "%imports%": ":".join(imports.to_list()), "%main%": main_py_path, "%main_module%": ctx.attr.main_module, @@ -755,7 +755,7 @@ template = ctx.file._bootstrap_template subs["%coverage_tool%"] = coverage_tool_runfiles_path - subs["%import_all%"] = ("True" if ctx.fragments.bazel_py.python_import_all_repositories else "False") + subs["%import_all%"] = ("True" if read_possibly_native_flag(ctx, "python_import_all_repositories") else "False") subs["%imports%"] = ":".join(imports.to_list()) subs["%main%"] = "{}/{}".format(ctx.workspace_name, main_py.short_path) @@ -1135,7 +1135,7 @@ # # TOOD(bazelbuild/bazel#7901): Remove this once --python_path flag is removed. - flag_interpreter_path = ctx.fragments.bazel_py.python_path + flag_interpreter_path = read_possibly_native_flag(ctx, "python_path") toolchain_runtime, effective_runtime = _maybe_get_runtime_from_ctx(ctx) if not effective_runtime: # Clear these just in case
diff --git a/python/private/py_runtime_pair_rule.bzl b/python/private/py_runtime_pair_rule.bzl index 775d53a..61cbdcd 100644 --- a/python/private/py_runtime_pair_rule.bzl +++ b/python/private/py_runtime_pair_rule.bzl
@@ -17,6 +17,7 @@ load("@bazel_skylib//rules:common_settings.bzl", "BuildSettingInfo") load("//python:py_runtime_info.bzl", "PyRuntimeInfo") load(":common_labels.bzl", "labels") +load(":flags.bzl", "read_possibly_native_flag") load(":reexports.bzl", "BuiltinPyRuntimeInfo") load(":util.bzl", "IS_BAZEL_7_OR_HIGHER") @@ -69,7 +70,7 @@ # TODO: Remove this once all supported Balze versions have this flag. if not hasattr(ctx.fragments.py, "disable_py"): return False - return ctx.fragments.py.disable_py2 + return read_possibly_native_flag(ctx, "disable_py2") _MaybeBuiltinPyRuntimeInfo = [[BuiltinPyRuntimeInfo]] if BuiltinPyRuntimeInfo != None else []
diff --git a/python/private/pypi/pip_repository.bzl b/python/private/pypi/pip_repository.bzl index 6d539a5..2cf20cd 100644 --- a/python/private/pypi/pip_repository.bzl +++ b/python/private/pypi/pip_repository.bzl
@@ -266,6 +266,9 @@ Those dependencies become available in a generated `requirements.bzl` file. You can instead check this `requirements.bzl` file into your repo, see the "vendoring" section below. +For advanced use-cases, such as handling multi-platform dependencies, see the +[How-to: Multi-Platform PyPI Dependencies guide](/howto/multi-platform-pypi-deps). + In your WORKSPACE file: ```starlark
diff --git a/python/runfiles/runfiles.py b/python/runfiles/runfiles.py index 3943be5..58f59c5 100644 --- a/python/runfiles/runfiles.py +++ b/python/runfiles/runfiles.py
@@ -15,14 +15,131 @@ """Runfiles lookup library for Bazel-built Python binaries and tests. See @rules_python//python/runfiles/README.md for usage instructions. + +:::{versionadded} VERSION_NEXT_FEATURE +Support for Bazel's `--incompatible_compact_repo_mapping_manifest` flag was added. +This enables prefix-based repository mappings to reduce memory usage for large +dependency graphs under bzlmod. +::: """ +import collections.abc import inspect import os import posixpath import sys +from collections import defaultdict from typing import Dict, Optional, Tuple, Union +class _RepositoryMapping: + """Repository mapping for resolving apparent repository names to canonical ones. + + Handles both exact mappings and prefix-based mappings introduced by the + --incompatible_compact_repo_mapping_manifest flag. + """ + + def __init__( + self, + exact_mappings: Dict[Tuple[str, str], str], + prefixed_mappings: Dict[Tuple[str, str], str], + ) -> None: + """Initialize repository mapping with exact and prefixed mappings. + + Args: + exact_mappings: Dict mapping (source_canonical, target_apparent) -> target_canonical + prefixed_mappings: Dict mapping (source_prefix, target_apparent) -> target_canonical + """ + self._exact_mappings = exact_mappings + + # Group prefixed mappings by target_apparent for faster lookups + self._grouped_prefixed_mappings = defaultdict(list) + for ( + prefix_source, + target_app, + ), target_canonical in prefixed_mappings.items(): + self._grouped_prefixed_mappings[target_app].append( + (prefix_source, target_canonical) + ) + + @staticmethod + def create_from_file(repo_mapping_path: Optional[str]) -> "_RepositoryMapping": + """Create RepositoryMapping from a repository mapping manifest file. + + Args: + repo_mapping_path: Path to the repository mapping file, or None if not available + + Returns: + RepositoryMapping instance with parsed mappings + """ + # If the repository mapping file can't be found, that is not an error: We + # might be running without Bzlmod enabled or there may not be any runfiles. + # In this case, just apply empty repo mappings. + if not repo_mapping_path: + return _RepositoryMapping({}, {}) + + try: + with open(repo_mapping_path, "r", encoding="utf-8", newline="\n") as f: + content = f.read() + except FileNotFoundError: + return _RepositoryMapping({}, {}) + + exact_mappings = {} + prefixed_mappings = {} + for line in content.splitlines(): + source_canonical, target_apparent, target_canonical = line.split(",") + if source_canonical.endswith("*"): + # This is a prefixed mapping - remove the '*' for prefix matching + prefix = source_canonical[:-1] + prefixed_mappings[(prefix, target_apparent)] = target_canonical + else: + # This is an exact mapping + exact_mappings[(source_canonical, target_apparent)] = target_canonical + + return _RepositoryMapping(exact_mappings, prefixed_mappings) + + def lookup(self, source_repo: Optional[str], target_apparent: str) -> Optional[str]: + """Look up repository mapping for the given source and target. + + This handles both exact mappings and prefix-based mappings introduced by the + --incompatible_compact_repo_mapping_manifest flag. Exact mappings are tried + first, followed by prefix-based mappings where order matters. + + Args: + source_repo: Source canonical repository name + target_apparent: Target apparent repository name + + Returns: + target_canonical repository name, or None if no mapping exists + """ + if source_repo is None: + return None + + key = (source_repo, target_apparent) + + # Try exact mapping first + if key in self._exact_mappings: + return self._exact_mappings[key] + + # Try prefixed mapping if no exact match found + if target_apparent in self._grouped_prefixed_mappings: + for prefix_source, target_canonical in self._grouped_prefixed_mappings[ + target_apparent + ]: + if source_repo.startswith(prefix_source): + return target_canonical + + # No mapping found + return None + + def is_empty(self) -> bool: + """Check if this repository mapping is empty (no exact or prefixed mappings). + + Returns: + True if there are no mappings, False otherwise + """ + return len(self._exact_mappings) == 0 and len(self._grouped_prefixed_mappings) == 0 + + class _ManifestBased: """`Runfiles` strategy that parses a runfiles-manifest to look up runfiles.""" @@ -130,7 +247,7 @@ def __init__(self, strategy: Union[_ManifestBased, _DirectoryBased]) -> None: self._strategy = strategy self._python_runfiles_root = _FindPythonRunfilesRoot() - self._repo_mapping = _ParseRepoMapping( + self._repo_mapping = _RepositoryMapping.create_from_file( strategy.RlocationChecked("_repo_mapping") ) @@ -179,7 +296,7 @@ if os.path.isabs(path): return path - if source_repo is None and self._repo_mapping: + if source_repo is None and not self._repo_mapping.is_empty(): # Look up runfiles using the repository mapping of the caller of the # current method. If the repo mapping is empty, determining this # name is not necessary. @@ -188,7 +305,8 @@ # Split off the first path component, which contains the repository # name (apparent or canonical). target_repo, _, remainder = path.partition("/") - if not remainder or (source_repo, target_repo) not in self._repo_mapping: + target_canonical = self._repo_mapping.lookup(source_repo, target_repo) + if not remainder or target_canonical is None: # One of the following is the case: # - not using Bzlmod, so the repository mapping is empty and # apparent and canonical repository names are the same @@ -202,11 +320,15 @@ source_repo is not None ), "BUG: if the `source_repo` is None, we should never go past the `if` statement above" - # target_repo is an apparent repository name. Look up the corresponding - # canonical repository name with respect to the current repository, - # identified by its canonical name. - target_canonical = self._repo_mapping[(source_repo, target_repo)] - return self._strategy.RlocationChecked(target_canonical + "/" + remainder) + # Look up the target repository using the repository mapping + if target_canonical is not None: + return self._strategy.RlocationChecked( + target_canonical + "/" + remainder + ) + + # No mapping found - assume target_repo is already canonical or + # we're not using Bzlmod + return self._strategy.RlocationChecked(path) def EnvVars(self) -> Dict[str, str]: """Returns environment variables for subprocesses. @@ -359,30 +481,6 @@ return root -def _ParseRepoMapping(repo_mapping_path: Optional[str]) -> Dict[Tuple[str, str], str]: - """Parses the repository mapping manifest.""" - # If the repository mapping file can't be found, that is not an error: We - # might be running without Bzlmod enabled or there may not be any runfiles. - # In this case, just apply an empty repo mapping. - if not repo_mapping_path: - return {} - try: - with open(repo_mapping_path, "r", encoding="utf-8", newline="\n") as f: - content = f.read() - except FileNotFoundError: - return {} - - repo_mapping = {} - for line in content.split("\n"): - if not line: - # Empty line following the last line break - break - current_canonical, target_local, target_canonical = line.split(",") - repo_mapping[(current_canonical, target_local)] = target_canonical - - return repo_mapping - - def CreateManifestBased(manifest_path: str) -> Runfiles: return Runfiles.CreateManifestBased(manifest_path)
diff --git a/tests/runfiles/runfiles_test.py b/tests/runfiles/runfiles_test.py index a3837ac..b8a3d5f 100644 --- a/tests/runfiles/runfiles_test.py +++ b/tests/runfiles/runfiles_test.py
@@ -18,6 +18,7 @@ from typing import Any, List, Optional from python.runfiles import runfiles +from python.runfiles.runfiles import _RepositoryMapping class RunfilesTest(unittest.TestCase): @@ -525,6 +526,165 @@ r.Rlocation("config.json", "protobuf~3.19.2"), dir + "/config.json" ) + def testDirectoryBasedRlocationWithCompactRepoMappingFromMain(self) -> None: + """Test repository mapping with prefix-based entries (compact format).""" + with _MockFile( + name="_repo_mapping", + contents=[ + # Exact mappings (no asterisk) + "_,config.json,config.json~1.2.3", + ",my_module,_main", + ",my_workspace,_main", + # Prefixed mappings (with asterisk) - these apply to any repo starting with the prefix + "deps+*,external_dep,external_dep~1.0.0", + "test_deps+*,test_lib,test_lib~2.1.0", + ], + ) as rm: + dir = os.path.dirname(rm.Path()) + r = runfiles.CreateDirectoryBased(dir) + + # Test exact mappings still work + self.assertEqual( + r.Rlocation("my_module/bar/runfile", ""), dir + "/_main/bar/runfile" + ) + self.assertEqual( + r.Rlocation("my_workspace/bar/runfile", ""), dir + "/_main/bar/runfile" + ) + + # Test prefixed mappings - should match any repo starting with "deps+" + self.assertEqual( + r.Rlocation("external_dep/foo/file", "deps+dep1"), + dir + "/external_dep~1.0.0/foo/file", + ) + self.assertEqual( + r.Rlocation("external_dep/bar/file", "deps+dep2"), + dir + "/external_dep~1.0.0/bar/file", + ) + self.assertEqual( + r.Rlocation("external_dep/nested/path/file", "deps+some_long_dep_name"), + dir + "/external_dep~1.0.0/nested/path/file", + ) + + # Test that prefixed mappings work for test_deps+ prefix too + self.assertEqual( + r.Rlocation("test_lib/test/file", "test_deps+junit"), + dir + "/test_lib~2.1.0/test/file", + ) + + # Test that non-matching prefixes don't match + self.assertEqual( + r.Rlocation("external_dep/foo/file", "other_prefix"), + dir + "/external_dep/foo/file", # No mapping applied, use as-is + ) + + def testDirectoryBasedRlocationWithCompactRepoMappingPrecedence(self) -> None: + """Test that exact mappings take precedence over prefixed mappings.""" + with _MockFile( + name="_repo_mapping", + contents=[ + # Exact mapping for a specific source repo + "deps+specific_repo,external_dep,external_dep~exact", + # Prefixed mapping for repos starting with "deps+" + "deps+*,external_dep,external_dep~prefix", + # Another prefixed mapping with different prefix + "other+*,external_dep,external_dep~other", + ], + ) as rm: + dir = os.path.dirname(rm.Path()) + r = runfiles.CreateDirectoryBased(dir) + + # Exact mapping should take precedence over prefix + self.assertEqual( + r.Rlocation("external_dep/foo/file", "deps+specific_repo"), + dir + "/external_dep~exact/foo/file", + ) + + # Other repos with deps+ prefix should use the prefixed mapping + self.assertEqual( + r.Rlocation("external_dep/foo/file", "deps+other_repo"), + dir + "/external_dep~prefix/foo/file", + ) + + # Different prefix should use its own mapping + self.assertEqual( + r.Rlocation("external_dep/foo/file", "other+some_repo"), + dir + "/external_dep~other/foo/file", + ) + + def testDirectoryBasedRlocationWithCompactRepoMappingOrderMatters(self) -> None: + """Test that order matters for prefixed mappings (first match wins).""" + with _MockFile( + name="_repo_mapping", + contents=[ + # More specific prefix comes first + "deps+specific+*,lib,lib~specific", + # More general prefix comes second + "deps+*,lib,lib~general", + ], + ) as rm: + dir = os.path.dirname(rm.Path()) + r = runfiles.CreateDirectoryBased(dir) + + # Should match the more specific prefix first + self.assertEqual( + r.Rlocation("lib/foo/file", "deps+specific+repo"), + dir + "/lib~specific/foo/file", + ) + + # Should match the general prefix for non-specific repos + self.assertEqual( + r.Rlocation("lib/foo/file", "deps+other_repo"), + dir + "/lib~general/foo/file", + ) + + def testRepositoryMappingLookup(self) -> None: + """Test _RepositoryMapping.lookup() method for both exact and prefix-based mappings.""" + exact_mappings = { + ("", "my_workspace"): "_main", + ("", "config_lib"): "config_lib~1.0.0", + ("deps+specific_repo", "external_dep"): "external_dep~exact", + } + prefixed_mappings = { + ("deps+", "external_dep"): "external_dep~prefix", + ("test_deps+", "test_lib"): "test_lib~2.1.0", + } + + repo_mapping = _RepositoryMapping(exact_mappings, prefixed_mappings) + + # Test exact lookups + self.assertEqual(repo_mapping.lookup("", "my_workspace"), "_main") + self.assertEqual(repo_mapping.lookup("", "config_lib"), "config_lib~1.0.0") + self.assertEqual( + repo_mapping.lookup("deps+specific_repo", "external_dep"), + "external_dep~exact", + ) + + # Test prefix-based lookups + self.assertEqual( + repo_mapping.lookup("deps+some_repo", "external_dep"), "external_dep~prefix" + ) + self.assertEqual( + repo_mapping.lookup("test_deps+another_repo", "test_lib"), "test_lib~2.1.0" + ) + + # Test that exact takes precedence over prefix + self.assertEqual( + repo_mapping.lookup("deps+specific_repo", "external_dep"), + "external_dep~exact", + ) + + # Test non-existent mapping + self.assertIsNone(repo_mapping.lookup("nonexistent", "repo")) + self.assertIsNone(repo_mapping.lookup("unknown+repo", "missing")) + + # Test empty mapping + empty_mapping = _RepositoryMapping({}, {}) + self.assertIsNone(empty_mapping.lookup("any", "repo")) + + # Test is_empty() method + self.assertFalse(repo_mapping.is_empty()) # Should have mappings + self.assertTrue(empty_mapping.is_empty()) # Should be empty + def testCurrentRepository(self) -> None: # Under bzlmod, the current repository name is the empty string instead # of the name in the workspace file.
diff --git a/tools/publish/requirements_darwin.txt b/tools/publish/requirements_darwin.txt index bc7e1c6..39248d4 100644 --- a/tools/publish/requirements_darwin.txt +++ b/tools/publish/requirements_darwin.txt
@@ -113,9 +113,9 @@ --hash=sha256:9bae4ea555cf0b14938dc0aee7c9f32ed303aa20a3b73e7dc80111628792d1b3 \ --hash=sha256:f797fc481b490edb305122c9181830a3a5b76d84ef6d1aef2fb9b47ab956f9e4 # via keyring -jaraco-functools==4.1.0 \ - --hash=sha256:70f7e0e2ae076498e212562325e805204fc092d7b4c17e0e86c959e249701a9d \ - --hash=sha256:ad159f13428bc4acbf5541ad6dec511f91573b90fba04df61dafa2a1231cf649 +jaraco-functools==4.3.0 \ + --hash=sha256:227ff8ed6f7b8f62c56deff101545fa7543cf2c8e7b82a7c2116e672f29c26e8 \ + --hash=sha256:cfd13ad0dd2c47a3600b439ef72d8615d482cedcff1632930d6f28924d92f294 # via keyring keyring==25.6.0 \ --hash=sha256:0b39998aa941431eb3d9b0d4b2460bc773b9df6fed7621c2dfb291a7e0187a66 \
diff --git a/tools/publish/requirements_linux.txt b/tools/publish/requirements_linux.txt index 522a5f3..c078a0e 100644 --- a/tools/publish/requirements_linux.txt +++ b/tools/publish/requirements_linux.txt
@@ -238,9 +238,9 @@ --hash=sha256:9bae4ea555cf0b14938dc0aee7c9f32ed303aa20a3b73e7dc80111628792d1b3 \ --hash=sha256:f797fc481b490edb305122c9181830a3a5b76d84ef6d1aef2fb9b47ab956f9e4 # via keyring -jaraco-functools==4.1.0 \ - --hash=sha256:70f7e0e2ae076498e212562325e805204fc092d7b4c17e0e86c959e249701a9d \ - --hash=sha256:ad159f13428bc4acbf5541ad6dec511f91573b90fba04df61dafa2a1231cf649 +jaraco-functools==4.3.0 \ + --hash=sha256:227ff8ed6f7b8f62c56deff101545fa7543cf2c8e7b82a7c2116e672f29c26e8 \ + --hash=sha256:cfd13ad0dd2c47a3600b439ef72d8615d482cedcff1632930d6f28924d92f294 # via keyring jeepney==0.9.0 \ --hash=sha256:97e5714520c16fc0a45695e5365a2e11b81ea79bba796e26f9f1d178cb182683 \
diff --git a/tools/publish/requirements_universal.txt b/tools/publish/requirements_universal.txt index e8d1c74..a3a3f23 100644 --- a/tools/publish/requirements_universal.txt +++ b/tools/publish/requirements_universal.txt
@@ -221,9 +221,9 @@ --hash=sha256:9bae4ea555cf0b14938dc0aee7c9f32ed303aa20a3b73e7dc80111628792d1b3 \ --hash=sha256:f797fc481b490edb305122c9181830a3a5b76d84ef6d1aef2fb9b47ab956f9e4 # via keyring -jaraco-functools==4.1.0 \ - --hash=sha256:70f7e0e2ae076498e212562325e805204fc092d7b4c17e0e86c959e249701a9d \ - --hash=sha256:ad159f13428bc4acbf5541ad6dec511f91573b90fba04df61dafa2a1231cf649 +jaraco-functools==4.3.0 \ + --hash=sha256:227ff8ed6f7b8f62c56deff101545fa7543cf2c8e7b82a7c2116e672f29c26e8 \ + --hash=sha256:cfd13ad0dd2c47a3600b439ef72d8615d482cedcff1632930d6f28924d92f294 # via keyring jeepney==0.9.0 ; sys_platform == 'linux' \ --hash=sha256:97e5714520c16fc0a45695e5365a2e11b81ea79bba796e26f9f1d178cb182683 \
diff --git a/tools/publish/requirements_windows.txt b/tools/publish/requirements_windows.txt index 38d854e..b4eb83d 100644 --- a/tools/publish/requirements_windows.txt +++ b/tools/publish/requirements_windows.txt
@@ -113,9 +113,9 @@ --hash=sha256:9bae4ea555cf0b14938dc0aee7c9f32ed303aa20a3b73e7dc80111628792d1b3 \ --hash=sha256:f797fc481b490edb305122c9181830a3a5b76d84ef6d1aef2fb9b47ab956f9e4 # via keyring -jaraco-functools==4.1.0 \ - --hash=sha256:70f7e0e2ae076498e212562325e805204fc092d7b4c17e0e86c959e249701a9d \ - --hash=sha256:ad159f13428bc4acbf5541ad6dec511f91573b90fba04df61dafa2a1231cf649 +jaraco-functools==4.3.0 \ + --hash=sha256:227ff8ed6f7b8f62c56deff101545fa7543cf2c8e7b82a7c2116e672f29c26e8 \ + --hash=sha256:cfd13ad0dd2c47a3600b439ef72d8615d482cedcff1632930d6f28924d92f294 # via keyring keyring==25.6.0 \ --hash=sha256:0b39998aa941431eb3d9b0d4b2460bc773b9df6fed7621c2dfb291a7e0187a66 \