Standardise on pip_parse (#807)
diff --git a/README.md b/README.md
index 7359a2a..944493c 100644
--- a/README.md
+++ b/README.md
@@ -7,8 +7,7 @@
This repository is the home of the core Python rules -- `py_library`,
`py_binary`, `py_test`, and related symbols that provide the basis for Python
-support in Bazel. It also contains packaging rules for integrating with PyPI
-(`pip`). Documentation lives in the
+support in Bazel. It also contains package installation rules for integrating with PyPI and other package indices. Documentation lives in the
[`docs/`](https://github.com/bazelbuild/rules_python/tree/main/docs)
directory and in the
[Bazel Build Encyclopedia](https://docs.bazel.build/versions/master/be/python.html).
@@ -24,7 +23,7 @@
rate, but this repository will still follow
[semantic versioning](https://semver.org).
-The packaging rules (`pip_install`, etc.) are less stable. We may make breaking
+The package installation rules (`pip_install`, `pip_parse` etc.) are less stable. We may make breaking
changes as they evolve.
This repository is maintained by the Bazel community. Neither Google, nor the
@@ -101,14 +100,14 @@
)
```
-## Using the packaging rules
+## Using the package installation rules
Usage of the packaging rules involves two main steps.
-1. [Installing `pip` dependencies](#installing-pip-dependencies)
-2. [Consuming `pip` dependencies](#consuming-pip-dependencies)
+1. [Installing third_party packages](#installing-third_party-packages)
+2. [Using third_party packages as dependencies](#using-third_party-packages-as-dependencies)
-The packaging rules create two kinds of repositories: A central external repo that holds
+The package installation rules create two kinds of repositories: A central external repo that holds
downloaded wheel files, and individual external repos for each wheel's extracted
contents. Users only need to interact with the central external repo; the wheel repos
are essentially an implementation detail. The central external repo provides a
@@ -116,56 +115,13 @@
`BUILD` files that translates a pip package name into the label of a `py_library`
target in the appropriate wheel repo.
-### Installing `pip` dependencies
+### Installing third_party packages
To add pip dependencies to your `WORKSPACE`, load the `pip_install` function, and call it to create the
central external repo and individual wheel external repos.
```python
-load("@rules_python//python:pip.bzl", "pip_install")
-
-# Create a central external repo, @my_deps, that contains Bazel targets for all the
-# third-party packages specified in the requirements.txt file.
-pip_install(
- name = "my_deps",
- requirements = "//path/to:requirements.txt",
-)
-```
-
-Note that since `pip_install` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no
-information about the Python toolchain and cannot enforce that the interpreter
-used to invoke pip matches the interpreter used to run `py_binary` targets. By
-default, `pip_install` uses the system command `"python3"`. This can be overridden by passing the
-`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_install`.
-
-You can have multiple `pip_install`s in the same workspace. This will create multiple external repos that have no relation to
-one another, and may result in downloading the same wheels multiple times.
-
-As with any repository rule, if you would like to ensure that `pip_install` is
-re-executed in order to pick up a non-hermetic change to your environment (e.g.,
-updating your system `python` interpreter), you can force it to re-execute by running
-`bazel sync --only [pip_install name]`.
-
-### Fetch `pip` dependencies lazily
-
-One pain point with `pip_install` is the need to download all dependencies resolved by
-your requirements.txt before the bazel analysis phase can start. For large python monorepos
-this can take a long time, especially on slow connections.
-
-`pip_parse` provides a solution to this problem. If you can provide a lock
-file of all your python dependencies `pip_parse` will translate each requirement into its own external repository.
-Bazel will only fetch/build wheels for the requirements in the subgraph of your build target.
-
-There are API differences between `pip_parse` and `pip_install`:
-1. `pip_parse` requires a fully resolved lock file of your python dependencies. You can generate this by using the `compile_pip_requirements` rule,
- running `pip-compile` directly, or using virtualenv and `pip freeze`. `pip_parse` uses a label argument called `requirements_lock` instead of
- `requirements` to make this distinction clear.
-2. `pip_parse` translates your requirements into a starlark macro called `install_deps`. You must call this macro in your WORKSPACE to
- declare your dependencies.
-
-
-```python
load("@rules_python//python:pip.bzl", "pip_parse")
# Create a central repo that knows about the dependencies needed from
@@ -174,14 +130,33 @@
name = "my_deps",
requirements_lock = "//path/to:requirements_lock.txt",
)
-
# Load the starlark macro which will define your dependencies.
load("@my_deps//:requirements.bzl", "install_deps")
# Call it to define repos for your requirements.
install_deps()
```
-### Consuming `pip` dependencies
+Note that since `pip_parse` is a repository rule and therefore executes pip at WORKSPACE-evaluation time, Bazel has no
+information about the Python toolchain and cannot enforce that the interpreter
+used to invoke pip matches the interpreter used to run `py_binary` targets. By
+default, `pip_parse` uses the system command `"python3"`. This can be overridden by passing the
+`python_interpreter` attribute or `python_interpreter_target` attribute to `pip_parse`.
+
+You can have multiple `pip_parse`s in the same workspace. This will create multiple external repos that have no relation to
+one another, and may result in downloading the same wheels multiple times.
+
+As with any repository rule, if you would like to ensure that `pip_parse` is
+re-executed in order to pick up a non-hermetic change to your environment (e.g.,
+updating your system `python` interpreter), you can force it to re-execute by running
+`bazel sync --only [pip_parse name]`.
+
+Note: The `pip_install` rule is deprecated. `pip_parse` offers identical functionality and both `pip_install`
+and `pip_parse` now have the same implementation. The name `pip_install` may be removed in a future version of the rules.
+The maintainers have taken all reasonable efforts to faciliate a smooth transition, but some users of `pip_install` will
+need to replace their existing `requirements.txt` with a fully resolved set of dependencies using a tool such as
+`pip-tools` or the `compile_pip_requirements` repository rule.
+
+### Using third_party packages as dependencies
Each extracted wheel repo contains a `py_library` target representing
the wheel's contents. There are two ways to access this library. The
diff --git a/docs/pip.md b/docs/pip.md
index 4853e52..f6d8430 100644
--- a/docs/pip.md
+++ b/docs/pip.md
@@ -73,69 +73,19 @@
pip_install(<a href="#pip_install-requirements">requirements</a>, <a href="#pip_install-name">name</a>, <a href="#pip_install-kwargs">kwargs</a>)
</pre>
-Accepts a `requirements.txt` file and installs the dependencies listed within.
-
-Those dependencies become available in a generated `requirements.bzl` file.
-
-This macro wraps the [`pip_repository`](./pip_repository.md) rule that invokes `pip`.
-In your WORKSPACE file:
+Accepts a locked/compiled requirements file and installs the dependencies listed within.
```python
+load("@rules_python//python:pip.bzl", "pip_install")
+
pip_install(
+ name = "pip_deps",
requirements = ":requirements.txt",
)
-```
-You can then reference installed dependencies from a `BUILD` file with:
+load("@pip_deps//:requirements.bzl", "install_deps")
-```python
-load("@pip//:requirements.bzl", "requirement")
-py_library(
- name = "bar",
- ...
- deps = [
- "//my/other:dep",
- requirement("requests"),
- requirement("numpy"),
- ],
-)
-```
-
-> Note that this convenience comes with a cost.
-> Analysis of any BUILD file which loads the requirements helper in this way will
-> cause an eager-fetch of all the pip dependencies,
-> even if no python targets are requested to be built.
-> In a multi-language repo, this may cause developers to fetch dependencies they don't need,
-> so consider using the long form for dependencies if this happens.
-
-In addition to the `requirement` macro, which is used to access the `py_library`
-target generated from a package's wheel, the generated `requirements.bzl` file contains
-functionality for exposing [entry points][whl_ep] as `py_binary` targets.
-
-[whl_ep]: https://packaging.python.org/specifications/entry-points/
-
-```python
-load("@pip_deps//:requirements.bzl", "entry_point")
-
-alias(
- name = "pip-compile",
- actual = entry_point(
- pkg = "pip-tools",
- script = "pip-compile",
- ),
-)
-```
-
-Note that for packages whose name and script are the same, only the name of the package
-is needed when calling the `entry_point` macro.
-
-```python
-load("@pip_deps//:requirements.bzl", "entry_point")
-
-alias(
- name = "flake8",
- actual = entry_point("flake8"),
-)
+install_deps()
```
@@ -154,7 +104,7 @@
## pip_parse
<pre>
-pip_parse(<a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-kwargs">kwargs</a>)
+pip_parse(<a href="#pip_parse-requirements">requirements</a>, <a href="#pip_parse-requirements_lock">requirements_lock</a>, <a href="#pip_parse-name">name</a>, <a href="#pip_parse-kwargs">kwargs</a>)
</pre>
Accepts a locked/compiled requirements file and installs the dependencies listed within.
@@ -247,7 +197,8 @@
| Name | Description | Default Value |
| :-------------: | :-------------: | :-------------: |
-| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | none |
+| requirements | Deprecated. See requirements_lock. | <code>None</code> |
+| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. Note that if your lockfile is platform-dependent, you can use the <code>requirements_[platform]</code> attributes. | <code>None</code> |
| name | The name of the generated repository. The generated repositories containing each requirement will be of the form <name>_<requirement-name>. | <code>"pip_parsed_deps"</code> |
| kwargs | Additional arguments to the [<code>pip_repository</code>](./pip_repository.md) repository rule. | none |
diff --git a/docs/pip_repository.md b/docs/pip_repository.md
index c66d8bf..875ea11 100644
--- a/docs/pip_repository.md
+++ b/docs/pip_repository.md
@@ -6,9 +6,9 @@
<pre>
pip_repository(<a href="#pip_repository-name">name</a>, <a href="#pip_repository-annotations">annotations</a>, <a href="#pip_repository-download_only">download_only</a>, <a href="#pip_repository-enable_implicit_namespace_pkgs">enable_implicit_namespace_pkgs</a>, <a href="#pip_repository-environment">environment</a>,
- <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-incremental">incremental</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>,
- <a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>, <a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements">requirements</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>,
- <a href="#pip_repository-requirements_linux">requirements_linux</a>, <a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>)
+ <a href="#pip_repository-extra_pip_args">extra_pip_args</a>, <a href="#pip_repository-isolated">isolated</a>, <a href="#pip_repository-pip_data_exclude">pip_data_exclude</a>, <a href="#pip_repository-python_interpreter">python_interpreter</a>,
+ <a href="#pip_repository-python_interpreter_target">python_interpreter_target</a>, <a href="#pip_repository-quiet">quiet</a>, <a href="#pip_repository-repo_prefix">repo_prefix</a>, <a href="#pip_repository-requirements_darwin">requirements_darwin</a>, <a href="#pip_repository-requirements_linux">requirements_linux</a>,
+ <a href="#pip_repository-requirements_lock">requirements_lock</a>, <a href="#pip_repository-requirements_windows">requirements_windows</a>, <a href="#pip_repository-timeout">timeout</a>)
</pre>
A rule for importing `requirements.txt` dependencies into Bazel.
@@ -62,14 +62,12 @@
| enable_implicit_namespace_pkgs | If true, disables conversion of native namespace packages into pkg-util style namespace packages. When set all py_binary and py_test targets must specify either <code>legacy_create_init=False</code> or the global Bazel option <code>--incompatible_default_to_explicit_init_py</code> to prevent <code>__init__.py</code> being automatically generated in every directory.<br><br>This option is required to support some packages which cannot handle the conversion to pkg-util style. | Boolean | optional | False |
| environment | Environment variables to set in the pip subprocess. Can be used to set common variables such as <code>http_proxy</code>, <code>https_proxy</code> and <code>no_proxy</code> Note that pip is run with "--isolated" on the CLI so PIP_<VAR>_<NAME> style env vars are ignored, but env vars that control requests and urllib3 can be passed. | <a href="https://bazel.build/docs/skylark/lib/dict.html">Dictionary: String -> String</a> | optional | {} |
| extra_pip_args | Extra arguments to pass on to pip. Must not contain spaces. | List of strings | optional | [] |
-| incremental | Create the repository in incremental mode. | Boolean | optional | False |
| isolated | Whether or not to pass the [--isolated](https://pip.pypa.io/en/stable/cli/pip/#cmdoption-isolated) flag to the underlying pip command. Alternatively, the <code>RULES_PYTHON_PIP_ISOLATED</code> enviornment varaible can be used to control this flag. | Boolean | optional | True |
| pip_data_exclude | Additional data exclusion parameters to add to the pip packages BUILD file. | List of strings | optional | [] |
| python_interpreter | The python interpreter to use. This can either be an absolute path or the name of a binary found on the host's <code>PATH</code> environment variable. If no value is set <code>python3</code> is defaulted for Unix systems and <code>python.exe</code> for Windows. | String | optional | "" |
| python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
| quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True |
-| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@<name>//<prefix><sanitized-package-name>/...<br><br>For incremental mode the packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" |
-| requirements | A 'requirements.txt' pip requirements file. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
+| repo_prefix | Prefix for the generated packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" |
| requirements_darwin | Override the requirements_lock attribute when the host platform is Mac OS | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
| requirements_linux | Override the requirements_lock attribute when the host platform is Linux | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
| requirements_lock | A fully resolved 'requirements.txt' pip requirement file containing the transitive set of your dependencies. If this file is passed instead of 'requirements' no resolve will take place and pip_repository will create individual repositories for each of your dependencies so that wheels are fetched/built only for the targets specified by 'build/run/test'. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
@@ -108,7 +106,7 @@
| python_interpreter_target | If you are using a custom python interpreter built by another repository rule, use this attribute to specify its BUILD target. This allows pip_repository to invoke pip using the same interpreter as your toolchain. If set, takes precedence over python_interpreter. | <a href="https://bazel.build/docs/build-ref.html#labels">Label</a> | optional | None |
| quiet | If True, suppress printing stdout and stderr output to the terminal. | Boolean | optional | True |
| repo | Pointer to parent repo name. Used to make these rules rerun if the parent repo changes. | String | required | |
-| repo_prefix | Prefix for the generated packages. For non-incremental mode the packages will be of the form<br><br>@<name>//<prefix><sanitized-package-name>/...<br><br>For incremental mode the packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" |
+| repo_prefix | Prefix for the generated packages will be of the form<br><br>@<prefix><sanitized-package-name>//... | String | optional | "" |
| requirement | Python requirement string describing the package to make available | String | required | |
| timeout | Timeout (in seconds) on the rule's execution duration. | Integer | optional | 600 |
diff --git a/examples/BUILD b/examples/BUILD
index 41dd875..ee4d7e4 100644
--- a/examples/BUILD
+++ b/examples/BUILD
@@ -39,10 +39,5 @@
)
bazel_integration_test(
- name = "relative_requirements_example",
- timeout = "long",
-)
-
-bazel_integration_test(
name = "bzlmod_example",
)
diff --git a/examples/pip_install/BUILD b/examples/pip_install/BUILD
index ad983b2..35f5a93 100644
--- a/examples/pip_install/BUILD
+++ b/examples/pip_install/BUILD
@@ -88,9 +88,9 @@
genquery(
name = "yamllint_lib_by_version",
expression = """
- attr("tags", "\\bpypi_version=1.26.3\\b", "@pip//pypi__yamllint")
+ attr("tags", "\\bpypi_version=1.26.3\\b", "@pip_yamllint//:pkg")
intersect
- attr("tags", "\\bpypi_name=yamllint\\b", "@pip//pypi__yamllint")
+ attr("tags", "\\bpypi_name=yamllint\\b", "@pip_yamllint//:pkg")
""",
scope = [requirement("yamllint")],
)
@@ -99,7 +99,7 @@
name = "write_expected",
out = "expected",
content = [
- "@pip//pypi__yamllint:pypi__yamllint",
+ "@pip_yamllint//:pkg",
"",
],
)
diff --git a/examples/pip_install/WORKSPACE b/examples/pip_install/WORKSPACE
index 0b33a2b..f63d928 100644
--- a/examples/pip_install/WORKSPACE
+++ b/examples/pip_install/WORKSPACE
@@ -57,6 +57,11 @@
requirements = "//:requirements.txt",
)
+load("@pip//:requirements.bzl", "install_deps")
+
+# Initialize repositories for all packages in requirements.txt.
+install_deps()
+
# You could optionally use an in-build, compiled python interpreter as a toolchain,
# and also use it to execute pip.
#
diff --git a/examples/pip_install/pip_install_test.py b/examples/pip_install/pip_install_test.py
index 6092768..9fe51fa 100644
--- a/examples/pip_install/pip_install_test.py
+++ b/examples/pip_install/pip_install_test.py
@@ -37,11 +37,11 @@
self.assertListEqual(
env.split(" "),
[
- "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/INSTALL.md",
- "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/LICENSE",
- "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/NEWS",
- "external/pip/pypi__s3cmd/data/share/doc/packages/s3cmd/README.md",
- "external/pip/pypi__s3cmd/data/share/man/man1/s3cmd.1",
+ "external/pip_s3cmd/data/share/doc/packages/s3cmd/INSTALL.md",
+ "external/pip_s3cmd/data/share/doc/packages/s3cmd/LICENSE",
+ "external/pip_s3cmd/data/share/doc/packages/s3cmd/NEWS",
+ "external/pip_s3cmd/data/share/doc/packages/s3cmd/README.md",
+ "external/pip_s3cmd/data/share/man/man1/s3cmd.1",
],
)
@@ -51,13 +51,13 @@
self.assertListEqual(
env.split(" "),
[
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/DESCRIPTION.rst",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/INSTALLER",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/METADATA",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/RECORD",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/WHEEL",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/metadata.json",
- "external/pip/pypi__boto3/site-packages/boto3-1.14.51.dist-info/top_level.txt",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/DESCRIPTION.rst",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/INSTALLER",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/METADATA",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/RECORD",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/WHEEL",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/metadata.json",
+ "external/pip_boto3/site-packages/boto3-1.14.51.dist-info/top_level.txt",
],
)
diff --git a/examples/pip_repository_annotations/BUILD b/examples/pip_repository_annotations/BUILD
index 8c69c40..4fd124e 100644
--- a/examples/pip_repository_annotations/BUILD
+++ b/examples/pip_repository_annotations/BUILD
@@ -27,7 +27,7 @@
py_test(
name = "pip_install_annotations_test",
srcs = ["pip_repository_annotations_test.py"],
- env = {"WHEEL_PKG_DIR": "pip_installed/pypi__wheel"},
+ env = {"WHEEL_PKG_DIR": "pip_installed_wheel"},
main = "pip_repository_annotations_test.py",
deps = [
requirement("wheel"),
diff --git a/examples/pip_repository_annotations/WORKSPACE b/examples/pip_repository_annotations/WORKSPACE
index 8ee885d..aeea842 100644
--- a/examples/pip_repository_annotations/WORKSPACE
+++ b/examples/pip_repository_annotations/WORKSPACE
@@ -54,9 +54,9 @@
requirements_lock = "//:requirements.txt",
)
-load("@pip_parsed//:requirements.bzl", "install_deps")
+load("@pip_parsed//:requirements.bzl", install_pip_parse_deps = "install_deps")
-install_deps()
+install_pip_parse_deps()
# For a more thorough example of `pip_install`. See `@rules_python//examples/pip_install`
pip_install(
@@ -65,3 +65,7 @@
python_interpreter_target = interpreter,
requirements = "//:requirements.txt",
)
+
+load("@pip_installed//:requirements.bzl", install_pip_install_deps = "install_deps")
+
+install_pip_install_deps()
diff --git a/examples/relative_requirements/BUILD b/examples/relative_requirements/BUILD
deleted file mode 100644
index d24ee5f..0000000
--- a/examples/relative_requirements/BUILD
+++ /dev/null
@@ -1,10 +0,0 @@
-load("@pip//:requirements.bzl", "requirement")
-load("@rules_python//python:defs.bzl", "py_test")
-
-py_test(
- name = "main",
- srcs = ["main.py"],
- deps = [
- requirement("relative_package_name"),
- ],
-)
diff --git a/examples/relative_requirements/README.md b/examples/relative_requirements/README.md
deleted file mode 100644
index 4b9258e..0000000
--- a/examples/relative_requirements/README.md
+++ /dev/null
@@ -1,4 +0,0 @@
-# relative_requirements example
-
-This example shows how to use pip to fetch relative dependencies from a requirements.txt file,
-then use them in BUILD files as dependencies of Bazel targets.
diff --git a/examples/relative_requirements/WORKSPACE b/examples/relative_requirements/WORKSPACE
deleted file mode 100644
index 4ae91c3..0000000
--- a/examples/relative_requirements/WORKSPACE
+++ /dev/null
@@ -1,21 +0,0 @@
-workspace(name = "example_repo")
-
-local_repository(
- name = "rules_python",
- path = "../..",
-)
-
-load("@rules_python//python:repositories.bzl", "python_register_toolchains")
-
-python_register_toolchains(
- name = "python39",
- python_version = "3.9",
-)
-
-load("@python39//:defs.bzl", "interpreter")
-load("@rules_python//python:pip.bzl", "pip_install")
-
-pip_install(
- python_interpreter_target = interpreter,
- requirements = "//:requirements.txt",
-)
diff --git a/examples/relative_requirements/main.py b/examples/relative_requirements/main.py
deleted file mode 100644
index b8ac021..0000000
--- a/examples/relative_requirements/main.py
+++ /dev/null
@@ -1,5 +0,0 @@
-import relative_package_name
-
-if __name__ == "__main__":
- # Run a function from the relative package
- print(relative_package_name.test())
diff --git a/examples/relative_requirements/relative_package/relative_package_name/__init__.py b/examples/relative_requirements/relative_package/relative_package_name/__init__.py
deleted file mode 100644
index c031192..0000000
--- a/examples/relative_requirements/relative_package/relative_package_name/__init__.py
+++ /dev/null
@@ -1,2 +0,0 @@
-def test():
- return True
diff --git a/examples/relative_requirements/relative_package/setup.py b/examples/relative_requirements/relative_package/setup.py
deleted file mode 100644
index 052b519..0000000
--- a/examples/relative_requirements/relative_package/setup.py
+++ /dev/null
@@ -1,7 +0,0 @@
-from setuptools import setup
-
-setup(
- name="relative_package_name",
- version="1.0.0",
- packages=["relative_package_name"],
-)
diff --git a/examples/relative_requirements/requirements.txt b/examples/relative_requirements/requirements.txt
deleted file mode 100644
index 9a81317..0000000
--- a/examples/relative_requirements/requirements.txt
+++ /dev/null
@@ -1 +0,0 @@
-./relative_package
diff --git a/python/pip.bzl b/python/pip.bzl
index 954317f..dfafefe 100644
--- a/python/pip.bzl
+++ b/python/pip.bzl
@@ -21,69 +21,19 @@
package_annotation = _package_annotation
def pip_install(requirements = None, name = "pip", **kwargs):
- """Accepts a `requirements.txt` file and installs the dependencies listed within.
-
- Those dependencies become available in a generated `requirements.bzl` file.
-
- This macro wraps the [`pip_repository`](./pip_repository.md) rule that invokes `pip`.
- In your WORKSPACE file:
+ """Accepts a locked/compiled requirements file and installs the dependencies listed within.
```python
+ load("@rules_python//python:pip.bzl", "pip_install")
+
pip_install(
+ name = "pip_deps",
requirements = ":requirements.txt",
)
- ```
- You can then reference installed dependencies from a `BUILD` file with:
+ load("@pip_deps//:requirements.bzl", "install_deps")
- ```python
- load("@pip//:requirements.bzl", "requirement")
- py_library(
- name = "bar",
- ...
- deps = [
- "//my/other:dep",
- requirement("requests"),
- requirement("numpy"),
- ],
- )
- ```
-
- > Note that this convenience comes with a cost.
- > Analysis of any BUILD file which loads the requirements helper in this way will
- > cause an eager-fetch of all the pip dependencies,
- > even if no python targets are requested to be built.
- > In a multi-language repo, this may cause developers to fetch dependencies they don't need,
- > so consider using the long form for dependencies if this happens.
-
- In addition to the `requirement` macro, which is used to access the `py_library`
- target generated from a package's wheel, the generated `requirements.bzl` file contains
- functionality for exposing [entry points][whl_ep] as `py_binary` targets.
-
- [whl_ep]: https://packaging.python.org/specifications/entry-points/
-
- ```python
- load("@pip_deps//:requirements.bzl", "entry_point")
-
- alias(
- name = "pip-compile",
- actual = entry_point(
- pkg = "pip-tools",
- script = "pip-compile",
- ),
- )
- ```
-
- Note that for packages whose name and script are the same, only the name of the package
- is needed when calling the `entry_point` macro.
-
- ```python
- load("@pip_deps//:requirements.bzl", "entry_point")
-
- alias(
- name = "flake8",
- actual = entry_point("flake8"),
- )
+ install_deps()
```
Args:
@@ -92,17 +42,11 @@
**kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository.md) repository rule.
"""
- # Just in case our dependencies weren't already fetched
- pip_install_dependencies()
+ # buildifier: disable=print
+ print("pip_install is deprecated. Please switch to pip_parse. pip_install will be removed in a future release.")
+ pip_parse(requirements = requirements, name = name, **kwargs)
- pip_repository(
- name = name,
- requirements = requirements,
- repo_prefix = "pypi__",
- **kwargs
- )
-
-def pip_parse(requirements_lock, name = "pip_parsed_deps", **kwargs):
+def pip_parse(requirements = None, requirements_lock = None, name = "pip_parsed_deps", **kwargs):
"""Accepts a locked/compiled requirements file and installs the dependencies listed within.
Those dependencies become available in a generated `requirements.bzl` file.
@@ -195,6 +139,7 @@
fetched/built only for the targets specified by 'build/run/test'.
Note that if your lockfile is platform-dependent, you can use the `requirements_[platform]`
attributes.
+ requirements (Label): Deprecated. See requirements_lock.
name (str, optional): The name of the generated repository. The generated repositories
containing each requirement will be of the form <name>_<requirement-name>.
**kwargs (dict): Additional arguments to the [`pip_repository`](./pip_repository.md) repository rule.
@@ -203,10 +148,14 @@
# Just in case our dependencies weren't already fetched
pip_install_dependencies()
+ # Temporary compatibility shim.
+ # pip_install was previously document to use requirements while pip_parse was using requirements_lock.
+ # We would prefer everyone move to using requirements_lock, but we maintain a temporary shim.
+ reqs_to_use = requirements_lock if requirements_lock else requirements
+
pip_repository(
name = name,
- requirements_lock = requirements_lock,
+ requirements_lock = reqs_to_use,
repo_prefix = "{}_".format(name),
- incremental = True,
**kwargs
)
diff --git a/python/pip_install/extract_wheels/BUILD b/python/pip_install/extract_wheels/BUILD
index 158d34b..bc11885 100644
--- a/python/pip_install/extract_wheels/BUILD
+++ b/python/pip_install/extract_wheels/BUILD
@@ -9,7 +9,6 @@
"arguments.py",
"bazel.py",
"extract_single_wheel.py",
- "extract_wheels.py",
"namespace_pkgs.py",
"parse_requirements_to_bzl.py",
"requirements.py",
@@ -22,14 +21,6 @@
)
py_binary(
- name = "extract_wheels",
- srcs = [
- "extract_wheels.py",
- ],
- deps = [":lib"],
-)
-
-py_binary(
name = "extract_single_wheel",
srcs = [
"extract_single_wheel.py",
diff --git a/python/pip_install/extract_wheels/extract_single_wheel.py b/python/pip_install/extract_wheels/extract_single_wheel.py
index a7cc672..9c44eff 100644
--- a/python/pip_install/extract_wheels/extract_single_wheel.py
+++ b/python/pip_install/extract_wheels/extract_single_wheel.py
@@ -8,11 +8,32 @@
from python.pip_install.extract_wheels import arguments, bazel, requirements
from python.pip_install.extract_wheels.annotation import annotation_from_str_path
-from python.pip_install.extract_wheels.extract_wheels import (
- configure_reproducible_wheels,
-)
+def configure_reproducible_wheels() -> None:
+ """Modifies the environment to make wheel building reproducible.
+ Wheels created from sdists are not reproducible by default. We can however workaround this by
+ patching in some configuration with environment variables.
+ """
+
+ # wheel, by default, enables debug symbols in GCC. This incidentally captures the build path in the .so file
+ # We can override this behavior by disabling debug symbols entirely.
+ # https://github.com/pypa/pip/issues/6505
+ if "CFLAGS" in os.environ:
+ os.environ["CFLAGS"] += " -g0"
+ else:
+ os.environ["CFLAGS"] = "-g0"
+
+ # set SOURCE_DATE_EPOCH to 1980 so that we can use python wheels
+ # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl
+ if "SOURCE_DATE_EPOCH" not in os.environ:
+ os.environ["SOURCE_DATE_EPOCH"] = "315532800"
+
+ # Python wheel metadata files can be unstable.
+ # See https://bitbucket.org/pypa/wheel/pull-requests/74/make-the-output-of-metadata-files/diff
+ if "PYTHONHASHSEED" not in os.environ:
+ os.environ["PYTHONHASHSEED"] = "0"
+
def main() -> None:
parser = argparse.ArgumentParser(
description="Build and/or fetch a single wheel based on the requirement passed in"
diff --git a/python/pip_install/extract_wheels/extract_wheels.py b/python/pip_install/extract_wheels/extract_wheels.py
deleted file mode 100644
index 2addaf8..0000000
--- a/python/pip_install/extract_wheels/extract_wheels.py
+++ /dev/null
@@ -1,132 +0,0 @@
-"""extract_wheels
-
-extract_wheels resolves and fetches artifacts transitively from the Python Package Index (PyPI) based on a
-requirements.txt. It generates the required BUILD files to consume these packages as Python libraries.
-
-Under the hood, it depends on the `pip wheel` command to do resolution, download, and compilation into wheels.
-"""
-import argparse
-import glob
-import os
-import pathlib
-import subprocess
-import sys
-
-from python.pip_install.extract_wheels import (
- annotation,
- arguments,
- bazel,
- requirements,
- wheel,
-)
-
-
-def configure_reproducible_wheels() -> None:
- """Modifies the environment to make wheel building reproducible.
-
- Wheels created from sdists are not reproducible by default. We can however workaround this by
- patching in some configuration with environment variables.
- """
-
- # wheel, by default, enables debug symbols in GCC. This incidentally captures the build path in the .so file
- # We can override this behavior by disabling debug symbols entirely.
- # https://github.com/pypa/pip/issues/6505
- if "CFLAGS" in os.environ:
- os.environ["CFLAGS"] += " -g0"
- else:
- os.environ["CFLAGS"] = "-g0"
-
- # set SOURCE_DATE_EPOCH to 1980 so that we can use python wheels
- # https://github.com/NixOS/nixpkgs/blob/master/doc/languages-frameworks/python.section.md#python-setuppy-bdist_wheel-cannot-create-whl
- if "SOURCE_DATE_EPOCH" not in os.environ:
- os.environ["SOURCE_DATE_EPOCH"] = "315532800"
-
- # Python wheel metadata files can be unstable.
- # See https://bitbucket.org/pypa/wheel/pull-requests/74/make-the-output-of-metadata-files/diff
- if "PYTHONHASHSEED" not in os.environ:
- os.environ["PYTHONHASHSEED"] = "0"
-
-
-def main() -> None:
- """Main program.
-
- Exits zero on successful program termination, non-zero otherwise.
- """
-
- configure_reproducible_wheels()
-
- parser = argparse.ArgumentParser(
- description="Resolve and fetch artifacts transitively from PyPI"
- )
- parser.add_argument(
- "--requirements",
- action="store",
- required=True,
- help="Path to requirements.txt from where to install dependencies",
- )
- parser.add_argument(
- "--annotations",
- type=annotation.annotations_map_from_str_path,
- help="A json encoded file containing annotations for rendered packages.",
- )
- arguments.parse_common_args(parser)
- args = parser.parse_args()
- deserialized_args = dict(vars(args))
- arguments.deserialize_structured_args(deserialized_args)
-
- # Pip is run with the working directory changed to the folder containing the requirements.txt file, to allow for
- # relative requirements to be correctly resolved. The --wheel-dir is therefore required to be repointed back to the
- # current calling working directory (the repo root in .../external/name), where the wheel files should be written to
- pip_args = (
- [sys.executable, "-m", "pip"]
- + (["--isolated"] if args.isolated else [])
- + ["download" if args.download_only else "wheel", "-r", args.requirements]
- + ["--wheel-dir", os.getcwd()]
- + deserialized_args["extra_pip_args"]
- )
-
- env = os.environ.copy()
- env.update(deserialized_args["environment"])
-
- # Assumes any errors are logged by pip so do nothing. This command will fail if pip fails
- subprocess.run(
- pip_args,
- check=True,
- env=env,
- cwd=str(pathlib.Path(args.requirements).parent.resolve()),
- )
-
- extras = requirements.parse_extras(args.requirements)
-
- repo_label = "@%s" % args.repo
-
- # Locate all wheels
- wheels = [whl for whl in glob.glob("*.whl")]
-
- # Collect all annotations
- reqs = {whl: wheel.Wheel(whl).name for whl in wheels}
- annotations = args.annotations.collect(reqs.values())
-
- targets = [
- '"{}{}"'.format(
- repo_label,
- bazel.extract_wheel(
- wheel_file=whl,
- extras=extras,
- pip_data_exclude=deserialized_args["pip_data_exclude"],
- enable_implicit_namespace_pkgs=args.enable_implicit_namespace_pkgs,
- repo_prefix=args.repo_prefix,
- annotation=annotations.get(name),
- ),
- )
- for whl, name in reqs.items()
- ]
-
- with open("requirements.bzl", "w") as requirement_file:
- requirement_file.write(
- bazel.generate_requirements_file_contents(repo_label, targets)
- )
-
-
-if __name__ == "__main__":
- main()
diff --git a/python/pip_install/pip_repository.bzl b/python/pip_install/pip_repository.bzl
index d729ae9..bc7da73 100644
--- a/python/pip_install/pip_repository.bzl
+++ b/python/pip_install/pip_repository.bzl
@@ -222,8 +222,7 @@
requirements_txt = rctx.attr.requirements_windows
if not requirements_txt:
fail("""\
-Incremental mode requires a requirements_lock attribute be specified,
-or a platform-specific lockfile using one of the requirements_* attributes.
+A requirements_lock attribute must be specified, or a platform-specific lockfile using one of the requirements_* attributes.
""")
return requirements_txt
@@ -235,40 +234,28 @@
annotations_file = rctx.path("annotations.json")
rctx.file(annotations_file, json.encode_indent(annotations, indent = " " * 4))
- if rctx.attr.incremental:
- requirements_txt = _locked_requirements(rctx)
- args = [
- python_interpreter,
- "-m",
- "python.pip_install.extract_wheels.parse_requirements_to_bzl",
- "--requirements_lock",
- rctx.path(requirements_txt),
- "--requirements_lock_label",
- str(requirements_txt),
- # pass quiet and timeout args through to child repos.
- "--quiet",
- str(rctx.attr.quiet),
- "--timeout",
- str(rctx.attr.timeout),
- "--annotations",
- annotations_file,
- ]
+ requirements_txt = _locked_requirements(rctx)
+ args = [
+ python_interpreter,
+ "-m",
+ "python.pip_install.extract_wheels.parse_requirements_to_bzl",
+ "--requirements_lock",
+ rctx.path(requirements_txt),
+ "--requirements_lock_label",
+ str(requirements_txt),
+ # pass quiet and timeout args through to child repos.
+ "--quiet",
+ str(rctx.attr.quiet),
+ "--timeout",
+ str(rctx.attr.timeout),
+ "--annotations",
+ annotations_file,
+ ]
- args += ["--python_interpreter", _get_python_interpreter_attr(rctx)]
- if rctx.attr.python_interpreter_target:
- args += ["--python_interpreter_target", str(rctx.attr.python_interpreter_target)]
- progress_message = "Parsing requirements to starlark"
- else:
- args = [
- python_interpreter,
- "-m",
- "python.pip_install.extract_wheels.extract_wheels",
- "--requirements",
- rctx.path(rctx.attr.requirements),
- "--annotations",
- annotations_file,
- ]
- progress_message = "Extracting wheels"
+ args += ["--python_interpreter", _get_python_interpreter_attr(rctx)]
+ if rctx.attr.python_interpreter_target:
+ args += ["--python_interpreter_target", str(rctx.attr.python_interpreter_target)]
+ progress_message = "Parsing requirements to starlark"
args += ["--repo", rctx.attr.name, "--repo-prefix", rctx.attr.repo_prefix]
args = _parse_optional_attrs(rctx, args)
@@ -361,12 +348,7 @@
),
"repo_prefix": attr.string(
doc = """
-Prefix for the generated packages. For non-incremental mode the
-packages will be of the form
-
-@<name>//<prefix><sanitized-package-name>/...
-
-For incremental mode the packages will be of the form
+Prefix for the generated packages will be of the form
@<prefix><sanitized-package-name>//...
""",
@@ -387,14 +369,6 @@
"annotations": attr.string_dict(
doc = "Optional annotations to apply to packages",
),
- "incremental": attr.bool(
- default = False,
- doc = "Create the repository in incremental mode.",
- ),
- "requirements": attr.label(
- allow_single_file = True,
- doc = "A 'requirements.txt' pip requirements file.",
- ),
"requirements_darwin": attr.label(
allow_single_file = True,
doc = "Override the requirements_lock attribute when the host platform is Mac OS",
diff --git a/python/pip_install/private/srcs.bzl b/python/pip_install/private/srcs.bzl
index bdd76b1..e42bb8e 100644
--- a/python/pip_install/private/srcs.bzl
+++ b/python/pip_install/private/srcs.bzl
@@ -12,7 +12,6 @@
"@rules_python//python/pip_install/extract_wheels:arguments.py",
"@rules_python//python/pip_install/extract_wheels:bazel.py",
"@rules_python//python/pip_install/extract_wheels:extract_single_wheel.py",
- "@rules_python//python/pip_install/extract_wheels:extract_wheels.py",
"@rules_python//python/pip_install/extract_wheels:namespace_pkgs.py",
"@rules_python//python/pip_install/extract_wheels:parse_requirements_to_bzl.py",
"@rules_python//python/pip_install/extract_wheels:requirements.py",
diff --git a/tests/pip_repository_entry_points/WORKSPACE b/tests/pip_repository_entry_points/WORKSPACE
index 07a5d3a..dd80db4 100644
--- a/tests/pip_repository_entry_points/WORKSPACE
+++ b/tests/pip_repository_entry_points/WORKSPACE
@@ -24,9 +24,9 @@
requirements_lock = "//:requirements.txt",
)
-load("@pip_parsed//:requirements.bzl", "install_deps")
+load("@pip_parsed//:requirements.bzl", install_pip_parse_deps = "install_deps")
-install_deps()
+install_pip_parse_deps()
# For a more thorough example of `pip_install`. See `@rules_python//examples/pip_install`
pip_install(
@@ -34,3 +34,7 @@
python_interpreter_target = interpreter,
requirements = "//:requirements.txt",
)
+
+load("@pip_installed//:requirements.bzl", install_pip_install_deps = "install_deps")
+
+install_pip_install_deps()
