Compare commits

..

No commits in common. "devel" and "0.9.0" have entirely different histories.
devel ... 0.9.0

29 changed files with 2558 additions and 2478 deletions

View File

@ -8,7 +8,7 @@
set -e; set -e;
CI_CACHE=$HOME/.cache; CI_CACHE=$HOME/.cache;
POETRY_VERSION=1.3.2; POETRY_VERSION=1.2.0;
mkdir --parents "$CI_CACHE"; mkdir --parents "$CI_CACHE";
@ -26,6 +26,7 @@ poetry --version --no-ansi;
poetry run pip --version; poetry run pip --version;
poetry install \ poetry install \
--extras poetry \
--quiet \ --quiet \
--remove-untracked \ --remove-untracked \
--no-ansi; --no-ansi;

View File

@ -12,16 +12,14 @@ jobs:
strategy: strategy:
matrix: matrix:
python: python:
- version: "3.7"
toxenv: py37
- version: "3.8" - version: "3.8"
toxenv: py38 toxenv: py38
- version: "3.9" - version: "3.9"
toxenv: py39 toxenv: py39
- version: "3.10" - version: "3.10"
toxenv: py310 toxenv: py310
- version: "3.11"
toxenv: py311
- version: "3.12"
toxenv: py312
fail-fast: true fail-fast: true
steps: steps:
- name: Checkout - name: Checkout

View File

@ -47,13 +47,12 @@ repos:
- id: reorder-python-imports - id: reorder-python-imports
name: reorder-python-imports name: reorder-python-imports
entry: isort entry: reorder-python-imports
language: system language: system
require_serial: true args:
- "--unclassifiable-application-module=tox_poetry_installer"
types: types:
- python - python
args:
- "--filter-files"
- id: black - id: black
name: black name: black

View File

@ -11,6 +11,7 @@
# --disable=W" # --disable=W"
disable=logging-fstring-interpolation disable=logging-fstring-interpolation
,logging-format-interpolation ,logging-format-interpolation
,bad-continuation
,line-too-long ,line-too-long
,ungrouped-imports ,ungrouped-imports
,typecheck ,typecheck

View File

@ -2,65 +2,13 @@
See also: [Github Release Page](https://github.com/enpaul/tox-poetry-installer/releases). See also: [Github Release Page](https://github.com/enpaul/tox-poetry-installer/releases).
## Version 1.0.0 Beta 1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/1.0.0b1),
[PyPI](https://pypi.org/project/tox-poetry-installer/1.0.0b1)
- Update Poetry compatibility to include >=1.5
- Update Tox compatibility to use Tox 4
- Remove support for Tox 3
- Remove deprecated `--require-poetry` command line option
- Remove deprecated `install_dev_deps` confguration option
- Remove deprecated `--parallelize-locked-install` command line option
## Version 0.10.3
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.10.3),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.10.3/)
- Update Poetry requirement to exclude usage with incompatible 1.5 release
## Version 0.10.2
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.10.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.10.2/)
- Update documentation with best practices and Poetry 1.2+ command syntax
- Fix failed install of sdist package not raising an exception in multi-threaded mode.
Contributed by [chriskuehl](https://github.com/chriskuehl) (#86)
## Version 0.10.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.10.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.10.1/)
- Add PyPI classifier for Python-3.11 compatibility
- Add CI support for Python-3.11
- Add support for Poetry-1.3.x (#83)
## Version 0.10.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.10.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.10.0/)
- Add `poetry_dep_groups` option to support installing groups of Poetry dependencies.
Contributed by [Oshmoun](https://github.com/oshmoun) (#76)
- Deprecate `install_dev_deps` option
## Version 0.9.0 ## Version 0.9.0
View this release on: View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.9.0), [Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.9.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.9.0/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.9.0/)
- Add support for Poetry-1.2.x. Contributed by [Justin Wood](https://github.com/Callek) - Add support for Poetry-1.2.x (#73)
(#73)
- Update Black formatting to stable release version - Update Black formatting to stable release version
- Remove support for Python-3.6 - Remove support for Python-3.6
- Remove support for Poetry-1.1.x - Remove support for Poetry-1.1.x
@ -81,8 +29,8 @@ View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.8.4), [Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.8.4),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.8.4/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.8.4/)
- Fix issue where incompatible package versions were selected for installation when - Fix issue where incompatible package versions were selected for installation when multiple
multiple package versions were in the lockfile package versions were in the lockfile
## Version 0.8.3 ## Version 0.8.3
@ -99,9 +47,9 @@ View this release on:
[PyPI](https://pypi.org/project/tox-poetry-installer/0.8.2/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.8.2/)
- Improve debug-level logging for package installation, and time how long installing each - Improve debug-level logging for package installation, and time how long installing each
package takes. Contributed by [Rebecca Turner](https://github.com/9999years) (#63). package takes. Contributed by [Rebecca
- Fix crash caused by the package-under-test depending on Poetry's unsafe dependencies Turner](https://github.com/9999years).
([#65](https://github.com/enpaul/tox-poetry-installer/issues/65)) - Fix crash caused by the package-under-test depending on Poetry's unsafe dependencies ([#65](https://github.com/enpaul/tox-poetry-installer/issues/65))
## Version 0.8.1 ## Version 0.8.1
@ -151,8 +99,7 @@ View this release on:
process process
- Add integration with Poetry's compatibility - Add integration with Poetry's compatibility
[`Marker`](https://github.com/python-poetry/poetry-core/blob/master/poetry/core/version/markers.py) [`Marker`](https://github.com/python-poetry/poetry-core/blob/master/poetry/core/version/markers.py)
object system for determining package compatibility with the current platform object system for determining package compatibility with the current platform ([#43](https://github.com/enpaul/tox-poetry-installer/issues/43))
([#43](https://github.com/enpaul/tox-poetry-installer/issues/43))
- Add missing PyPI classifier for Python 3 - Add missing PyPI classifier for Python 3
## Version 0.6.3 ## Version 0.6.3
@ -177,8 +124,8 @@ View this release on:
[CVE-2020-29651](https://nvd.nist.gov/vuln/detail/CVE-2020-29651) [CVE-2020-29651](https://nvd.nist.gov/vuln/detail/CVE-2020-29651)
- Fix dependency identification failing when the package under test is a transient - Fix dependency identification failing when the package under test is a transient
dependency of a locked dependency specified for installation dependency of a locked dependency specified for installation
- Fix `AttributeError` being raised while creating the Tox self-provisioned environment - Fix `AttributeError` being raised while creating the Tox self-provisioned environment when
when using either the using either the
[`minversion`](https://tox.readthedocs.io/en/latest/config.html#conf-minversion) or [`minversion`](https://tox.readthedocs.io/en/latest/config.html#conf-minversion) or
[`requires`](https://tox.readthedocs.io/en/latest/config.html#conf-requires) Tox config [`requires`](https://tox.readthedocs.io/en/latest/config.html#conf-requires) Tox config
options options
@ -193,8 +140,8 @@ View this release on:
installation problems installation problems
- Fix regression around handling of Poetry's unsafe packages when the unsafe package is a - Fix regression around handling of Poetry's unsafe packages when the unsafe package is a
transient dependency ([#33](https://github.com/enpaul/tox-poetry-installer/issues/33)) transient dependency ([#33](https://github.com/enpaul/tox-poetry-installer/issues/33))
- Fix handling of Poetry's unsafe packages when the unsafe package is a primary - Fix handling of Poetry's unsafe packages when the unsafe package is a primary (environment
(environment or package) dependency or package) dependency
## Version 0.6.0 ## Version 0.6.0
@ -218,8 +165,7 @@ View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.5.2), [Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.5.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.5.2/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.5.2/)
- Fix always attempting to install dependencies with incompatible python version - Fix always attempting to install dependencies with incompatible python version constraints
constraints
- Fix always attempting to install dependencies with incompatible python platforms - Fix always attempting to install dependencies with incompatible python platforms
## Version 0.5.1 ## Version 0.5.1
@ -241,8 +187,8 @@ View this release on:
- Add option `locked_deps` to better support both locked and unlocked dependencies in a - Add option `locked_deps` to better support both locked and unlocked dependencies in a
single environment single environment
- Add blocking functionality when using `require_locked_deps = true` to prevent other - Add blocking functionality when using `require_locked_deps = true` to prevent other hooks
hooks from running after this one from running after this one
- Update documentation to include new configuration options and errors - Update documentation to include new configuration options and errors
- Update documentation to improve future maintainability - Update documentation to improve future maintainability
- Update module structure to move from single-file module to multi-file directory module - Update module structure to move from single-file module to multi-file directory module
@ -266,8 +212,7 @@ View this release on:
[PyPI](https://pypi.org/project/tox-poetry-installer/0.3.1/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.3.1/)
- Fix error when installing an environment with no extras specified in the configuration - Fix error when installing an environment with no extras specified in the configuration
- Fix problem where only the dependencies of the sequentially last extra would be - Fix problem where only the dependencies of the sequentially last extra would be installed
installed
- Fix regression causing no project dependencies to be installed - Fix regression causing no project dependencies to be installed
## Version 0.3.0 ## Version 0.3.0
@ -289,8 +234,7 @@ View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.4), [Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.4),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.4/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.2.4/)
- Fix support for Poetry-1.1 - Fix support for Poetry-1.1 ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
- Include tests in sdist ([#8](https://github.com/enpaul/tox-poetry-installer/issues/8)) - Include tests in sdist ([#8](https://github.com/enpaul/tox-poetry-installer/issues/8))
## Version 0.2.3 ## Version 0.2.3
@ -299,10 +243,8 @@ View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.3), [Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.3),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.3/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.2.3/)
- Fix usage of the plugin in non-Poetry based projects - Fix usage of the plugin in non-Poetry based projects ([#1](https://github.com/enpaul/tox-poetry-installer/issues/1))
([#1](https://github.com/enpaul/tox-poetry-installer/issues/1)) - Fix treating dependency names as case sensitive when they shouldn't be ([#7](https://github.com/enpaul/tox-poetry-installer/issues/7))
- Fix treating dependency names as case sensitive when they shouldn't be
([#7](https://github.com/enpaul/tox-poetry-installer/issues/7))
## Version 0.2.2 ## Version 0.2.2
@ -311,8 +253,7 @@ View this release on:
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.2/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.2.2/)
- Fix breaking when running Tox in projects that do not use Poetry for their - Fix breaking when running Tox in projects that do not use Poetry for their
environment/dependency management environment/dependency management ([#1](https://github.com/enpaul/tox-poetry-installer/issues/1))
([#1](https://github.com/enpaul/tox-poetry-installer/issues/1))
## Version 0.2.1 ## Version 0.2.1
@ -363,8 +304,8 @@ View this release on:
[PyPI](https://pypi.org/project/tox-poetry-installer/0.1.2/) [PyPI](https://pypi.org/project/tox-poetry-installer/0.1.2/)
- Test trivial functionality on Python-3.6 and Python-3.7 - Test trivial functionality on Python-3.6 and Python-3.7
- Fix disagreement between `pyproject.toml` and module metadata on what the current - Fix disagreement between `pyproject.toml` and module metadata on what the current version
version is is
- Fix constant named for PEP-440 that should have been named for PEP-508 - Fix constant named for PEP-440 that should have been named for PEP-508
## Version 0.1.1 ## Version 0.1.1

View File

@ -27,10 +27,9 @@ Examples of unacceptable behavior include:
- The use of sexualized language or imagery, and sexual attention or advances of any kind - The use of sexualized language or imagery, and sexual attention or advances of any kind
- Trolling, insulting or derogatory comments, and personal or political attacks - Trolling, insulting or derogatory comments, and personal or political attacks
- Public or private harassment - Public or private harassment
- Publishing others' private information, such as a physical or email address, without - Publishing others' private information, such as a physical or email address, without their
their explicit permission explicit permission
- Other conduct which could reasonably be considered inappropriate in a professional - Other conduct which could reasonably be considered inappropriate in a professional setting
setting
## Enforcement Responsibilities ## Enforcement Responsibilities
@ -53,8 +52,8 @@ offline event.
## Enforcement ## Enforcement
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the
community leaders responsible for enforcement at \[INSERT CONTACT METHOD\]. All complaints community leaders responsible for enforcement at \[INSERT CONTACT METHOD\]. All
will be reviewed and investigated promptly and fairly. complaints will be reviewed and investigated promptly and fairly.
All community leaders are obligated to respect the privacy and security of the reporter of All community leaders are obligated to respect the privacy and security of the reporter of
any incident. any incident.
@ -106,8 +105,8 @@ toward or disparagement of classes of individuals.
This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0, This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 2.0,
available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html. available at https://www.contributor-covenant.org/version/2/0/code_of_conduct.html.
Community Impact Guidelines were inspired by Community Impact Guidelines were inspired by [Mozilla's code of conduct
[Mozilla's code of conduct enforcement ladder](https://github.com/mozilla/diversity). enforcement ladder](https://github.com/mozilla/diversity).
For answers to common questions about this code of conduct, see the FAQ at For answers to common questions about this code of conduct, see the FAQ at
https://www.contributor-covenant.org/faq. Translations are available at https://www.contributor-covenant.org/faq. Translations are available at

View File

@ -30,10 +30,10 @@ source: ## Build Python source distribution package
poetry build --format sdist poetry build --format sdist
test: ## Run the project testsuite(s) test: ## Run the project testsuite(s)
poetry run tox --recreate --parallel poetry run tox --recreate
dev: ## Create the local dev environment dev: ## Create the local dev environment
poetry install --extras poetry --sync poetry install -E poetry --sync
poetry run pre-commit install poetry run pre-commit install
publish: test wheel source ## Build and upload to pypi (requires $PYPI_API_KEY be set) publish: test wheel source ## Build and upload to pypi (requires $PYPI_API_KEY be set)

158
README.md
View File

@ -25,6 +25,7 @@ environment dependencies from the [Poetry](https://python-poetry.org/) lockfile.
- [Other Notes](#other-notes) - [Other Notes](#other-notes)
- [Unsupported Tox config options](#unsupported-tox-config-options) - [Unsupported Tox config options](#unsupported-tox-config-options)
- [Updating locked dependencies in a testenv](#updating-locked-dependencies-in-a-testenv) - [Updating locked dependencies in a testenv](#updating-locked-dependencies-in-a-testenv)
- [Installing unsafe dependencies](#installing-unsafe-dependencies)
- [Using with an unmanaged Poetry installation](#using-with-an-unmanaged-poetry-installation) - [Using with an unmanaged Poetry installation](#using-with-an-unmanaged-poetry-installation)
- [Developing the Plugin](#developer-documentation) - [Developing the Plugin](#developer-documentation)
- [Road Map](#road-map) - [Road Map](#road-map)
@ -33,17 +34,16 @@ See the
[Changelog](https://github.com/enpaul/tox-poetry-installer/blob/devel/CHANGELOG.md) for [Changelog](https://github.com/enpaul/tox-poetry-installer/blob/devel/CHANGELOG.md) for
release history. release history.
*See also: [official Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html) and *See also: [official Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html), [Poetry Tox plugin](https://github.com/tkukushkin/tox-poetry)*
[the official Poetry documentation on using Tox](https://python-poetry.org/docs/faq/#is-tox-supported)*
## Feature Overview ## Feature Overview
- Manage package versions in exactly one place and with exactly one tool: Poetry. - Manage package versions in exactly one place and with exactly one tool: Poetry.
- Ensure CI/CD and other automation tools are using the same package versions that you are - Ensure CI/CD and other automation tools are using the same package versions that you are
in your local development environment. in your local development environment.
- Add only the packages or custom groups you need to a Tox test environment, instead of - Add only the packages you need to a Tox test environment, instead of everything in your
everything in your lockfile. lockfile.
- Directly integrate with Poetry, re-using your existing package indexes and credentials, - Directly integrate with Poetry, re-using your existing package indexes and credentials
with no additional configuration. with no additional configuration.
- Wherever possible, built-in Tox config options are always respected and their behavior - Wherever possible, built-in Tox config options are always respected and their behavior
kept consistent. kept consistent.
@ -53,19 +53,17 @@ release history.
## User Documentation ## User Documentation
*This section is for users looking to integrate the plugin with their project or CI *This section is for users looking to integrate the plugin with their project or CI system. For information on contributing to the plugin please see the [Developer Docs](#developer-documentation)*
system. For information on contributing to the plugin please see the
[Developer Docs](#developer-documentation)*
### Installing ### Installing
The recommended way to install the plugin is to add it to a project using Poetry: The recommended way to install the plugin is to add it to a project using Poetry:
```bash ```bash
poetry add -G dev tox-poetry-installer[poetry] poetry add tox-poetry-installer[poetry] --dev
``` ```
> **Note:** Always install the plugin with the `[poetry]` extra, unless you are > **Note:** Always install the plugin with the `[poetry]` extra, unless you are
> [managing the Poetry installation yourself](#externally-managed-poetry-installation). > [managing the Poetry installation yourself](#externally-managed-poetry-installation).
Alternatively, it can be installed directly to a virtual environment using Pip, though Alternatively, it can be installed directly to a virtual environment using Pip, though
@ -82,16 +80,16 @@ adding the below to `tox.ini`, though this is also not recommended:
```ini ```ini
requires = requires =
tox-poetry-installer[poetry] == 0.10.2 tox-poetry-installer[poetry] == 0.8.0
``` ```
After installing, check that Tox recognizes the plugin by running After installing, check that Tox recognizes the plugin by running
`poetry run tox --version`. The command should give output similar to below: `poetry run tox --version`. The command should give output similar to below:
``` ```
3.20.0 imported from .venv/lib64/python3.10/site-packages/tox/__init__.py 3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py
registered plugins: registered plugins:
tox-poetry-installer-0.10.2 at .venv/lib64/python3.10/site-packages/tox_poetry_installer/__init__.py tox-poetry-installer-0.8.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer/__init__.py
``` ```
### Quick Start ### Quick Start
@ -148,24 +146,23 @@ locked_deps =
commands = ... commands = ...
``` ```
> **Note:** Settings configured on the main `testenv` environment are inherited by > **Note:** Settings configured on the main `testenv` environment are inherited by child
> child test environments (for example, `testenv:foo`). To override this, specify the > test environments (for example, `testenv:foo`). To override this, specify the setting in
> setting in the child environment with a different value. > the child environment with a different value.
Alternatively, we can skip specifying all of our dependencies for a test environment in Alternatively, we can skip specifying all of our dependencies for a test environment in
the Tox config and install Poetry dependency groups directly: the Tox config and just install all of our Poetry dev-dependencies automatically:
```ini ```ini
[testenv] [testenv]
description = Some very cool tests description = Some very cool tests
require_locked_deps = true require_locked_deps = true
poetry_dep_groups = install_dev_deps = true
dev
commands = ... commands = ...
``` ```
> **Note:** The `install_dev_deps` configuration option is deprecated. See > **Note:** Setting `install_dev_deps = true` on an environment that also installs the
> [Configuration Options](#configuration-options) for more information. > project package is functionally equivalent to running `poetry install`.
Finally, we can also install an unlocked dependency (a dependency which doesn't take its Finally, we can also install an unlocked dependency (a dependency which doesn't take its
version from the Poetry lockfile) into the test environment alongside the locked ones. We version from the Poetry lockfile) into the test environment alongside the locked ones. We
@ -193,17 +190,17 @@ All options listed below are Tox environment options and can be applied to one o
environment sections of the `tox.ini` file. They cannot be applied to the global Tox environment sections of the `tox.ini` file. They cannot be applied to the global Tox
configuration section. configuration section.
> **Note:** Settings configured on the main `testenv` environment are inherited by > **Note:** Settings configured on the main `testenv` environment are inherited by child
> child test environments (for example, `testenv:foo`). To override this, specify the > test environments (for example, `testenv:foo`). To override this, specify the setting in
> setting in the child environment with a different value. > the child environment with a different value.
| Option | Type | Default | Description | | Option | Type | Default | Description |
| :--------------------- | :-----: | :-----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | :--------------------- | :-----: | :-----: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `locked_deps` | List | `[]` | Names of packages to install to the test environment from the Poetry lockfile. Transient dependencies (packages required by these dependencies) are automatically included. | | `locked_deps` | List | `[]` | Names of packages to install to the test environment from the Poetry lockfile. Transient dependencies (packages required by these dependencies) are automatically included. |
| `require_locked_deps` | Boolean | False | Whether the plugin should block attempts to install unlocked dependencies to the test environment. If enabled, then the [`tox_testenv_install_deps`](https://tox.readthedocs.io/en/latest/plugins.html#tox.hookspecs.tox_testenv_install_deps) plugin hook will be intercepted and an error will be raised if the test environment has the `deps` option configured. | | `require_locked_deps` | Boolean | False | Whether the plugin should block attempts to install unlocked dependencies to the test environment. If enabled, then the [`tox_testenv_install_deps`](https://tox.readthedocs.io/en/latest/plugins.html#tox.hookspecs.tox_testenv_install_deps) plugin hook will be intercepted and an error will be raised if the test environment has the `deps` option configured. |
| `install_dev_deps` | Boolean | False | Whether all of the Poetry dev-dependencies should be installed to the test environment. |
| `install_project_deps` | Boolean | True | Whether all of the Poetry primary dependencies for the project package should be installed to the test environment. | | `install_project_deps` | Boolean | True | Whether all of the Poetry primary dependencies for the project package should be installed to the test environment. |
| `require_poetry` | Boolean | False | Whether Tox should be forced to fail if the plugin cannot import Poetry locally. If `False` then the plugin will be skipped for the test environment if Poetry cannot be imported. If `True` then the plugin will force the environment to error and the Tox run to fail. | | `require_poetry` | Boolean | False | Whether Tox should be forced to fail if the plugin cannot import Poetry locally. If `False` then the plugin will be skipped for the test environment if Poetry cannot be imported. If `True` then the plugin will force the environment to error and the Tox run to fail. |
| `poetry_dep_groups` | List | `[]` | Names of Poetry dependency groups specified in `pyproject.toml` to install to the test environment. |
### Runtime Options ### Runtime Options
@ -214,6 +211,13 @@ of the plugin.
| :--------------------------- | :-----: | :-----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | :--------------------------- | :-----: | :-----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `--parallel-install-threads` | Integer | `10` | Number of worker threads to use to install dependencies in parallel. Installing in parallel with more threads can greatly speed up the install process, but can cause race conditions during install. Pass this option with the value `0` to entirely disable parallel installation. | | `--parallel-install-threads` | Integer | `10` | Number of worker threads to use to install dependencies in parallel. Installing in parallel with more threads can greatly speed up the install process, but can cause race conditions during install. Pass this option with the value `0` to entirely disable parallel installation. |
> **Note:** The `--require-poetry` runtime option is deprecated and will be removed in
> version 1.0.0. Please set `require_poetry = true` in `tox.ini` for environments that
> should fail if Poetry is not available.
> **Note:** The `--parallelize-locked-install` option is deprecated and will be removed in
> version 1.0.0. Please use the `--parallel-install-threads` option.
### Errors ### Errors
There are several errors that the plugin can encounter for a test environment when Tox is There are several errors that the plugin can encounter for a test environment when Tox is
@ -227,10 +231,11 @@ error will be set to one of the "Status" values below to indicate what the error
| `LockedDepNotFoundError` | Indicates that an item specified in the `locked_deps` config option does not match the name of a package in the Poetry lockfile. | | `LockedDepNotFoundError` | Indicates that an item specified in the `locked_deps` config option does not match the name of a package in the Poetry lockfile. |
| `LockedDepsRequiredError` | Indicates that a test environment with the `require_locked_deps` config option set to `true` also specified unlocked dependencies using the [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) config option. | | `LockedDepsRequiredError` | Indicates that a test environment with the `require_locked_deps` config option set to `true` also specified unlocked dependencies using the [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) config option. |
| `PoetryNotInstalledError` | Indicates that the `poetry` module could not be imported under the current runtime environment, and `require_poetry = true` was specified. | | `PoetryNotInstalledError` | Indicates that the `poetry` module could not be imported under the current runtime environment, and `require_poetry = true` was specified. |
| `RequiresUnsafeDepError` | Indicates that the package-under-test depends on a package that Poetry has classified as unsafe and cannot be installed. |
> **Note:** One or more of these errors can be caused by the `pyproject.toml` being out > **Note:** One or more of these errors can be caused by the `pyproject.toml` being out of
> of sync with the Poetry lockfile. If this is the case, than a warning will be logged > sync with the Poetry lockfile. If this is the case, than a warning will be logged when Tox
> when Tox is run. > is run.
### Other Notes ### Other Notes
@ -241,8 +246,8 @@ these options are made obsolete by the Poetry lockfile: either they aren't neede
equivalent functionality is instead taken directly from the package details Poetry stores equivalent functionality is instead taken directly from the package details Poetry stores
in its lockfile. in its lockfile.
> **Note:** The unsupported Tox config options will still apply to unlocked > **Note:** The unsupported Tox config options will still apply to unlocked dependencies
> dependencies being installed with the default Tox installation backend. > being installed with the default Tox installation backend.
- [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command) - [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command)
- [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre) - [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre)
@ -263,24 +268,41 @@ built-in
[`--recreate`](https://tox.readthedocs.io/en/latest/example/basic.html#forcing-re-creation-of-virtual-environments) [`--recreate`](https://tox.readthedocs.io/en/latest/example/basic.html#forcing-re-creation-of-virtual-environments)
option) for the new version to be found and installed. option) for the new version to be found and installed.
> **Note:** To force Tox to always recreate a test environment the > **Note:** To force Tox to always recreate a test environment the
> [`recreate`](https://tox.readthedocs.io/en/latest/config.html#conf-recreate) config > [`recreate`](https://tox.readthedocs.io/en/latest/config.html#conf-recreate) config option
> option can be set. > can be set.
#### Installing unsafe dependencies
There are several packages that cannot be installed from the lockfile because they are
excluded by Poetry itself. As a result these packages cannot be installed by this plugin
either as test environment dependencies passed directly to `locked_deps` or as a transient
dependency. When one of these packages is encountered by the plugin a warning will be
logged to the console and
**the unsafe package will not be installed to the test environment**.
This list can be found in the Poetry source code
[here](https://github.com/python-poetry/poetry/blob/master/poetry/puzzle/provider.py). As
of [Poetry 1.1.6](https://github.com/python-poetry/poetry/releases/tag/1.1.6) there are
four packages classified as "unsafe" by Poetry and excluded from the lockfile:
- `setuptools`
- `distribute`
- `pip`
- `wheel`
#### Using with an unmanaged Poetry installation #### Using with an unmanaged Poetry installation
In CI/CD systems, automation environments, or other Python environments where the loaded In CI/CD systems, automation environments, or other Python environments where the loaded
site packages are not managed by Poetry, it can be useful to manage the local installation site packages are not managed by Poetry, it can be useful to manage the local installation
of Poetry externally. This also helps to avoid problems that can be caused by the of Poetry externally. This also helps to avoid problems that can be caused by the
`--no-root`, `--without dev`, or `--sync` arguments to the `poetry install` command which, `--no-root`, `--no-dev`, or `--remove-untracked` arguments to the `poetry install` command
in some situations, can cause Poetry to uninstall itself if Poetry is specified as a which, in some situations, can cause Poetry to uninstall itself if Poetry is specified as
dependency of one of the packages it is managing (like this plugin). To support these use a dependency of one of the packages it is managing (like this plugin). To support these
cases, this plugin specifies the `poetry` package as an optional dependency that can be use cases, this plugin specifies the `poetry` package as an optional dependency that can
installed using a setuptools extra also named `poetry`. be installed using a setuptools extra also named `poetry`.
> ⚠️ **Warning:** This plugin requires Poetry to function. If the plugin is installed **Critical Warning: This plugin requires Poetry to function. If the plugin is installed without the `poetry` setuptools extra then Poetry must be installed independently for the plugin to function properly.**
> without the `poetry` setuptools extra then Poetry must be installed independently for
> the plugin to function properly.
To skip installing the `poetry` package as a dependency of `tox-poetry-installer`, do not To skip installing the `poetry` package as a dependency of `tox-poetry-installer`, do not
specify the `poetry` extra when adding the plugin: specify the `poetry` extra when adding the plugin:
@ -288,11 +310,11 @@ specify the `poetry` extra when adding the plugin:
```bash ```bash
# Adding the package without the "[poetry]" extra specifier so that # Adding the package without the "[poetry]" extra specifier so that
# Poetry is not added as a transient dev-dependency: # Poetry is not added as a transient dev-dependency:
poetry add -G dev tox-poetry-installer poetry add tox-poetry-installer --dev
# Adding the package with the "[poetry]" extra specifier, so the Poetry # Adding the package with the "[poetry]" extra specifier, so the Poetry
# package will be added to the environment and tracked in the lockfile: # package will be added to the environment and tracked in the lockfile:
poetry add -G dev tox-poetry-installer[poetry] poetry add tox-poetry-installer[poetry] --dev
``` ```
Once the plugin is installed- either with or without the Poetry extra- you can validate Once the plugin is installed- either with or without the Poetry extra- you can validate
@ -308,15 +330,14 @@ python -c '\
' '
``` ```
> **Note:** To force Tox to fail if Poetry is not installed, add the > **Note:** To force Tox to fail if Poetry is not installed, add the `require_poetry = true`
> `require_poetry = true` option to the tox `[testenv]` configuration. See the > option to the tox `[testenv]` configuration. See the
> [Config Options](#configuration-options) for more information. > [Config Options](#configuration-options) for more information.
## Developer Documentation ## Developer Documentation
All project contributors and participants are expected to adhere to the All project contributors and participants are expected to adhere to the
[Contributor Covenant Code of Conduct, v2](CODE_OF_CONDUCT.md) [Contributor Covenant Code of Conduct, v2](CODE_OF_CONDUCT.md) ([external link](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)).
([external link](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)).
The `devel` branch has the latest (and potentially unstable) changes. The stable releases The `devel` branch has the latest (and potentially unstable) changes. The stable releases
are tracked on [Github](https://github.com/enpaul/tox-poetry-installer/releases), are tracked on [Github](https://github.com/enpaul/tox-poetry-installer/releases),
@ -331,8 +352,8 @@ are tracked on [Github](https://github.com/enpaul/tox-poetry-installer/releases)
[fork the repository](https://docs.github.com/en/enterprise/2.20/user/github/getting-started-with-github/fork-a-repo) [fork the repository](https://docs.github.com/en/enterprise/2.20/user/github/getting-started-with-github/fork-a-repo)
and [open a pull request](https://github.com/enpaul/tox-poetry-installer/compare). and [open a pull request](https://github.com/enpaul/tox-poetry-installer/compare).
Developing this project requires [Python 3.10+](https://www.python.org/downloads/) and Developing this project requires [Python 3.7+](https://www.python.org/downloads/) and
[Poetry 1.4](https://python-poetry.org/docs/#installation) or later. GNU Make can [Poetry 1.2](https://python-poetry.org/docs/#installation) or later. GNU Make can
optionally be used to quickly setup a local development environment, but this is not optionally be used to quickly setup a local development environment, but this is not
required. required.
@ -357,10 +378,9 @@ make test
make help make help
``` ```
> **Note:** The pre-commit hooks require dependencies in the Poetry environment to run. > **Note:** The pre-commit hooks require dependencies in the Poetry environment to run. To
> To make a commit with the pre-commit hooks, you will need to run `poetry run git commit` > make a commit with the pre-commit hooks, you will need to run `poetry run git commit` or,
> or, alternatively, > alternatively, [launch an environment shell](https://python-poetry.org/docs/cli/#shell).
> [launch an environment shell](https://python-poetry.org/docs/cli/#shell).
## Road Map ## Road Map
@ -369,39 +389,35 @@ production environments on a provisional basis only.
- Beta classification was assigned with - Beta classification was assigned with
[v0.6.0](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.0) [v0.6.0](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.0)
- Stable classification will be assigned when the test suite covers an acceptable number - Stable classification will be assigned when the test suite covers an acceptable number of
of use cases use cases
### Path to Beta ### Path to Beta
- [x] Verify that primary package dependencies (from the `.package` env) are installed - [x] Verify that primary package dependencies (from the `.package` env) are installed
correctly using the Poetry backend. correctly using the Poetry backend.
- [x] Support the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) - [x] Support the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) Tox
Tox configuration option ([#4](https://github.com/enpaul/tox-poetry-installer/issues/4)) configuration option ([#4](https://github.com/enpaul/tox-poetry-installer/issues/4))
- [x] Add per-environment Tox configuration option to fall back to default installation - [x] Add per-environment Tox configuration option to fall back to default installation
backend. backend.
- [ ] ~Add warnings when an unsupported Tox configuration option is detected while using - [ ] ~Add warnings when an unsupported Tox configuration option is detected while using the
the Poetry backend.~ ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5)) Poetry backend.~ ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5))
- [x] Add trivial tests to ensure the project metadata is consistent between the - [x] Add trivial tests to ensure the project metadata is consistent between the pyproject.toml
pyproject.toml and the module constants. and the module constants.
- [x] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) and - [x] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) and improve
improve robustness of the Tox and Poetry module imports to avoid potentially breaking robustness of the Tox and Poetry module imports to avoid potentially breaking API changes
API changes in upstream packages. in upstream packages. ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
- [ ] ~Find and implement a way to mitigate the - [ ] ~Find and implement a way to mitigate the
[UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in
Poetry.~ ([#6](https://github.com/enpaul/tox-poetry-installer/issues/6)) Poetry.~ ([#6](https://github.com/enpaul/tox-poetry-installer/issues/6))
- [x] Fix logging to make proper use of Tox's logging reporter infrastructure - [x] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3))
([#3](https://github.com/enpaul/tox-poetry-installer/issues/3)) - [x] Add configuration option for installing all dev-dependencies to a testenv ([#14](https://github.com/enpaul/tox-poetry-installer/issues/14))
- [x] Add configuration option for installing all dev-dependencies to a testenv
([#14](https://github.com/enpaul/tox-poetry-installer/issues/14))
### Path to Stable ### Path to Stable
Everything in Beta plus... Everything in Beta plus...
- [ ] Fully replace dependency on `poetry` with dependency on `poetry-core` - [ ] Fully replace dependency on `poetry` with dependency on `poetry-core` ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
- [x] Add comprehensive unit tests - [x] Add comprehensive unit tests
- [ ] ~Add tests for each feature version of Tox between 3.8 and 3.20~ - [ ] ~Add tests for each feature version of Tox between 3.8 and 3.20~
- [x] Add tests for Python-3.6, 3.7, 3.8, and 3.9 - [x] Add tests for Python-3.6, 3.7, 3.8, and 3.9

3249
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,6 +1,6 @@
[tool.poetry] [tool.poetry]
name = "tox-poetry-installer" name = "tox-poetry-installer"
version = "1.0.0b1" version = "0.9.0"
license = "MIT" license = "MIT"
authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"] authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
description = "A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile" description = "A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile"
@ -23,46 +23,44 @@ classifiers = [
"Natural Language :: English", "Natural Language :: English",
"Operating System :: OS Independent", "Operating System :: OS Independent",
"Programming Language :: Python :: 3", "Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8", "Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9", "Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10", "Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: CPython",
] ]
[tool.poetry.plugins.tox] [tool.poetry.plugins.tox]
poetry_installer = "tox_poetry_installer" poetry_installer = "tox_poetry_installer"
[tool.poetry.extras]
poetry = ["poetry", "cleo"]
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = "^3.8" python = "^3.7"
cleo = ">=1.0,<3.0" cleo = {version = "^1.0.0a5", optional = true, allow-prereleases = true}
poetry = "^1.5.0" poetry = {version = "^1.2.0", optional = true}
poetry-core = "^1.1.0" poetry-core = "^1.1.0"
tox = "^4.1" tox = "^3.8.0"
[tool.poetry.group.dev.dependencies] [tool.poetry.group.dev.dependencies]
bandit = {version = "^1.7.7", python = "^3.10"} bandit = "^1.6.2"
black = {version = "^24.3.0", python = "^3.10"} black = "^22.3.0"
blacken-docs = {version = "^1.18.0", python = "^3.10"} blacken-docs = "^1.8.0"
ipython = {version = "^8.10.1", python = "^3.10"} ipython = "^7.18.1"
isort = {version = "^5.13.2", python = "^3.10"} mdformat = "^0.6"
mdformat = {version = "^0.7", python = "^3.10"} mdformat-gfm = "^0.2"
mdformat-gfm = {version = "^0.3", python = "^3.10"} mypy = "^0.930"
mypy = {version = "^1.11.1", python = "^3.10"} pre-commit = "^2.7.1"
pre-commit = {version = "^3.8.0", python = "^3.10"} pre-commit-hooks = "^3.3.0"
pre-commit-hooks = {version = "^4.6.0", python = "^3.10"} pylint = "^2.13.0"
pylint = {version = "^3.2.6", python = "^3.10"} pytest = "^6.0.2"
pytest = {version = "^8.3.2", python = "^3.10"} pytest-cov = "^2.10.1"
pytest-cov = {version = "^5.0.0", python = "^3.10"} reorder-python-imports = "^2.3.5"
toml = {version = "^0.10.1", python = "^3.10"} safety = "^1.9.0"
tox = "^4.1" toml = "^0.10.1"
types-toml = {version = "^0.10.1", python = "^3.10"} tox = "^3.20.0"
types-toml = "^0.10.1"
[tool.isort]
profile = "black"
force_single_line = "true"
lines_after_imports = 2
[build-system] [build-system]
requires = ["poetry-core>=1.1.0"] requires = ["poetry-core>=1.1.0"]

View File

@ -1,16 +1,15 @@
# pylint: disable=missing-module-docstring,missing-function-docstring,unused-argument,too-few-public-methods,protected-access # pylint: disable=missing-module-docstring, missing-function-docstring, unused-argument, too-few-public-methods
import time import time
from pathlib import Path from pathlib import Path
from typing import List
import poetry.factory import poetry.factory
import poetry.installation.executor import poetry.installation.pip_installer
import poetry.installation.operations.operation
import poetry.utils.env import poetry.utils.env
import pytest import pytest
import tox.tox_env.python.virtual_env.runner import tox
from poetry.core.packages.package import Package as PoetryPackage
import tox_poetry_installer.hooks._tox_on_install_helpers from tox_poetry_installer import utilities
TEST_PROJECT_PATH = Path(__file__).parent.resolve() / "test-project" TEST_PROJECT_PATH = Path(__file__).parent.resolve() / "test-project"
@ -21,8 +20,11 @@ FAKE_VENV_PATH = Path("nowhere")
class MockVirtualEnv: class MockVirtualEnv:
"""Mock class for the :class:`poetry.utils.env.VirtualEnv` and :class:`tox.venv.VirtualEnv`""" """Mock class for the :class:`poetry.utils.env.VirtualEnv` and :class:`tox.venv.VirtualEnv`"""
class MockTestenvConfig: # pylint: disable=missing-class-docstring
envdir = FAKE_VENV_PATH
def __init__(self, *args, **kwargs): def __init__(self, *args, **kwargs):
self.env_dir = FAKE_VENV_PATH self.envconfig = self.MockTestenvConfig()
self.installed = [] self.installed = []
@staticmethod @staticmethod
@ -34,38 +36,32 @@ class MockVirtualEnv:
return (1, 2, 3) return (1, 2, 3)
class MockExecutor: class MockPipInstaller:
"""Mock class for the :class:`poetry.installation.executor.Executor`""" """Mock class for the :class:`poetry.installation.pip_installer.PipInstaller`"""
def __init__(self, env: MockVirtualEnv, **kwargs): def __init__(self, env: MockVirtualEnv, **kwargs):
self.env = env self.env = env
def execute( def install(self, package: PoetryPackage):
self, operations: List[poetry.installation.operations.operation.Operation] self.env.installed.append(package)
):
self.env.installed.extend([operation.package for operation in operations])
time.sleep(1) time.sleep(1)
@pytest.fixture @pytest.fixture
def mock_venv(monkeypatch): def mock_venv(monkeypatch):
monkeypatch.setattr(utilities, "convert_virtualenv", lambda venv: venv)
monkeypatch.setattr( monkeypatch.setattr(
tox_poetry_installer.hooks._tox_on_install_helpers, poetry.installation.pip_installer, "PipInstaller", MockPipInstaller
"convert_virtualenv",
lambda venv: venv,
)
monkeypatch.setattr(poetry.installation.executor, "Executor", MockExecutor)
monkeypatch.setattr(
tox.tox_env.python.virtual_env.runner, "VirtualEnvRunner", MockVirtualEnv
) )
monkeypatch.setattr(tox.venv, "VirtualEnv", MockVirtualEnv)
monkeypatch.setattr(poetry.utils.env, "VirtualEnv", MockVirtualEnv) monkeypatch.setattr(poetry.utils.env, "VirtualEnv", MockVirtualEnv)
@pytest.fixture(scope="function") @pytest.fixture(scope="function")
def mock_poetry_factory(monkeypatch): def mock_poetry_factory(monkeypatch):
project = poetry.factory.Factory().create_poetry(cwd=TEST_PROJECT_PATH) pypoetry = poetry.factory.Factory().create_poetry(cwd=TEST_PROJECT_PATH)
def mock_factory(*args, **kwargs): def mock_factory(*args, **kwargs):
return project return pypoetry
monkeypatch.setattr(poetry.factory.Factory, "create_poetry", mock_factory) monkeypatch.setattr(poetry.factory.Factory, "create_poetry", mock_factory)

View File

@ -1,40 +1,35 @@
# pylint: disable=missing-module-docstring,redefined-outer-name,unused-argument,unused-import,protected-access # pylint: disable=missing-module-docstring, redefined-outer-name, unused-argument, wrong-import-order, unused-import
import time import time
from unittest import mock
import poetry.factory import tox.venv
import poetry.installation.executor from poetry.factory import Factory
import pytest
import tox.tox_env.python.virtual_env.runner
import tox_poetry_installer.hooks._tox_on_install_helpers
from .fixtures import mock_poetry_factory from .fixtures import mock_poetry_factory
from .fixtures import mock_venv from .fixtures import mock_venv
from tox_poetry_installer import installer
from tox_poetry_installer import utilities
def test_deduplication(mock_venv, mock_poetry_factory): def test_deduplication(mock_venv, mock_poetry_factory):
"""Test that the installer does not install duplicate dependencies""" """Test that the installer does not install duplicate dependencies"""
project = poetry.factory.Factory().create_poetry(None) poetry = Factory().create_poetry(None)
packages: tox_poetry_installer.hooks._tox_on_install_helpers.PackageMap = { packages: utilities.PackageMap = {
item.name: item for item in project.locker.locked_repository().packages item.name: item for item in poetry.locker.locked_repository().packages
} }
venv = tox.tox_env.python.virtual_env.runner.VirtualEnvRunner() venv = tox.venv.VirtualEnv()
to_install = [packages["toml"], packages["toml"]] to_install = [packages["toml"], packages["toml"]]
tox_poetry_installer.hooks._tox_on_install_helpers.install_package( installer.install(poetry, venv, to_install)
project, venv, to_install
)
assert len(set(to_install)) == len(venv.installed) # pylint: disable=no-member assert len(set(to_install)) == len(venv.installed) # pylint: disable=no-member
def test_parallelization(mock_venv, mock_poetry_factory): def test_parallelization(mock_venv, mock_poetry_factory):
"""Test that behavior is consistent between parallel and non-parallel usage""" """Test that behavior is consistent between parallel and non-parallel usage"""
project = poetry.factory.Factory().create_poetry(None) poetry = Factory().create_poetry(None)
packages: tox_poetry_installer.hooks._tox_on_install_helpers.PackageMap = { packages: utilities.PackageMap = {
item.name: item for item in project.locker.locked_repository().packages item.name: item for item in poetry.locker.locked_repository().packages
} }
to_install = [ to_install = [
@ -46,18 +41,14 @@ def test_parallelization(mock_venv, mock_poetry_factory):
packages["attrs"], packages["attrs"],
] ]
venv_sequential = tox.tox_env.python.virtual_env.runner.VirtualEnvRunner() venv_sequential = tox.venv.VirtualEnv()
start_sequential = time.time() start_sequential = time.time()
tox_poetry_installer.hooks._tox_on_install_helpers.install_package( installer.install(poetry, venv_sequential, to_install, 0)
project, venv_sequential, to_install, 0
)
sequential = time.time() - start_sequential sequential = time.time() - start_sequential
venv_parallel = tox.tox_env.python.virtual_env.runner.VirtualEnvRunner() venv_parallel = tox.venv.VirtualEnv()
start_parallel = time.time() start_parallel = time.time()
tox_poetry_installer.hooks._tox_on_install_helpers.install_package( installer.install(poetry, venv_parallel, to_install, 5)
project, venv_parallel, to_install, 5
)
parallel = time.time() - start_parallel parallel = time.time() - start_parallel
# The mock delay during package install is static (one second) so these values should all # The mock delay during package install is static (one second) so these values should all
@ -66,32 +57,3 @@ def test_parallelization(mock_venv, mock_poetry_factory):
assert round(parallel * 5) == round(sequential) assert round(parallel * 5) == round(sequential)
assert round(sequential) == len(set(to_install)) assert round(sequential) == len(set(to_install))
assert round(parallel * 5) == len(set(to_install)) assert round(parallel * 5) == len(set(to_install))
@pytest.mark.parametrize("num_threads", (0, 8))
def test_propagates_exceptions_during_installation(
mock_venv, mock_poetry_factory, num_threads
):
"""Assert that an exception which occurs during installation is properly raised.
Regression test for https://github.com/enpaul/tox-poetry-installer/issues/86
"""
project = poetry.factory.Factory().create_poetry(None)
packages: tox_poetry_installer.hooks._tox_on_install_helpers.PackageMap = {
item.name: item for item in project.locker.locked_repository().packages
}
to_install = [packages["toml"]]
venv = tox.tox_env.python.virtual_env.runner.VirtualEnvRunner()
fake_exception = ValueError("my testing exception")
with mock.patch.object(
poetry.installation.executor,
"Executor",
**{"return_value.execute.side_effect": fake_exception},
):
with pytest.raises(ValueError) as exc_info:
tox_poetry_installer.hooks._tox_on_install_helpers.install_package(
project, venv, to_install, num_threads
)
assert exc_info.value is fake_exception

View File

@ -3,7 +3,6 @@
The next best thing to having one source of truth is having a way to ensure all of your The next best thing to having one source of truth is having a way to ensure all of your
sources of truth agree with each other. sources of truth agree with each other.
""" """
from pathlib import Path from pathlib import Path
import toml import toml

View File

@ -1,23 +1,33 @@
# pylint: disable=missing-module-docstring,redefined-outer-name,unused-argument,unused-import,protected-access # pylint: disable=missing-module-docstring, redefined-outer-name, unused-argument, wrong-import-order, unused-import
import poetry.factory import poetry.factory
import poetry.utils.env import poetry.utils.env
import pytest import pytest
from poetry.puzzle.provider import Provider
import tox_poetry_installer.hooks._tox_on_install_helpers
from tox_poetry_installer import exceptions
from .fixtures import mock_poetry_factory from .fixtures import mock_poetry_factory
from .fixtures import mock_venv from .fixtures import mock_venv
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer import utilities
def test_exclude_unsafe():
"""Test that the unsafe packages are properly excluded
Also ensure that the internal constant matches the value from Poetry
"""
assert Provider.UNSAFE_PACKAGES == constants.UNSAFE_PACKAGES
for dep in constants.UNSAFE_PACKAGES:
assert not utilities.identify_transients(dep, {}, None)
def test_allow_missing(): def test_allow_missing():
"""Test that the ``allow_missing`` parameter works as expected""" """Test that the ``allow_missing`` parameter works as expected"""
with pytest.raises(exceptions.LockedDepNotFoundError): with pytest.raises(exceptions.LockedDepNotFoundError):
tox_poetry_installer.hooks._tox_on_install_helpers.identify_transients( utilities.identify_transients("luke-skywalker", {}, None)
"luke-skywalker", {}, None
)
assert not tox_poetry_installer.hooks._tox_on_install_helpers.identify_transients( assert not utilities.identify_transients(
"darth-vader", {}, None, allow_missing=["darth-vader"] "darth-vader", {}, None, allow_missing=["darth-vader"]
) )
@ -37,9 +47,7 @@ def test_exclude_pep508():
"=>foo", "=>foo",
]: ]:
with pytest.raises(exceptions.LockedDepVersionConflictError): with pytest.raises(exceptions.LockedDepVersionConflictError):
tox_poetry_installer.hooks._tox_on_install_helpers.identify_transients( utilities.identify_transients(version, {}, None)
version, {}, None
)
def test_functional(mock_poetry_factory, mock_venv): def test_functional(mock_poetry_factory, mock_venv):
@ -48,10 +56,8 @@ def test_functional(mock_poetry_factory, mock_venv):
Trivially test that it resolves dependencies properly and that the parent package Trivially test that it resolves dependencies properly and that the parent package
is always the last in the returned list. is always the last in the returned list.
""" """
project = poetry.factory.Factory().create_poetry(None) pypoetry = poetry.factory.Factory().create_poetry(None)
packages = tox_poetry_installer.hooks._tox_on_install_helpers.build_package_map( packages = utilities.build_package_map(pypoetry)
project
)
venv = poetry.utils.env.VirtualEnv() # pylint: disable=no-value-for-parameter venv = poetry.utils.env.VirtualEnv() # pylint: disable=no-value-for-parameter
requests_requires = [ requests_requires = [
@ -62,18 +68,12 @@ def test_functional(mock_poetry_factory, mock_venv):
packages["requests"][0], packages["requests"][0],
] ]
transients = tox_poetry_installer.hooks._tox_on_install_helpers.identify_transients( transients = utilities.identify_transients("requests", packages, venv)
"requests", packages, venv
)
assert all((item in requests_requires) for item in transients) assert all((item in requests_requires) for item in transients)
assert all((item in transients) for item in requests_requires) assert all((item in transients) for item in requests_requires)
for package in [packages["requests"][0], packages["tox"][0], packages["flask"][0]]: for package in [packages["requests"][0], packages["tox"][0], packages["flask"][0]]:
transients = ( transients = utilities.identify_transients(package.name, packages, venv)
tox_poetry_installer.hooks._tox_on_install_helpers.identify_transients(
package.name, packages, venv
)
)
assert transients[-1] == package assert transients[-1] == package
assert len(transients) == len(set(transients)) assert len(transients) == len(set(transients))

22
tox.ini
View File

@ -1,11 +1,14 @@
[tox] [tox]
envlist = py3{8,9,10,11,12} static, static-tests, security envlist = py37, py38, py39, py310, static, static-tests, security
isolated_build = true
skip_missing_interpreters = true skip_missing_interpreters = true
[testenv] [testenv]
description = Run the tests description = Run the tests
require_locked_deps = true require_locked_deps = true
require_poetry = true require_poetry = true
extras =
poetry
locked_deps = locked_deps =
pytest pytest
pytest-cov pytest-cov
@ -18,16 +21,16 @@ commands =
[testenv:static] [testenv:static]
description = Static formatting and quality enforcement description = Static formatting and quality enforcement
basepython = py310 basepython = python3.10
platform = linux platform = linux
ignore_errors = true ignore_errors = true
locked_deps = locked_deps =
black black
blacken-docs blacken-docs
isort
mdformat mdformat
mdformat-gfm mdformat-gfm
mypy mypy
reorder-python-imports
pre-commit pre-commit
pre-commit-hooks pre-commit-hooks
pylint pylint
@ -43,14 +46,13 @@ commands =
[testenv:static-tests] [testenv:static-tests]
description = Static formatting and quality enforcement for the tests description = Static formatting and quality enforcement for the tests
basepython = py310 basepython = python3.10
platform = linux platform = linux
ignore_errors = true ignore_errors = true
locked_deps = locked_deps =
pylint pylint
pytest pytest
mypy mypy
toml
types-toml types-toml
commands = commands =
pylint {toxinidir}/tests/ \ pylint {toxinidir}/tests/ \
@ -61,7 +63,7 @@ commands =
[testenv:security] [testenv:security]
description = Security checks description = Security checks
basepython = py310 basepython = python3.10
platform = linux platform = linux
ignore_errors = true ignore_errors = true
skip_install = true skip_install = true
@ -77,3 +79,11 @@ commands =
--recursive \ --recursive \
--quiet \ --quiet \
--skip B101 --skip B101
poetry export \
--format requirements.txt \
--output {envtmpdir}/requirements.txt \
--without-hashes \
--dev
safety check \
--file {envtmpdir}/requirements.txt \
--json

View File

@ -1,7 +1,7 @@
# pylint: disable=missing-docstring # pylint: disable=missing-docstring
__title__ = "tox-poetry-installer" __title__ = "tox-poetry-installer"
__summary__ = "A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile" __summary__ = "A plugin for Tox that lets you install test environment dependencies from the Poetry lockfile"
__version__ = "1.0.0b1" __version__ = "0.9.0"
__url__ = "https://github.com/enpaul/tox-poetry-installer/" __url__ = "https://github.com/enpaul/tox-poetry-installer/"
__license__ = "MIT" __license__ = "MIT"
__authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"] __authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]

View File

@ -1,4 +1,3 @@
# pylint: disable=missing-docstring # pylint: disable=missing-docstring
from tox_poetry_installer.hooks import tox_add_env_config from tox_poetry_installer.hooks import tox_addoption
from tox_poetry_installer.hooks import tox_add_option from tox_poetry_installer.hooks import tox_testenv_install_deps
from tox_poetry_installer.hooks import tox_on_install

View File

@ -0,0 +1,39 @@
"""You've heard of vendoirization, now get ready for internal namespace shadowing
Poetry is an optional dependency of this package explicitly to support the use case of having the
plugin and the `poetry` package installed to the same python environment; this is most common in
containers and/or CI. In this case there are two potential problems that can arise in this case:
* The installation of the plugin overwrites the installed version of Poetry resulting in
compatibility issues.
* Running `poetry install --no-dev`, when this plugin is in the dev-deps, results in poetry being
uninstalled from the environment.
To support these edge cases, and more broadly to support not messing with a system package manager,
the `poetry` package dependency is listed as optional dependency. This allows the plugin to be
installed to the same environment as Poetry and import that same Poetry installation here.
However, simply importing Poetry on the assumption that it is installed breaks another valid use
case: having this plugin installed alongside Tox when not using a Poetry-based project. To account
for this the imports in this module are isolated and the resultant import error that would result
is converted to an internal error that can be caught by callers. Rather than importing this module
at the module scope it is imported into function scope wherever Poetry components are needed. This
moves import errors from load time to runtime which allows the plugin to be skipped if Poetry isn't
installed and/or a more helpful error be raised within the Tox framework.
"""
# pylint: disable=unused-import
import sys
from tox_poetry_installer import exceptions
try:
from cleo.io.null_io import NullIO
from poetry.factory import Factory
from poetry.installation.pip_installer import PipInstaller
from poetry.poetry import Poetry
from poetry.utils.env import VirtualEnv
except ImportError:
raise exceptions.PoetryNotInstalledError(
f"No version of Poetry could be imported under the current environment for '{sys.executable}'"
) from None

View File

@ -5,7 +5,7 @@ in this module.
All constants should be type hinted. All constants should be type hinted.
""" """
from typing import Set
from typing import Tuple from typing import Tuple
from tox_poetry_installer import __about__ from tox_poetry_installer import __about__
@ -19,5 +19,9 @@ PEP508_VERSION_DELIMITERS: Tuple[str, ...] = ("~=", "==", "!=", ">", "<")
# console output. # console output.
REPORTER_PREFIX: str = f"{__about__.__title__}:" REPORTER_PREFIX: str = f"{__about__.__title__}:"
# Internal list of packages that poetry has deemed unsafe and are excluded from the lockfile
# TODO: This functionality is no longer needed, should remove in a future update.
UNSAFE_PACKAGES: Set[str] = set()
# Number of threads to use for installing dependencies by default # Number of threads to use for installing dependencies by default
DEFAULT_INSTALL_THREADS: int = 10 DEFAULT_INSTALL_THREADS: int = 10

View File

@ -11,7 +11,7 @@ All exceptions should inherit from the common base exception :exc:`ToxPoetryInst
+-- LockedDepNotFoundError +-- LockedDepNotFoundError
+-- ExtraNotFoundError +-- ExtraNotFoundError
+-- LockedDepsRequiredError +-- LockedDepsRequiredError
+-- LockfileParsingError +-- RequiresUnsafeDepError
""" """
@ -44,5 +44,5 @@ class LockedDepsRequiredError(ToxPoetryInstallerException):
"""Environment cannot specify unlocked dependencies when locked dependencies are required""" """Environment cannot specify unlocked dependencies when locked dependencies are required"""
class LockfileParsingError(ToxPoetryInstallerException): class RequiresUnsafeDepError(ToxPoetryInstallerException):
"""Failed to load or parse the Poetry lockfile""" """Package under test depends on an unsafe dependency and cannot be installed"""

View File

@ -0,0 +1,259 @@
"""Main hook definition module
All implementations of tox hooks are defined here, as well as any single-use helper functions
specifically related to implementing the hooks (to keep the size/readability of the hook functions
themselves manageable).
"""
from typing import Optional
import tox
from tox.action import Action as ToxAction
from tox.config import Parser as ToxParser
from tox.venv import VirtualEnv as ToxVirtualEnv
from tox_poetry_installer import __about__
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer import installer
from tox_poetry_installer import logger
from tox_poetry_installer import utilities
def _postprocess_install_project_deps(
testenv_config, value: Optional[str] # pylint: disable=unused-argument
) -> Optional[bool]:
"""An awful hack to patch on three-state boolean logic to a config parameter
.. warning: This logic should 100% be removed in the next feature release. It's here to work
around a bad design for now but should not persist.
The bug filed in `#61`_ is caused by a combination of poor design and attempted cleverness. The
name of the ``install_project_deps`` config option implies that it has ultimate control over
whether the project dependencies are installed to the testenv, but this is not actually correct.
What it actually allows the user to do is force the project dependencies to not be installed to
an environment that would otherwise install them. This was intended behavior, however the
intention was wrong.
.. _`#61`: https://github.com/enpaul/tox-poetry-installer/issues/61
In an effort to be clever the plugin automatically skips installing project dependencies when
the project package is not installed to the testenv (``skip_install = true``) or if packaging
as a whole is disabled (``skipsdist = true``). The intention of this behavior is to install only
the expected dependencies to a testenv and no more. However, this conflicts with the
``install_project_deps`` config option, which cannot override this behavior because it defaults
to ``True``. In effect, ``install_project_deps = true`` in fact means "automatically
determine whether to install project dependencies" and ``install_project_deps = false`` means
"never install the project dependencies". This is not ideal and unintuitive.
To avoid having to make a breaking change this workaround has been added to support three-state
logic between ``True``, ``False``, and ``None``. The ``install_project_deps`` option is now
parsed by Tox as a string with a default value of ``None``. If the value is not ``None`` then
this post processing function will try to convert it to a boolean the same way that Tox's
`SectionReader.getbool()`_ method does, raising an error to mimic the default behavior if it
can't.
.. _`SectionReader.getbool()`: https://github.com/tox-dev/tox/blob/f8459218ee5ab5753321b3eb989b7beee5b391ad/src/tox/config/__init__.py#L1724
The three states for the ``install_project_deps`` setting are:
* ``None`` - User did not configure the setting, package dependency installation is
determined automatically
* ``True`` - User configured the setting to ``True``, package dependencies will be installed
* ``False`` - User configured the setting to ``False``, package dependencies will not be
installed
This config option should be deprecated with the 1.0.0 release and instead an option like
``always_install_project_deps`` should be added which overrides the default determination and
just installs the project dependencies. The counterpart (``never_install_project_deps``)
shouldn't be needed, since I don't think there's a real use case for that.
"""
if value is None:
return value
if value.lower() == "true":
return True
if value.lower() == "false":
return False
raise tox.exception.ConfigError(
f"install_project_deps: boolean value '{value}' needs to be 'True' or 'False'"
)
@tox.hookimpl
def tox_addoption(parser: ToxParser):
"""Add required configuration options to the tox INI file
Adds the ``require_locked_deps`` configuration option to the venv to check whether all
dependencies should be treated as locked or not.
"""
parser.add_argument(
"--require-poetry",
action="store_true",
dest="require_poetry",
help="(deprecated) Trigger a failure if Poetry is not available to Tox",
)
parser.add_argument(
"--parallelize-locked-install",
type=int,
dest="parallelize_locked_install",
default=None,
help="(deprecated) Number of worker threads to use for installing dependencies from the Poetry lockfile in parallel",
)
parser.add_argument(
"--parallel-install-threads",
type=int,
dest="parallel_install_threads",
default=constants.DEFAULT_INSTALL_THREADS,
help="Number of locked dependencies to install simultaneously; set to 0 to disable parallel installation",
)
parser.add_testenv_attribute(
name="install_dev_deps",
type="bool",
default=False,
help="Automatically install all Poetry development dependencies to the environment",
)
parser.add_testenv_attribute(
name="install_project_deps",
type="string",
default=None,
help="Automatically install all Poetry primary dependencies to the environment",
postprocess=_postprocess_install_project_deps,
)
parser.add_testenv_attribute(
name="require_locked_deps",
type="bool",
default=False,
help="Require all dependencies in the environment be installed using the Poetry lockfile",
)
parser.add_testenv_attribute(
name="require_poetry",
type="bool",
default=False,
help="Trigger a failure if Poetry is not available to Tox",
)
parser.add_testenv_attribute(
name="locked_deps",
type="line-list",
help="List of locked dependencies to install to the environment using the Poetry lockfile",
)
@tox.hookimpl
def tox_testenv_install_deps(venv: ToxVirtualEnv, action: ToxAction) -> Optional[bool]:
"""Install the dependencies for the current environment
Loads the local Poetry environment and the corresponding lockfile then pulls the dependencies
specified by the Tox environment. Finally these dependencies are installed into the Tox
environment using the Poetry ``PipInstaller`` backend.
:param venv: Tox virtual environment object with configuration for the local Tox environment.
:param action: Tox action object
"""
try:
poetry = utilities.check_preconditions(venv, action)
except exceptions.SkipEnvironment as err:
if isinstance(err, exceptions.PoetryNotInstalledError) and (
venv.envconfig.config.option.require_poetry or venv.envconfig.require_poetry
):
venv.status = err.__class__.__name__
logger.error(str(err))
return False
logger.info(str(err))
return None
logger.info(f"Loaded project pyproject.toml from {poetry.file}")
virtualenv = utilities.convert_virtualenv(venv)
if not poetry.locker.is_fresh():
logger.warning(
f"The Poetry lock file is not up to date with the latest changes in {poetry.file}"
)
try:
if venv.envconfig.require_locked_deps and venv.envconfig.deps:
raise exceptions.LockedDepsRequiredError(
f"Unlocked dependencies '{venv.envconfig.deps}' specified for environment '{venv.name}' which requires locked dependencies"
)
packages = utilities.build_package_map(poetry)
if venv.envconfig.install_dev_deps:
dev_deps = utilities.find_dev_deps(packages, virtualenv, poetry)
logger.info(
f"Identified {len(dev_deps)} development dependencies to install to env"
)
else:
dev_deps = []
logger.info("Env does not install development dependencies, skipping")
env_deps = utilities.find_additional_deps(
packages, virtualenv, poetry, venv.envconfig.locked_deps
)
logger.info(
f"Identified {len(env_deps)} environment dependencies to install to env"
)
install_project_deps = (
venv.envconfig.install_project_deps
if venv.envconfig.install_project_deps is not None
else (
not venv.envconfig.skip_install and not venv.envconfig.config.skipsdist
)
)
if install_project_deps:
project_deps = utilities.find_project_deps(
packages, virtualenv, poetry, venv.envconfig.extras
)
logger.info(
f"Identified {len(project_deps)} project dependencies to install to env"
)
else:
project_deps = []
logger.info("Env does not install project package dependencies, skipping")
except exceptions.ToxPoetryInstallerException as err:
venv.status = err.__class__.__name__
logger.error(str(err))
return False
except Exception as err:
venv.status = "InternalError"
logger.error(f"Internal plugin error: {err}")
raise err
dependencies = utilities.dedupe_packages(dev_deps + env_deps + project_deps)
if (
venv.envconfig.config.option.parallel_install_threads
!= constants.DEFAULT_INSTALL_THREADS
):
parallel_threads = venv.envconfig.config.option.parallel_install_threads
else:
parallel_threads = (
venv.envconfig.config.option.parallelize_locked_install
if venv.envconfig.config.option.parallelize_locked_install is not None
else constants.DEFAULT_INSTALL_THREADS
)
log_parallel = f" (using {parallel_threads} threads)" if parallel_threads else ""
action.setactivity(
__about__.__title__,
f"Installing {len(dependencies)} dependencies from Poetry lock file{log_parallel}",
)
installer.install(
poetry,
venv,
dependencies,
parallel_threads,
)
return venv.envconfig.require_locked_deps or None

View File

@ -1,4 +0,0 @@
# pylint: disable=missing-module-docstring
from tox_poetry_installer.hooks.tox_add_env_config import tox_add_env_config
from tox_poetry_installer.hooks.tox_add_option import tox_add_option
from tox_poetry_installer.hooks.tox_on_install import tox_on_install

View File

@ -1,366 +0,0 @@
"""Helper functions for the :func:`tox_on_install` hook"""
import collections
import concurrent.futures
import contextlib
import datetime
import pathlib
from typing import Collection
from typing import Dict
from typing import List
from typing import Sequence
from typing import Set
import cleo.io.null_io
import packaging.utils
import poetry.config.config
import poetry.core.packages.dependency
import poetry.core.packages.package
import poetry.factory
import poetry.installation.executor
import poetry.installation.operations.install
import poetry.poetry
import poetry.utils.env
import tox.tox_env.api
import tox.tox_env.package
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer import logger
PackageMap = Dict[str, List[poetry.core.packages.package.Package]]
def check_preconditions(venv: tox.tox_env.api.ToxEnv) -> poetry.poetry.Poetry:
"""Check that the local project environment meets expectations"""
# Skip running the plugin for the provisioning environment. The provisioned environment,
# for alternative Tox versions and/or the ``requires`` meta dependencies is specially
# handled by Tox and is out of scope for this plugin. Since one of the ways to install this
# plugin in the first place is via the Tox provisioning environment, it quickly becomes a
# chicken-and-egg problem.
if isinstance(venv, tox.tox_env.package.PackageToxEnv):
raise exceptions.SkipEnvironment(f"Skipping Tox provisioning env '{venv.name}'")
try:
return poetry.factory.Factory().create_poetry(venv.core["tox_root"])
# Support running the plugin when the current tox project does not use Poetry for its
# environment/dependency management.
#
# ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case,
# is) raised in many different places for different purposes.
except RuntimeError as err:
raise exceptions.SkipEnvironment(
f"Skipping installation of locked dependencies due to a Poetry error: {err}"
) from None
def identify_transients(
dep_name: str,
packages: PackageMap,
venv: poetry.utils.env.VirtualEnv,
allow_missing: Sequence[str] = (),
) -> List[poetry.core.packages.package.Package]:
"""Using a pool of packages, identify all transient dependencies of a given package name
:param dep_name: Either the Poetry dependency or the dependency's bare package name to recursively
identify the transient dependencies of
:param packages: All packages from the lockfile to use for identifying dependency relationships.
:param venv: Poetry virtual environment to use for package compatibility checks
:param allow_missing: Sequence of package names to allow to be missing from the lockfile. Any
packages that are not found in the lockfile but their name appears in this
list will be silently skipped from installation.
:returns: List of packages that need to be installed for the requested dependency.
.. note:: The package corresponding to the dependency specified by the ``dep`` parameter will
be included in the returned list of packages.
"""
searched: Set[str] = set()
def _transients(
transient: poetry.core.packages.dependency.Dependency,
) -> List[poetry.core.packages.package.Package]:
searched.add(transient.name)
results: List[poetry.core.packages.package.Package] = []
for option in packages[transient.name]:
if venv.is_valid_for_marker(option.to_dependency().marker):
for requirement in option.requires:
if requirement.name not in searched:
results += _transients(requirement)
logger.debug(f"Including {option} for installation")
results.append(option)
break
else:
logger.debug(
f"Skipping {transient.name}: target python version is {'.'.join([str(item) for item in venv.get_version_info()])} but package requires {transient.marker}"
)
return results
try:
for option in packages[dep_name]:
if venv.is_valid_for_marker(option.to_dependency().marker):
dep = option.to_dependency()
break
else:
logger.warning(
f"Skipping {dep_name}: no locked version found compatible with target python version {'.'.join([str(item) for item in venv.get_version_info()])}"
)
return []
return _transients(dep)
except KeyError as err:
missing = err.args[0]
if missing in allow_missing:
logger.debug(f"Skipping {missing}: package is allowed to be unlocked")
return []
if any(
delimiter in missing for delimiter in constants.PEP508_VERSION_DELIMITERS
):
raise exceptions.LockedDepVersionConflictError(
f"Locked dependency '{missing}' cannot include version specifier"
) from None
raise exceptions.LockedDepNotFoundError(
f"No version of locked dependency '{missing}' found in the project lockfile"
) from None
def find_project_deps(
packages: PackageMap,
venv: poetry.utils.env.VirtualEnv,
project: poetry.poetry.Poetry,
extras: Sequence[str] = (),
) -> List[poetry.core.packages.package.Package]:
"""Find the root project dependencies
Recursively identify the dependencies of the root project package
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param project: Poetry object for the current project
:param extras: Sequence of extra names to include the dependencies of
"""
required_dep_names = [
item.name for item in project.package.requires if not item.is_optional()
]
extra_dep_names: List[str] = []
for extra in extras:
logger.info(f"Processing project extra '{extra}'")
try:
extra_dep_names += [
item.name
for item in project.package.extras[
packaging.utils.NormalizedName(extra)
]
]
except KeyError:
raise exceptions.ExtraNotFoundError(
f"Environment specifies project extra '{extra}' which was not found in the lockfile"
) from None
dependencies: List[poetry.core.packages.package.Package] = []
for dep_name in required_dep_names + extra_dep_names:
dependencies += identify_transients(
dep_name.lower(), packages, venv, allow_missing=[project.package.name]
)
return dedupe_packages(dependencies)
def find_additional_deps(
packages: PackageMap,
venv: poetry.utils.env.VirtualEnv,
project: poetry.poetry.Poetry,
dep_names: Sequence[str],
) -> List[poetry.core.packages.package.Package]:
"""Find additional dependencies
Recursively identify the dependencies of an arbitrary list of package names
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param project: Poetry object for the current project
:param dep_names: Sequence of additional dependency names to recursively find the transient
dependencies for
"""
dependencies: List[poetry.core.packages.package.Package] = []
for dep_name in dep_names:
dependencies += identify_transients(
dep_name.lower(), packages, venv, allow_missing=[project.package.name]
)
return dedupe_packages(dependencies)
def find_group_deps(
group: str,
packages: PackageMap,
venv: poetry.utils.env.VirtualEnv,
project: poetry.poetry.Poetry,
) -> List[poetry.core.packages.package.Package]:
"""Find the dependencies belonging to a dependency group
Recursively identify the Poetry dev dependencies
:param group: Name of the dependency group from the project's ``pyproject.toml``
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param project: Poetry object for the current project
"""
return find_additional_deps(
packages,
venv,
project,
# the type ignore here is due to the difficulties around getting nested data
# from the inherrently unstructured toml structure (which necessarily is flexibly
# typed) but in a situation where there is a meta-structure applied to it (i.e. a
# pyproject.toml structure).
project.pyproject.data["tool"]["poetry"] # type: ignore
.get("group", {})
.get(group, {})
.get("dependencies", {})
.keys(),
)
def find_dev_deps(
packages: PackageMap,
venv: poetry.utils.env.VirtualEnv,
project: poetry.poetry.Poetry,
) -> List[poetry.core.packages.package.Package]:
"""Find the dev dependencies
Recursively identify the Poetry dev dependencies
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param project: Poetry object for the current project
"""
dev_group_deps = find_group_deps("dev", packages, venv, project)
# Legacy pyproject.toml poetry format:
legacy_dev_group_deps = find_additional_deps(
packages,
venv,
project,
# the type ignore here is due to the difficulties around getting nested data
# from the inherrently unstructured toml structure (which necessarily is flexibly
# typed) but in a situation where there is a meta-structure applied to it (i.e. a
# pyproject.toml structure).
project.pyproject.data["tool"]["poetry"].get("dev-dependencies", {}).keys(), # type: ignore
)
# Poetry 1.2 unions these two toml sections.
return dedupe_packages(dev_group_deps + legacy_dev_group_deps)
@contextlib.contextmanager
def _optional_parallelize(parallels: int):
"""A bit of cheat, really
A context manager that exposes a common interface for the caller that optionally
enables/disables the usage of the parallel thread pooler depending on the value of
the ``parallels`` parameter.
"""
if parallels > 0:
with concurrent.futures.ThreadPoolExecutor(max_workers=parallels) as executor:
yield executor.submit
else:
yield lambda func, arg: func(arg)
def install_package(
project: poetry.poetry.Poetry,
venv: tox.tox_env.api.ToxEnv,
packages: Collection[poetry.core.packages.package.Package],
parallels: int = 0,
):
"""Install a bunch of packages to a virtualenv
:param project: Poetry object the packages were sourced from
:param venv: Tox virtual environment to install the packages to
:param packages: List of packages to install to the virtual environment
:param parallels: Number of parallel processes to use for installing dependency packages, or
``None`` to disable parallelization.
"""
logger.info(f"Installing {len(packages)} packages to environment at {venv.env_dir}")
install_executor = poetry.installation.executor.Executor(
env=convert_virtualenv(venv),
io=cleo.io.null_io.NullIO(),
pool=project.pool,
config=poetry.config.config.Config(),
)
installed: Set[poetry.core.packages.package.Package] = set()
def logged_install(dependency: poetry.core.packages.package.Package) -> None:
start = datetime.datetime.now()
logger.debug(f"Installing {dependency}")
install_executor.execute(
[poetry.installation.operations.install.Install(package=dependency)]
)
end = datetime.datetime.now()
logger.debug(f"Finished installing {dependency} in {end - start}")
with _optional_parallelize(parallels) as executor:
futures = []
for dependency in packages:
if dependency not in installed:
installed.add(dependency)
logger.debug(f"Queuing {dependency}")
future = executor(logged_install, dependency)
if future is not None:
futures.append(future)
else:
logger.debug(f"Skipping {dependency}, already installed")
logger.debug("Waiting for installs to finish...")
for future in concurrent.futures.as_completed(futures):
# Don't actually care about the return value, just waiting on the
# future to ensure any exceptions that were raised in the called
# function are propagated.
future.result()
def dedupe_packages(
packages: Sequence[poetry.core.packages.package.Package],
) -> List[poetry.core.packages.package.Package]:
"""Deduplicates a sequence of Packages while preserving ordering
Adapted from StackOverflow: https://stackoverflow.com/a/480227
"""
seen: Set[poetry.core.packages.package.Package] = set()
# Make this faster, avoid method lookup below
seen_add = seen.add
return [item for item in packages if not (item in seen or seen_add(item))]
def convert_virtualenv(venv: tox.tox_env.api.ToxEnv) -> poetry.utils.env.VirtualEnv:
"""Convert a Tox venv to a Poetry venv
:param venv: Tox ``VirtualEnv`` object representing a tox virtual environment
:returns: Poetry ``VirtualEnv`` object representing a poetry virtual environment
"""
return poetry.utils.env.VirtualEnv(path=pathlib.Path(venv.env_dir))
def build_package_map(project: poetry.poetry.Poetry) -> PackageMap:
"""Build the mapping of package names to objects
:param project: Populated poetry object to load locked packages from
:returns: Mapping of package names to Poetry package objects
"""
packages = collections.defaultdict(list)
for package in project.locker.locked_repository().packages:
packages[str(package.name)].append(package)
return packages

View File

@ -1,47 +0,0 @@
"""Add required env configuration options to the tox INI file"""
from typing import List
import tox.config.sets
import tox.plugin
# pylint: disable=missing-function-docstring
@tox.plugin.impl
def tox_add_env_config(
env_conf: tox.config.sets.EnvConfigSet,
):
env_conf.add_config(
"poetry_dep_groups",
of_type=List[str],
default=[],
desc="List of Poetry dependency groups to install to the environment",
)
env_conf.add_config(
"install_project_deps",
of_type=bool,
default=True,
desc="Automatically install all Poetry primary dependencies to the environment",
)
env_conf.add_config(
"require_locked_deps",
of_type=bool,
default=False,
desc="Require all dependencies in the environment be installed using the Poetry lockfile",
)
env_conf.add_config(
"require_poetry",
of_type=bool,
default=False,
desc="Trigger a failure if Poetry is not available to Tox",
)
env_conf.add_config(
"locked_deps",
of_type=List[str],
default=[],
desc="List of locked dependencies to install to the environment using the Poetry lockfile",
)

View File

@ -1,18 +0,0 @@
"""Add additional command line arguments to tox to configure plugin behavior"""
import tox.config.cli.parser
import tox.plugin
from tox_poetry_installer import constants
# pylint: disable=missing-function-docstring
@tox.plugin.impl
def tox_add_option(parser: tox.config.cli.parser.ToxParser):
parser.add_argument(
"--parallel-install-threads",
type=int,
dest="parallel_install_threads",
default=constants.DEFAULT_INSTALL_THREADS,
help="Number of locked dependencies to install simultaneously; set to 0 to disable parallel installation",
)

View File

@ -1,114 +0,0 @@
"""Install the dependencies for the current environment
Loads the local Poetry environment and the corresponding lockfile then pulls the dependencies
specified by the Tox environment. Finally these dependencies are installed into the Tox
environment using the Poetry ``PipInstaller`` backend.
"""
import itertools
import tox.plugin
import tox.tox_env.api
from tox_poetry_installer import exceptions
from tox_poetry_installer import logger
from tox_poetry_installer.hooks._tox_on_install_helpers import build_package_map
from tox_poetry_installer.hooks._tox_on_install_helpers import check_preconditions
from tox_poetry_installer.hooks._tox_on_install_helpers import convert_virtualenv
from tox_poetry_installer.hooks._tox_on_install_helpers import dedupe_packages
from tox_poetry_installer.hooks._tox_on_install_helpers import find_additional_deps
from tox_poetry_installer.hooks._tox_on_install_helpers import find_group_deps
from tox_poetry_installer.hooks._tox_on_install_helpers import find_project_deps
from tox_poetry_installer.hooks._tox_on_install_helpers import install_package
# pylint: disable=missing-function-docstring,unused-argument
@tox.plugin.impl
def tox_on_install(tox_env: tox.tox_env.api.ToxEnv, *args) -> None:
try:
poetry = check_preconditions(tox_env)
except exceptions.SkipEnvironment as err:
if (
isinstance(err, exceptions.PoetryNotInstalledError)
and tox_env.conf["require_poetry"]
):
logger.error(str(err))
raise err
logger.info(str(err))
return
logger.info(f"Loaded project pyproject.toml from {poetry.file}")
virtualenv = convert_virtualenv(tox_env)
try:
if not poetry.locker.is_fresh():
logger.warning(
f"The Poetry lock file is not up to date with the latest changes in {poetry.file}"
)
except FileNotFoundError as err:
logger.error(f"Could not parse lockfile: {err}")
raise exceptions.LockfileParsingError(
f"Could not parse lockfile: {err}"
) from err
try:
if tox_env.conf["require_locked_deps"] and tox_env.conf["deps"].lines():
raise exceptions.LockedDepsRequiredError(
f"Unlocked dependencies '{tox_env.conf['deps']}' specified for environment '{tox_env.name}' which requires locked dependencies"
)
packages = build_package_map(poetry)
group_deps = dedupe_packages(
list(
itertools.chain(
*[
find_group_deps(group, packages, virtualenv, poetry)
for group in tox_env.conf["poetry_dep_groups"]
]
)
)
)
logger.info(
f"Identified {len(group_deps)} group dependencies to install to env"
)
env_deps = find_additional_deps(
packages, virtualenv, poetry, tox_env.conf["locked_deps"]
)
logger.info(
f"Identified {len(env_deps)} environment dependencies to install to env"
)
# extras are not set in a testenv if skip_install=true
try:
extras = tox_env.conf["extras"]
except KeyError:
extras = []
if tox_env.conf["install_project_deps"]:
project_deps = find_project_deps(packages, virtualenv, poetry, extras)
logger.info(
f"Identified {len(project_deps)} project dependencies to install to env"
)
else:
project_deps = []
logger.info("Env does not install project package dependencies, skipping")
except exceptions.ToxPoetryInstallerException as err:
logger.error(str(err))
raise err
except Exception as err:
logger.error(f"Internal plugin error: {err}")
raise err
dependencies = dedupe_packages(group_deps + env_deps + project_deps)
logger.info(f"Installing {len(dependencies)} dependencies from Poetry lock file")
install_package(
poetry,
tox_env,
dependencies,
tox_env.options.parallel_install_threads,
)

View File

@ -0,0 +1,81 @@
"""Funcationality for performing virtualenv installation"""
# Silence this one globally to support the internal function imports for the proxied poetry module.
# See the docstring in 'tox_poetry_installer._poetry' for more context.
# pylint: disable=import-outside-toplevel
import concurrent.futures
import contextlib
import typing
from datetime import datetime
from typing import Collection
from typing import Set
from poetry.core.packages.package import Package as PoetryPackage
from tox.venv import VirtualEnv as ToxVirtualEnv
from tox_poetry_installer import logger
from tox_poetry_installer import utilities
if typing.TYPE_CHECKING:
from tox_poetry_installer import _poetry
def install(
poetry: "_poetry.Poetry",
venv: ToxVirtualEnv,
packages: Collection[PoetryPackage],
parallels: int = 0,
):
"""Install a bunch of packages to a virtualenv
:param poetry: Poetry object the packages were sourced from
:param venv: Tox virtual environment to install the packages to
:param packages: List of packages to install to the virtual environment
:param parallels: Number of parallel processes to use for installing dependency packages, or
``None`` to disable parallelization.
"""
from tox_poetry_installer import _poetry
logger.info(
f"Installing {len(packages)} packages to environment at {venv.envconfig.envdir}"
)
pip = _poetry.PipInstaller(
env=utilities.convert_virtualenv(venv),
io=_poetry.NullIO(),
pool=poetry.pool,
)
installed: Set[PoetryPackage] = set()
def logged_install(dependency: PoetryPackage) -> None:
start = datetime.now()
logger.debug(f"Installing {dependency}")
pip.install(dependency)
end = datetime.now()
logger.debug(f"Finished installing {dependency} in {end - start}")
@contextlib.contextmanager
def _optional_parallelize():
"""A bit of cheat, really
A context manager that exposes a common interface for the caller that optionally
enables/disables the usage of the parallel thread pooler depending on the value of
the ``parallels`` parameter.
"""
if parallels > 0:
with concurrent.futures.ThreadPoolExecutor(
max_workers=parallels
) as executor:
yield executor.submit
else:
yield lambda func, arg: func(arg)
with _optional_parallelize() as executor:
for dependency in packages:
if dependency not in installed:
installed.add(dependency)
logger.debug(f"Queuing {dependency}")
executor(logged_install, dependency)
else:
logger.debug(f"Skipping {dependency}, already installed")
logger.debug("Waiting for installs to finish...")

View File

@ -4,27 +4,26 @@ Calling ``tox.reporter.something()`` and having to format a string with the pref
gets really old fast, but more importantly it also makes the flow of the main code gets really old fast, but more importantly it also makes the flow of the main code
more difficult to follow because of the added complexity. more difficult to follow because of the added complexity.
""" """
import tox
import logging
from tox_poetry_installer import constants from tox_poetry_installer import constants
def error(message: str): def error(message: str):
"""Wrapper around :func:`logging.error` that prefixes the reporter prefix onto the message""" """Wrapper around :func:`tox.reporter.error`"""
logging.error(f"{constants.REPORTER_PREFIX} {message}") tox.reporter.error(f"{constants.REPORTER_PREFIX} {message}")
def warning(message: str): def warning(message: str):
"""Wrapper around :func:`logging.warning`""" """Wrapper around :func:`tox.reporter.warning`"""
logging.warning(f"{constants.REPORTER_PREFIX} {message}") tox.reporter.warning(f"{constants.REPORTER_PREFIX} {message}")
def info(message: str): def info(message: str):
"""Wrapper around :func:`logging.info`""" """Wrapper around :func:`tox.reporter.verbosity1`"""
logging.info(f"{constants.REPORTER_PREFIX} {message}") tox.reporter.verbosity1(f"{constants.REPORTER_PREFIX} {message}")
def debug(message: str): def debug(message: str):
"""Wrapper around :func:`logging.debug`""" """Wrapper around :func:`tox.reporter.verbosity2`"""
logging.debug(f"{constants.REPORTER_PREFIX} {message}") tox.reporter.verbosity2(f"{constants.REPORTER_PREFIX} {message}")

View File

@ -0,0 +1,291 @@
"""Helper utility functions, usually bridging Tox and Poetry functionality"""
# Silence this one globally to support the internal function imports for the proxied poetry module.
# See the docstring in 'tox_poetry_installer._poetry' for more context.
# pylint: disable=import-outside-toplevel
import collections
import typing
from pathlib import Path
from typing import Dict
from typing import List
from typing import Sequence
from typing import Set
from poetry.core.packages.dependency import Dependency as PoetryDependency
from poetry.core.packages.package import Package as PoetryPackage
from tox.action import Action as ToxAction
from tox.venv import VirtualEnv as ToxVirtualEnv
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer import logger
if typing.TYPE_CHECKING:
from tox_poetry_installer import _poetry
PackageMap = Dict[str, List[PoetryPackage]]
def check_preconditions(venv: ToxVirtualEnv, action: ToxAction) -> "_poetry.Poetry":
"""Check that the local project environment meets expectations"""
# Skip running the plugin for the provisioning environment. The provisioned environment,
# for alternative Tox versions and/or the ``requires`` meta dependencies is specially
# handled by Tox and is out of scope for this plugin. Since one of the ways to install this
# plugin in the first place is via the Tox provisioning environment, it quickly becomes a
# chicken-and-egg problem.
if action.name == venv.envconfig.config.provision_tox_env:
raise exceptions.SkipEnvironment(
f"Skipping Tox provisioning env '{action.name}'"
)
# Skip running the plugin for the packaging environment. PEP-517 front ends can handle
# that better than we can, so let them do their thing. More to the point: if you're having
# problems in the packaging env that this plugin would solve, god help you.
if action.name == venv.envconfig.config.isolated_build_env:
raise exceptions.SkipEnvironment(
f"Skipping isolated packaging build env '{action.name}'"
)
if venv.envconfig.config.option.require_poetry:
logger.warning(
"DEPRECATION: The '--require-poetry' runtime option is deprecated and will be "
"removed in version 1.0.0. Please update test environments that require Poetry to "
"set the 'require_poetry = true' option in tox.ini"
)
if venv.envconfig.config.option.parallelize_locked_install is not None:
logger.warning(
"DEPRECATION: The '--parallelize-locked-install' option is deprecated and will "
"be removed in version 1.0.0. Please use the '--parallel-install-threads' option."
)
from tox_poetry_installer import _poetry
try:
return _poetry.Factory().create_poetry(venv.envconfig.config.toxinidir)
# Support running the plugin when the current tox project does not use Poetry for its
# environment/dependency management.
#
# ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case,
# is) raised in many different places for different purposes.
except RuntimeError:
raise exceptions.SkipEnvironment(
"Project does not use Poetry for env management, skipping installation of locked dependencies"
) from None
def convert_virtualenv(venv: ToxVirtualEnv) -> "_poetry.VirtualEnv":
"""Convert a Tox venv to a Poetry venv
:param venv: Tox ``VirtualEnv`` object representing a tox virtual environment
:returns: Poetry ``VirtualEnv`` object representing a poetry virtual environment
"""
from tox_poetry_installer import _poetry
return _poetry.VirtualEnv(path=Path(venv.envconfig.envdir))
def build_package_map(poetry: "_poetry.Poetry") -> PackageMap:
"""Build the mapping of package names to objects
:param poetry: Populated poetry object to load locked packages from
:returns: Mapping of package names to Poetry package objects
"""
packages = collections.defaultdict(list)
for package in poetry.locker.locked_repository().packages:
packages[package.name].append(package)
return packages
def identify_transients(
dep_name: str,
packages: PackageMap,
venv: "_poetry.VirtualEnv",
allow_missing: Sequence[str] = (),
) -> List[PoetryPackage]:
"""Using a pool of packages, identify all transient dependencies of a given package name
:param dep_name: Either the Poetry dependency or the dependency's bare package name to recursively
identify the transient dependencies of
:param packages: All packages from the lockfile to use for identifying dependency relationships.
:param venv: Poetry virtual environment to use for package compatibility checks
:param allow_missing: Sequence of package names to allow to be missing from the lockfile. Any
packages that are not found in the lockfile but their name appears in this
list will be silently skipped from installation.
:returns: List of packages that need to be installed for the requested dependency.
.. note:: The package corresponding to the dependency specified by the ``dep`` parameter will
be included in the returned list of packages.
"""
searched: Set[str] = set()
def _transients(transient: PoetryDependency) -> List[PoetryPackage]:
searched.add(transient.name)
results: List[PoetryPackage] = []
for option in packages[transient.name]:
if venv.is_valid_for_marker(option.to_dependency().marker):
for requirement in option.requires:
if requirement.name not in searched:
results += _transients(requirement)
logger.debug(f"Including {option} for installation")
results.append(option)
break
else:
logger.debug(
f"Skipping {transient.name}: target python version is {'.'.join([str(item) for item in venv.get_version_info()])} but package requires {transient.marker}"
)
return results
try:
for option in packages[dep_name]:
if venv.is_valid_for_marker(option.to_dependency().marker):
dep = option.to_dependency()
break
else:
logger.warning(
f"Skipping {dep_name}: no locked version found compatible with target python version {'.'.join([str(item) for item in venv.get_version_info()])}"
)
return []
return _transients(dep)
except KeyError as err:
missing = err.args[0]
if missing in constants.UNSAFE_PACKAGES:
logger.warning(
f"Installing package '{missing}' using Poetry is not supported and will be skipped"
)
logger.debug(f"Skipping {missing}: designated unsafe by Poetry")
return []
if missing in allow_missing:
logger.debug(f"Skipping {missing}: package is allowed to be unlocked")
return []
if any(
delimiter in missing for delimiter in constants.PEP508_VERSION_DELIMITERS
):
raise exceptions.LockedDepVersionConflictError(
f"Locked dependency '{missing}' cannot include version specifier"
) from None
raise exceptions.LockedDepNotFoundError(
f"No version of locked dependency '{missing}' found in the project lockfile"
) from None
def find_project_deps(
packages: PackageMap,
venv: "_poetry.VirtualEnv",
poetry: "_poetry.Poetry",
extras: Sequence[str] = (),
) -> List[PoetryPackage]:
"""Find the root project dependencies
Recursively identify the dependencies of the root project package
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param poetry: Poetry object for the current project
:param extras: Sequence of extra names to include the dependencies of
"""
if any(dep.name in constants.UNSAFE_PACKAGES for dep in poetry.package.requires):
raise exceptions.RequiresUnsafeDepError(
f"Project package requires one or more unsafe dependencies ({', '.join(constants.UNSAFE_PACKAGES)}) which cannot be installed with Poetry"
)
required_dep_names = [
item.name for item in poetry.package.requires if not item.is_optional()
]
extra_dep_names: List[str] = []
for extra in extras:
logger.info(f"Processing project extra '{extra}'")
try:
extra_dep_names += [item.name for item in poetry.package.extras[extra]]
except KeyError:
raise exceptions.ExtraNotFoundError(
f"Environment specifies project extra '{extra}' which was not found in the lockfile"
) from None
dependencies: List[PoetryPackage] = []
for dep_name in required_dep_names + extra_dep_names:
dependencies += identify_transients(
dep_name.lower(), packages, venv, allow_missing=[poetry.package.name]
)
return dedupe_packages(dependencies)
def find_additional_deps(
packages: PackageMap,
venv: "_poetry.VirtualEnv",
poetry: "_poetry.Poetry",
dep_names: Sequence[str],
) -> List[PoetryPackage]:
"""Find additional dependencies
Recursively identify the dependencies of an arbitrary list of package names
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param poetry: Poetry object for the current project
:param dep_names: Sequence of additional dependency names to recursively find the transient
dependencies for
"""
dependencies: List[PoetryPackage] = []
for dep_name in dep_names:
dependencies += identify_transients(
dep_name.lower(), packages, venv, allow_missing=[poetry.package.name]
)
return dedupe_packages(dependencies)
def find_dev_deps(
packages: PackageMap, venv: "_poetry.VirtualEnv", poetry: "_poetry.Poetry"
) -> List[PoetryPackage]:
"""Find the dev dependencies
Recursively identify the Poetry dev dependencies
:param packages: Mapping of all locked package names to their corresponding package object
:param venv: Poetry virtual environment to use for package compatibility checks
:param poetry: Poetry object for the current project
"""
dev_group_deps = find_additional_deps(
packages,
venv,
poetry,
poetry.pyproject.data["tool"]["poetry"]
.get("group", {})
.get("dev", {})
.get("dependencies", {})
.keys(),
)
# Legacy pyproject.toml poetry format:
legacy_dev_group_deps = find_additional_deps(
packages,
venv,
poetry,
poetry.pyproject.data["tool"]["poetry"].get("dev-dependencies", {}).keys(),
)
# Poetry 1.2 unions these two toml sections.
return dedupe_packages(dev_group_deps + legacy_dev_group_deps)
def dedupe_packages(packages: Sequence[PoetryPackage]) -> List[PoetryPackage]:
"""Deduplicates a sequence of PoetryPackages while preserving ordering
Adapted from StackOverflow: https://stackoverflow.com/a/480227
"""
seen: Set[PoetryPackage] = set()
# Make this faster, avoid method lookup below
seen_add = seen.add
return [p for p in packages if not (p in seen or seen_add(p))]