32 Commits
0.2.2 ... 0.3.1

Author SHA1 Message Date
ed039de674 Merge pull request #16 from enpaul/enp/fix
Fix support for extras installation
2020-10-24 11:55:35 -04:00
db0cf6ce0c Bump patch version 2020-10-24 11:53:38 -04:00
e8d3f4fcac Fix support for non-single extra dependency installation 2020-10-24 11:53:38 -04:00
653622fd35 Merge pull request #15 from enpaul/enp/housekeeping
Some housekeeping changes
2020-10-23 10:02:45 -04:00
de4c3515ec Refactor to maintain consistent terminology
With a dozen different "dependency" types, consistent naming is critical
2020-10-23 00:59:43 -04:00
01bfdd74bd Add missing documentation for internal functions/constants 2020-10-23 00:43:11 -04:00
3e98ec81eb Update beta roadmap features list 2020-10-22 21:16:32 -04:00
b6534f86d0 Merge pull request #11 from enpaul/enp/extras
Add support for the extras option
2020-10-22 21:10:18 -04:00
74e3ed01c0 Fix handling of nonexistent extras with custom app exception 2020-10-22 21:07:56 -04:00
f944843278 Bump feature version 2020-10-22 21:07:55 -04:00
7c761073b3 Add poetry-core as explicit dependency to support poetry-1.0 2020-10-22 20:47:46 -04:00
6075ea6a3e Update readme with development progress 2020-10-22 20:47:44 -04:00
fdee2d9a8d Add support for extras tox configuration option
Install only non-optional project dependencies by default
Install dependencies for extras when specified

Fixes #4
2020-10-22 20:47:19 -04:00
699fb347da Add links to other poetry tox plugins 2020-10-22 00:06:37 -04:00
a3bfd2687a Remove obsoleted documentation from readme
Clarify developer poetry requirement
Work on #12
2020-10-22 00:03:19 -04:00
0bf3b16091 Merge pull request #10 from enpaul/enp/core
Support poetry core
2020-10-11 20:15:46 -04:00
979fa58618 Bump patch version 2020-10-11 20:11:42 -04:00
462cc166a9 Include tests in sdist 2020-10-11 20:11:42 -04:00
30160985c1 Update poetry imports to use core module 2020-10-11 20:11:42 -04:00
5d87ffac72 Update poetry requirement to support 1.1+ 2020-10-11 20:11:42 -04:00
6b92189e50 !fixup 2020-10-11 20:10:30 -04:00
0fbc77c2c4 Add CI status badge 2020-10-11 20:08:43 -04:00
7867a7c98b Merge pull request #9 from enpaul/enp/ci
Add CI for PRs and CD for default branch
2020-10-11 20:03:12 -04:00
7a34c47168 Wrap test checks in bash to avoid globbing errors 2020-10-11 19:41:16 -04:00
4a1dc52755 Fix python version in pre-commit config 2020-10-11 19:41:16 -04:00
1e2156ecdb Add CI for PRs and CD for default branch 2020-10-11 19:41:16 -04:00
707a73c922 Bump patch version 2020-09-29 19:01:59 -04:00
640dbfe102 Fix implicit case-sensitive locked dependency name handling
Fixes #7
2020-09-29 19:01:52 -04:00
05c5a26cc4 Update non-poetry error handling to be more permissive
Fixes #1
2020-09-29 19:01:43 -04:00
ee6f939b6a Fix links to roadmap issues 2020-09-29 00:58:04 -04:00
82678e81e8 Add issue links for relevant roadmap items 2020-09-29 00:56:22 -04:00
5411025612 Proofreading and editing fixes
Check spelling more thoroughly
Improve clarity in a few places
Fix grammar mistakes
Fix docs that didn't get updated for the 0.2 overhaul
2020-09-29 00:51:08 -04:00
7 changed files with 698 additions and 680 deletions

41
.github/workflows/ci.yaml vendored Normal file
View File

@@ -0,0 +1,41 @@
---
name: CI
on:
pull_request:
types: ["opened", "synchronize"]
push:
branches: ["devel"]
jobs:
Test:
runs-on: ubuntu-latest
strategy:
matrix:
python:
- version: 3.6
toxenv: py36
- version: 3.7
toxenv: py37
- version: 3.8
toxenv: py38
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python.version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python.version }}
- name: Install tox
run: pip install "tox>=3.20.0,<3.21.0" --upgrade
- name: Run tests via ${{ matrix.python.toxenv }}
run: tox -e ${{ matrix.python.toxenv }}
Check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install tox requirements
run: pip install "tox>=3.20.0,<3.21.0" --upgrade
- name: Run meta checks
run: tox -e static -e static-tests -e security

View File

@@ -4,14 +4,14 @@ repos:
rev: 20.8b1 rev: 20.8b1
hooks: hooks:
- id: black - id: black
language_version: python3.7 language_version: python3
- repo: https://github.com/asottile/blacken-docs - repo: https://github.com/asottile/blacken-docs
rev: v0.5.0 rev: v0.5.0
hooks: hooks:
- id: blacken-docs - id: blacken-docs
additional_dependencies: [black==20.8b1] additional_dependencies: [black==20.8b1]
language_version: python3.7 language_version: python3
- repo: https://github.com/asottile/reorder_python_imports - repo: https://github.com/asottile/reorder_python_imports
rev: v2.3.5 rev: v2.3.5

191
README.md
View File

@@ -1,13 +1,14 @@
# tox-poetry-installer # tox-poetry-installer
A plugin for [Tox](https://tox.readthedocs.io/en/latest/) that allows test environment A plugin for [Tox](https://tox.readthedocs.io/en/latest/) that allows test environment
dependencies to be installed using [Poetry](https://python-poetry.org/) using its lockfile. dependencies to be installed using [Poetry](https://python-poetry.org/) from its lockfile.
⚠️ **This project is alpha software and should not be used in a production capacity** ⚠️ ⚠️ **This project is alpha software and should not be used in production environments** ⚠️
[![image](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT) [![ci-status](https://github.com/enpaul/tox-poetry-installer/workflows/CI/badge.svg?event=push)](https://github.com/enpaul/tox-poetry-installer/actions)
[![image](https://img.shields.io/pypi/v/tox-poetry-installer)](https://pypi.org/project/tox-poetry-installer/) [![license](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT)
[![image](https://img.shields.io/pypi/pyversions/tox-poetry-installer)](https://www.python.org) [![pypi-version](https://img.shields.io/pypi/v/tox-poetry-installer)](https://pypi.org/project/tox-poetry-installer/)
[![python-versions](https://img.shields.io/pypi/pyversions/tox-poetry-installer)](https://www.python.org)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black) [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
**Documentation** **Documentation**
@@ -26,12 +27,14 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) using it
Related resources: Related resources:
* [Poetry Python Project Manager](https://python-poetry.org/) * [Poetry Python Project Manager](https://python-poetry.org/)
* [Tox Automation Project](https://tox.readthedocs.io/en/latest/) * [Tox Automation Project](https://tox.readthedocs.io/en/latest/)
* [Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html) * [Poetry Dev-Dependencies Tox Plugin](https://github.com/sinoroc/tox-poetry-dev-dependencies)
* [Poetry Tox Plugin](https://github.com/tkukushkin/tox-poetry)
* [Other Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html)
## Installation ## Installation
Add the plugin as a development dependency a project using Poetry: Add the plugin as a development dependency of a Poetry project:
``` ```
~ $: poetry add tox-poetry-installer --dev ~ $: poetry add tox-poetry-installer --dev
@@ -43,11 +46,10 @@ Confirm that the plugin is installed, and Tox recognizes it, by checking the Tox
~ $: poetry run tox --version ~ $: poetry run tox --version
3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py 3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py
registered plugins: registered plugins:
tox-poetry-installer-0.2.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py tox-poetry-installer-0.2.2 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py
``` ```
If using in a CI/automation environment using Pip, ensure that the plugin is installed to the If using Pip, ensure that the plugin is installed to the same environment as Tox:
same environment as Tox:
``` ```
# Calling the virtualenv's 'pip' binary directly will cause pip to install to that virtualenv # Calling the virtualenv's 'pip' binary directly will cause pip to install to that virtualenv
@@ -77,8 +79,8 @@ commands = ...
``` ```
To require specific dependencies be installed from the Poetry lockfile, and let the rest be To require specific dependencies be installed from the Poetry lockfile, and let the rest be
installed using the default Tox installation method, add the suffix `@poetry` to the dependencies. installed using the default Tox installation backend, add the suffix `@poetry` to the dependencies.
In the example below the `pytest`, `pytest-cov`, and `black` dependencies will be installed using In the example below the `pytest`, `pytest-cov`, and `black` dependencies will be installed from
the lockfile while `pylint` and `mypy` will be installed using the versions specified here: the lockfile while `pylint` and `mypy` will be installed using the versions specified here:
```ini ```ini
@@ -100,7 +102,7 @@ one Tox is testing) will always be installed from the lockfile.
## Usage Examples ## Usage Examples
After installing the plugin to a project, your Tox automation is already benefiting from the After installing the plugin to a project your Tox automation is already benefiting from the
lockfile: when Tox installs your project package to one of your environments, all the dependencies lockfile: when Tox installs your project package to one of your environments, all the dependencies
of your project package will be installed using the versions specified in the lockfile. This of your project package will be installed using the versions specified in the lockfile. This
happens automatically and requires no configuration changes. happens automatically and requires no configuration changes.
@@ -108,7 +110,7 @@ happens automatically and requires no configuration changes.
But what about the rest of your Tox environment dependencies? But what about the rest of your Tox environment dependencies?
Let's use an example `tox.ini` file, below, that defines two environments: the main `testenv` for Let's use an example `tox.ini` file, below, that defines two environments: the main `testenv` for
running the project tests and `testenv:check` for running some other helpful checks: running the project tests and `testenv:check` for running some other helpful tools:
```ini ```ini
[tox] [tox]
@@ -152,9 +154,9 @@ Running Tox using this config gives us this error:
tox_poetry_installer.LockedDepVersionConflictError: Locked dependency 'pylint >=2.4.4,<2.6.0' cannot include version specifier tox_poetry_installer.LockedDepVersionConflictError: Locked dependency 'pylint >=2.4.4,<2.6.0' cannot include version specifier
``` ```
This is because we told the Tox environment to require all dependencies to be locked, but then also This is because we told the Tox environment to require all dependencies be locked, but then also
specified a specific version constraint for Pylint. With the `require_locked_deps = true` setting specified a specific version constraint for Pylint. With the `require_locked_deps = true` setting
Tox expects all dependencies to take their version from the lockfile, so when it got conflicting Tox expects all dependencies to take their version from the lockfile, so when it gets conflicting
information it errors. We can fix this by simply removing all version specifiers from the information it errors. We can fix this by simply removing all version specifiers from the
environment dependency list: environment dependency list:
@@ -175,9 +177,9 @@ recreated.
Now let's look at the `testenv` environment. Let's make the same changes to the `testenv` Now let's look at the `testenv` environment. Let's make the same changes to the `testenv`
environment that we made to `testenv:check` above; remove the PyTest version and add environment that we made to `testenv:check` above; remove the PyTest version and add
`require_locked_deps = true`. Then imagine that we want to add a new (made up) tool the test `require_locked_deps = true`. Then imagine that we want to add the
environment called `crash_override` to the environment: we can add `crash-override` as a dependency [Requests](https://requests.readthedocs.io/en/master/) library to the test environment: we
of the test environment, but this will cause an error: can add `requests` as a dependency of the test environment, but this will cause an error:
```ini ```ini
[testenv] [testenv]
@@ -185,30 +187,29 @@ description = Run the tests
require_locked_deps = true require_locked_deps = true
deps = deps =
pytest pytest
crash-override requests
commands = ... commands = ...
``` ```
Running Tox with this config gives us this error: Running Tox with this config gives us this error:
``` ```
tox_poetry_installer.LockedDepNotFoundError: No version of locked dependency 'crash-override' found in the project lockfile tox_poetry_installer.LockedDepNotFoundError: No version of locked dependency 'requests' found in the project lockfile
``` ```
This is because `crash-override` is not in our lockfile. Tox will refuse to install a dependency This is because `requests` is not in our lockfile yet. Tox will refuse to install a dependency
that isn't in the lockfile to an an environment that specifies `require_locked_deps = true`. We that isn't in the lockfile to an an environment that specifies `require_locked_deps = true`. We
could fix this (if `crash-override` was a real package) by running can fix this by running `poetry add requests --dev` to add it to the lockfile.
`poetry add crash-override --dev` to add it to the lockfile.
Now let's combine dependencies from the lockfile ("locked dependencies") with dependencies that are Now let's combine dependencies from the lockfile with dependencies that are
specified inline in the environment configuration ("unlocked dependencies"). specified in-line in the Tox environment configuration.
[This isn't generally recommended of course](#why-would-i-use-this), but it's a valid use case and [This isn't generally recommended](#why-would-i-use-this), but it is a valid use case and
fully supported by this plugin. Let's modify the `testenv` configuration to install PyTest from the fully supported by this plugin. Let's modify the `testenv` configuration to install PyTest
lockfile but then install an older version of the from the lockfile but then install an older version of the Requests library.
[Requests](https://requests.readthedocs.io/en/master/) library.
The first thing to do is remove the `require_locked_deps = true` setting so that we can install The first thing to do is remove the `require_locked_deps = true` setting so that we can install
Requests as an unlocked dependency. Then we can add our version of requests to the dependency list: Requests as an unlocked dependency. Then we can add our version specifier to the `requests`
entry in the dependency list:
```ini ```ini
[testenv] [testenv]
@@ -220,7 +221,8 @@ commands = ...
``` ```
However we still want `pytest` to be installed from the lockfile, so the final step is to tell Tox However we still want `pytest` to be installed from the lockfile, so the final step is to tell Tox
to install it from the lockfile by adding the suffix `@poetry` to it: to install it from the lockfile by adding the suffix `@poetry` to the `pytest` entry in the
dependency list:
```ini ```ini
[testenv] [testenv]
@@ -233,7 +235,7 @@ commands = ...
Now when the `testenv` environment is created it will install PyTest (and all of its dependencies) Now when the `testenv` environment is created it will install PyTest (and all of its dependencies)
from the lockfile while it will install Requests (and all of its dependencies) using the default from the lockfile while it will install Requests (and all of its dependencies) using the default
Tox installation backend using Pip. Tox installation backend.
## Known Drawbacks and Problems ## Known Drawbacks and Problems
@@ -247,16 +249,9 @@ Tox installation backend using Pip.
* [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver) * [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver) * [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* The [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) setting in `tox.ini`
does not work. Optional dependencies of the project package will not be installed to Tox
environments. (See the [road map](#roadmap))
* The plugin currently depends on `poetry<1.1.0`. This can be a different version than Poetry being
used for actual project development. (See the [road map](#roadmap))
* Tox environments automatically inherit their settings from the main `testenv` environment. This * Tox environments automatically inherit their settings from the main `testenv` environment. This
means that if the `require_locked_deps = true` is specified for the `testenv` environment then means that if the `require_locked_deps = true` is specified for the `testenv` environment then
all environments will also require locked dependencies. This can be overridden by explicitly all environments will also require locked dependencies. This can be overwritten by explicitly
specifying `require_locked_deps = false` on child environments where unlocked dependencies are specifying `require_locked_deps = false` on child environments where unlocked dependencies are
needed. needed.
@@ -272,20 +267,20 @@ Tox installation backend using Pip.
**Introduction** **Introduction**
The lockfile is a file generated by a package manager for a project that lists what The lockfile is a file generated by a package manager for a project that records what
dependencies are installed, the versions of those dependencies, and additional metadata that dependencies are installed, the versions of those dependencies, and any additional metadata that
the package manager can use to recreate the local project environment. This allows developers the package manager needs to recreate the local project environment. This allows developers
to have confidence that a bug they are encountering that may be caused by one of their to have confidence that a bug they are encountering that may be caused by one of their
dependencies will be reproducible on another device. In addition, installing a project dependencies will be reproducible on another device. In addition, installing a project
environment from a lockfile gives confidence that automated systems running tests or performing environment from a lockfile gives confidence that automated systems running tests or performing
builds are using the same environment that a developer is. builds are using the same environment as a developer.
[Poetry](https://python-poetry.org/) is a project dependency manager for Python projects, and [Poetry](https://python-poetry.org/) is a project dependency manager for Python projects, and
as such it creates and manages a lockfile so that its users can benefit from all the features so it creates and manages a lockfile so that its users can benefit from all the features
described above. [Tox](https://tox.readthedocs.io/en/latest/#what-is-tox) is an automation tool described above. [Tox](https://tox.readthedocs.io/en/latest/#what-is-tox) is an automation tool
that allows Python developers to run tests suites, perform builds, and automate tasks within that allows Python developers to run tests suites, perform builds, and automate tasks within
self contained [Python virtual environments](https://docs.python.org/3/tutorial/venv.html). self-contained [Python virtual environments](https://docs.python.org/3/tutorial/venv.html).
To make these environments useful, Tox supports installing per-environment dependencies. To make these environments useful Tox supports installing dependencies in each environment.
However, since these environments are created on the fly and Tox does not maintain a lockfile, However, since these environments are created on the fly and Tox does not maintain a lockfile,
there can be subtle differences between the dependencies a developer is using and the there can be subtle differences between the dependencies a developer is using and the
dependencies Tox uses. dependencies Tox uses.
@@ -293,19 +288,17 @@ dependencies Tox uses.
This is where this plugin comes into play. This is where this plugin comes into play.
By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the
PEP-508 compliant dependencies to a test environment. A more robust way to do this is to PEP-508 compliant dependencies to a test environment. This plugin extends the default
install dependencies directly from the lockfile so that the version installed to the Tox Tox dependency installation behavior to support installing dependencies using a Poetry-based
environment always matches the version Poetry specifies. This plugin overwrites the default installation method that makes use of the dependency metadata from Poetry's lockfile.
Tox dependency installation behavior and replaces it with a Poetry-based installation using
the dependency metadata from the lockfile.
**The Problem** **The Problem**
Environment dependencies for a Tox environment are usually done in PEP-508 format like the Environment dependencies for a Tox environment are usually specified in PEP-508 format, like
below example the below example:
```ini ```ini
# tox.ini # from tox.ini
... ...
[testenv] [testenv]
@@ -318,31 +311,35 @@ deps =
... ...
``` ```
Perhaps these dependencies are also useful during development, so they can be added to the Let's assume these dependencies are also useful during development, so they can be added to the
Poetry environment using this command: Poetry environment using this command:
``` ```
poetry add foo==1.2.3 bar>=1.3,<2.0 baz --dev poetry add --dev \
foo==1.2.3 \
bar>=1.3,<2.0 \
baz
``` ```
However there are three potential problems that could arise from each of these environment However there is a potential problem that could arise from each of these environment
dependencies that would _only_ appear in the Tox environment and not in the Poetry dependencies that would _only_ appear in the Tox environment and not in the Poetry
environment: environment in use by a developer:
* **The `foo` dependency is pinned to a specific version:** let's imagine a security * **The `foo` dependency is pinned to a specific version:** let's imagine a security
vulnerability is discovered in `foo` and the maintainers release version `1.2.4` to fix vulnerability is discovered in `foo` and the maintainers release version `1.2.4` to fix
it. A developer can run `poetry remove foo && poetry add foo^1.2` to get the new version, it. A developer can run `poetry remove foo` and then `poetry add foo^1.2` to get the new
but the Tox environment is left unchanged. The developer environment specified by the version, but the Tox environment is left unchanged. The development environment, as defined by
lockfile is now patched against the vulnerability, but the Tox environment is not. the lockfile, is now patched against the vulnerability but the Tox environment is not.
* **The `bar` dependency specifies a dynamic range:** a dynamic range allows a range of * **The `bar` dependency specifies a dynamic range:** a dynamic range allows a range of
versions to be installed, but the lockfile will have an exact version specified so that versions to be installed, but the lockfile will have an exact version specified so that
the Poetry environment is reproducible; this allows versions to be updated with the Poetry environment is reproducible; this allows versions to be updated with
`poetry update` rather than with the `remove` and `add` used above. If the maintainers of `poetry update` rather than with the `remove` and `add` commands used above. If the
`bar` release version `1.6.0` then the Tox environment will install it because it is valid maintainers of `bar` release version `1.6.0` then the Tox environment will install it
for the specified version range, meanwhile the Poetry environment will continue to install because it is valid for the specified version range. Meanwhile the Poetry environment will
the version from the lockfile until `poetry update bar` explicitly updates it. The continue to install the version from the lockfile until `poetry update bar` explicitly
development environment is now has a different version of `bar` than the Tox environment. updates it. The development environment is now has a different version of `bar` than the Tox
environment.
* **The `baz` dependency is unpinned:** unpinned dependencies are * **The `baz` dependency is unpinned:** unpinned dependencies are
[generally a bad idea](https://python-poetry.org/docs/faq/#why-are-unbound-version-constraints-a-bad-idea), [generally a bad idea](https://python-poetry.org/docs/faq/#why-are-unbound-version-constraints-a-bad-idea),
@@ -351,27 +348,26 @@ Poetry environment using this command:
but Pip (via Tox) will interpret it as a wildcard. If the latest version of `baz` is `1.0.0` but Pip (via Tox) will interpret it as a wildcard. If the latest version of `baz` is `1.0.0`
then `poetry add baz` will result in a constraint of `baz>=1.0.0,<2.0.0` while the Tox then `poetry add baz` will result in a constraint of `baz>=1.0.0,<2.0.0` while the Tox
environment will have a constraint of `baz==*`. The Tox environment can now install an environment will have a constraint of `baz==*`. The Tox environment can now install an
incompatible version of `baz` that cannot be easily caught using `poetry update`. incompatible version of `baz` and any errors that causes cannot be replicated using `poetry update`.
All of these problems can apply not only to the dependencies specified for a Tox environment, All of these problems can apply not only to the dependencies specified for a Tox environment,
but also to the dependencies of those dependencies, and so on. but also to the dependencies of those dependencies, those dependencies' dependencies, and so on.
**The Solution** **The Solution**
This plugin requires that all dependencies specified for all Tox environments be unbound This plugin allows dependencies specified in Tox environment take their version directly from
with no version constraint specified at all. This seems counter-intuitive given the problems the Poetry lockfile without needing an independent version to be specified in the Tox
outlined above, but what it allows the plugin to do is offload all version management to environment configuration. The modified version of the example environment given below appears
Poetry. less stable than the one presented above because it does not specify any versions for its
dependencies:
On initial inspection, the environment below appears less stable than the one presented above
because it does not specify any versions for its dependencies:
```ini ```ini
# tox.ini # from tox.ini
... ...
[testenv] [testenv]
description = Some very cool tests description = Some very cool tests
require_locked_deps = true
deps = deps =
foo foo
bar bar
@@ -380,19 +376,17 @@ deps =
... ...
``` ```
However with the `tox-poetry-installer` plugin installed this instructs Tox to install these However with the `tox-poetry-installer` plugin installed the `require_locked_deps = true`
dependencies using the Poetry lockfile so that the version installed to the Tox environment setting means that Tox will install these dependencies from the Poetry lockfile so that the
exactly matches the version Poetry is managing. When `poetry update` updates the lockfile version installed to the Tox environment exactly matches the version Poetry is managing. When
with new dependency versions, Tox will automatically install these new versions without needing `poetry update` updates the lockfile with new versions of these dependencies, Tox will
any changes to the configuration. automatically install these new versions without needing any changes to the configuration.
All dependencies are specified in one place (the lockfile) and dependency version management is
handled by a tool dedicated to that task (Poetry).
## Developing ## Developing
This project requires Poetry-1.0+, see the [installation instructions here](https://python-poetry.org/docs/#installation). This project requires a developer to have Poetry version 1.0+ installed on their workstation, see
the [installation instructions here](https://python-poetry.org/docs/#installation).
```bash ```bash
# Clone the repository... # Clone the repository...
@@ -435,7 +429,7 @@ releases on PyPI.
## Roadmap ## Roadmap
This project is under active development and is classified as alpha software, not yet ready This project is under active development and is classified as alpha software, not yet ready
usage in production systems. for usage in production environments.
* Beta classification will be assigned when the initial feature set is finalized * Beta classification will be assigned when the initial feature set is finalized
* Stable classification will be assigned when the test suite covers an acceptable number of * Stable classification will be assigned when the test suite covers an acceptable number of
@@ -445,20 +439,21 @@ usage in production systems.
- [X] Verify that primary package dependencies (from the `.package` env) are installed - [X] Verify that primary package dependencies (from the `.package` env) are installed
correctly using the Poetry backend. correctly using the Poetry backend.
- [ ] Support the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) - [X] Support the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras)
Tox configuration option Tox configuration option ([#4](https://github.com/enpaul/tox-poetry-installer/issues/4))
- [X] Add per-environment Tox configuration option to fall back to default installation - [X] Add per-environment Tox configuration option to fall back to default installation
backend. backend.
- [ ] Add detection of a changed lockfile to automatically trigger a rebuild of Tox
environments when necessary.
- [ ] Add warnings when an unsupported Tox configuration option is detected while using the - [ ] Add warnings when an unsupported Tox configuration option is detected while using the
Poetry backend. Poetry backend. ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5))
- [X] Add trivial tests to ensure the project metadata is consistent between the pyproject.toml - [X] Add trivial tests to ensure the project metadata is consistent between the pyproject.toml
and the module constants. and the module constants.
- [ ] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) - [X] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) and
Tox configuration option) and improve robustness of the Tox and Poetry module imports improve robustness of the Tox and Poetry module imports
to avoid potentially breaking API changes in upstream packages. to avoid potentially breaking API changes in upstream packages. ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
- [ ] Find and implement a way to mitigate the [Poetry UNSAFE_DEPENDENCIES bug](https://github.com/python-poetry/poetry/issues/1584). - [ ] Find and implement a way to mitigate the [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in Poetry.
([#6](https://github.com/enpaul/tox-poetry-installer/issues/6))
- [ ] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3))
- [ ] Add configuration option for installing all dev-dependencies to a testenv ([#14](https://github.com/enpaul/tox-poetry-installer/issues/14))
### Path to Stable ### Path to Stable
@@ -466,6 +461,6 @@ Everything in Beta plus...
- [ ] Add tests for each feature version of Tox between 2.3 and 3.20 - [ ] Add tests for each feature version of Tox between 2.3 and 3.20
- [ ] Add tests for Python-3.6, 3.7, and 3.8 - [ ] Add tests for Python-3.6, 3.7, and 3.8
- [ ] Add Github Actions based CI - [X] Add Github Actions based CI
- [ ] Add CI for CPython, PyPy, and Conda - [ ] Add CI for CPython, PyPy, and Conda
- [ ] Add CI for Linux and Windows - [ ] Add CI for Linux and Windows

971
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,11 +1,14 @@
[tool.poetry] [tool.poetry]
name = "tox-poetry-installer" name = "tox-poetry-installer"
version = "0.2.2" version = "0.3.1"
license = "MIT" license = "MIT"
authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"] authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
description = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile" description = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
repository = "https://github.com/enpaul/tox-poetry-installer/" repository = "https://github.com/enpaul/tox-poetry-installer/"
packages = [{include = "tox_poetry_installer.py"}] packages = [
{include = "tox_poetry_installer.py"},
{include = "tests/*.py", format = "sdist"}
]
keywords = ["tox", "poetry", "plugin"] keywords = ["tox", "poetry", "plugin"]
readme = "README.md" readme = "README.md"
classifiers = [ classifiers = [
@@ -27,7 +30,8 @@ poetry_installer = "tox_poetry_installer"
[tool.poetry.dependencies] [tool.poetry.dependencies]
python = "^3.6" python = "^3.6"
poetry = ">=1.0.0, <1.1.0" poetry = "^1.0.0"
poetry-core = "^1.0.0"
tox = "^2.3.0 || ^3.0.0" tox = "^2.3.0 || ^3.0.0"
[tool.poetry.dev-dependencies] [tool.poetry.dev-dependencies]

View File

@@ -45,8 +45,8 @@ allowlist_externals =
commands = commands =
black {toxinidir}/tests/ black {toxinidir}/tests/
bash -c "reorder-python-imports {toxinidir}/tests/*.py --unclassifiable-application-module tox_poetry_installer" bash -c "reorder-python-imports {toxinidir}/tests/*.py --unclassifiable-application-module tox_poetry_installer"
pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tests/ bash -c "pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tests/*.py"
mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tests/ bash -c "mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tests/*.py"
[testenv:security] [testenv:security]
description = Security checks description = Security checks

View File

@@ -1,9 +1,24 @@
"""Tox plugin for installing environments using Poetry """Tox plugin for installing environments using Poetry
This plugin makes use of the ``tox_testenv_install_deps`` Tox plugin hook to replace the default This plugin makes use of the ``tox_testenv_install_deps`` Tox plugin hook to augment the default
installation functionality to install dependencies from the Poetry lockfile for the project. It installation functionality to install dependencies from the Poetry lockfile for the project. It
does this by using ``poetry`` to read in the lockfile, identify necessary dependencies, and then does this by using ``poetry`` to read in the lockfile, identify necessary dependencies, and then
use Poetry's ``PipInstaller`` class to install those packages into the Tox environment. use Poetry's ``PipInstaller`` class to install those packages into the Tox environment.
Quick definition of terminology:
* "project package" - the package that Tox is testing, usually the one the current project is
is developing; definitionally, this is the package that is built by Tox in the ``.package`` env.
* "project package dependency" or "project dependency" - a dependency required by the project
package for installation; i.e. a package that would be installed when running
``pip install <project package>``.
* "environment dependency" - a dependency specified for a given testenv in the Tox configuration.
* "locked dependency" - a package that is present in the Poetry lockfile and will be installed
according to the metadata in the lockfile.
* "unlocked dependency" - a package that is either not present in the Poetry lockfile or is not
specified to be installed according to the metadata in the lockfile.
* "transiety dependency" - a package not explicitly specified for installation, but required by a
package that is explicitly specified.
""" """
from pathlib import Path from pathlib import Path
from typing import Dict from typing import Dict
@@ -13,11 +28,11 @@ from typing import Sequence
from typing import Set from typing import Set
from typing import Tuple from typing import Tuple
from poetry.core.packages import Package as PoetryPackage
from poetry.factory import Factory as PoetryFactory from poetry.factory import Factory as PoetryFactory
from poetry.factory import Poetry
from poetry.installation.pip_installer import PipInstaller as PoetryPipInstaller from poetry.installation.pip_installer import PipInstaller as PoetryPipInstaller
from poetry.io.null_io import NullIO as PoetryNullIO from poetry.io.null_io import NullIO as PoetryNullIO
from poetry.packages import Package as PoetryPackage from poetry.poetry import Poetry
from poetry.puzzle.provider import Provider as PoetryProvider from poetry.puzzle.provider import Provider as PoetryProvider
from poetry.utils.env import VirtualEnv as PoetryVirtualEnv from poetry.utils.env import VirtualEnv as PoetryVirtualEnv
from tox import hookimpl from tox import hookimpl
@@ -30,7 +45,7 @@ from tox.venv import VirtualEnv as ToxVirtualEnv
__title__ = "tox-poetry-installer" __title__ = "tox-poetry-installer"
__summary__ = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile" __summary__ = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
__version__ = "0.2.2" __version__ = "0.3.1"
__url__ = "https://github.com/enpaul/tox-poetry-installer/" __url__ = "https://github.com/enpaul/tox-poetry-installer/"
__license__ = "MIT" __license__ = "MIT"
__authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"] __authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
@@ -52,6 +67,10 @@ _REPORTER_PREFIX = f"[{__title__}]:"
_MAGIC_SUFFIX_MARKER = "@poetry" _MAGIC_SUFFIX_MARKER = "@poetry"
# Map of package names to the package object
PackageMap = Dict[str, PoetryPackage]
class _SortedEnvDeps(NamedTuple): class _SortedEnvDeps(NamedTuple):
unlocked_deps: List[ToxDepConfig] unlocked_deps: List[ToxDepConfig]
locked_deps: List[ToxDepConfig] locked_deps: List[ToxDepConfig]
@@ -69,6 +88,10 @@ class LockedDepNotFoundError(ToxPoetryInstallerException):
"""Locked dependency was not found in the lockfile""" """Locked dependency was not found in the lockfile"""
class ExtraNotFoundError(ToxPoetryInstallerException):
"""Project package extra not defined in project's pyproject.toml"""
def _sort_env_deps(venv: ToxVirtualEnv) -> _SortedEnvDeps: def _sort_env_deps(venv: ToxVirtualEnv) -> _SortedEnvDeps:
"""Sorts the environment dependencies by lock status """Sorts the environment dependencies by lock status
@@ -111,10 +134,10 @@ def _sort_env_deps(venv: ToxVirtualEnv) -> _SortedEnvDeps:
unlocked_deps.append(dep) unlocked_deps.append(dep)
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(locked_deps)} locked env dependencies for installation from poetry lockfile: {[item.name for item in locked_deps]}" f"{_REPORTER_PREFIX} identified {len(locked_deps)} locked env dependencies: {[item.name for item in locked_deps]}"
) )
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(unlocked_deps)} unlocked env dependencies for installation using default backend: {[item.name for item in unlocked_deps]}" f"{_REPORTER_PREFIX} identified {len(unlocked_deps)} unlocked env dependencies: {[item.name for item in unlocked_deps]}"
) )
return _SortedEnvDeps(locked_deps=locked_deps, unlocked_deps=unlocked_deps) return _SortedEnvDeps(locked_deps=locked_deps, unlocked_deps=unlocked_deps)
@@ -129,6 +152,11 @@ def _install_to_venv(
:param venv: Tox virtual environment to install the packages to :param venv: Tox virtual environment to install the packages to
:param packages: List of packages to install to the virtual environment :param packages: List of packages to install to the virtual environment
""" """
reporter.verbosity1(
f"{_REPORTER_PREFIX} Installing {len(packages)} packages to environment at {venv.envconfig.envdir}"
)
installer = PoetryPipInstaller( installer = PoetryPipInstaller(
env=PoetryVirtualEnv(path=Path(venv.envconfig.envdir)), env=PoetryVirtualEnv(path=Path(venv.envconfig.envdir)),
io=PoetryNullIO(), io=PoetryNullIO(),
@@ -140,7 +168,7 @@ def _install_to_venv(
installer.install(dependency) installer.install(dependency)
def _find_transients(poetry: Poetry, dependency_name: str) -> Set[PoetryPackage]: def _find_transients(packages: PackageMap, dependency_name: str) -> Set[PoetryPackage]:
"""Using a poetry object identify all dependencies of a specific dependency """Using a poetry object identify all dependencies of a specific dependency
:param poetry: Populated poetry object which can be used to build a populated locked :param poetry: Populated poetry object which can be used to build a populated locked
@@ -152,10 +180,6 @@ def _find_transients(poetry: Poetry, dependency_name: str) -> Set[PoetryPackage]
.. note:: The package corresponding to the dependency named by ``dependency_name`` is included .. note:: The package corresponding to the dependency named by ``dependency_name`` is included
in the list of returned packages. in the list of returned packages.
""" """
packages: Dict[str, PoetryPackage] = {
package.name: package
for package in poetry.locker.locked_repository(True).packages
}
try: try:
top_level = packages[dependency_name] top_level = packages[dependency_name]
@@ -163,7 +187,7 @@ def _find_transients(poetry: Poetry, dependency_name: str) -> Set[PoetryPackage]
def find_deps_of_deps(name: str) -> List[PoetryPackage]: def find_deps_of_deps(name: str) -> List[PoetryPackage]:
if name in PoetryProvider.UNSAFE_PACKAGES: if name in PoetryProvider.UNSAFE_PACKAGES:
reporter.warning( reporter.warning(
f"{_REPORTER_PREFIX} installing '{name}' using Poetry is not supported; skipping" f"{_REPORTER_PREFIX} installing package '{name}' using Poetry is not supported; skipping installation of package '{name}'"
) )
return [] return []
transients = [packages[name]] transients = [packages[name]]
@@ -185,20 +209,31 @@ def _find_transients(poetry: Poetry, dependency_name: str) -> Set[PoetryPackage]
) from None ) from None
def _install_env_dependencies(venv: ToxVirtualEnv, poetry: Poetry): def _install_env_dependencies(
venv: ToxVirtualEnv, poetry: Poetry, packages: PackageMap
):
"""Install the packages for a specified testenv
Processes the tox environment config, identifies any locked environment dependencies, pulls
them from the lockfile, and installs them to the virtual environment.
:param venv: Tox virtual environment to install the packages to
:param poetry: Poetry object the packages were sourced from
:param packages: Mapping of package names to the corresponding package object
"""
env_deps = _sort_env_deps(venv) env_deps = _sort_env_deps(venv)
dependencies: List[PoetryPackage] = [] dependencies: List[PoetryPackage] = []
for dep in env_deps.locked_deps: for dep in env_deps.locked_deps:
try: try:
dependencies += _find_transients(poetry, dep.name) dependencies += _find_transients(packages, dep.name.lower())
except ToxPoetryInstallerException as err: except ToxPoetryInstallerException as err:
venv.status = "lockfile installation failed" venv.status = "lockfile installation failed"
reporter.error(f"{_REPORTER_PREFIX} {err}") reporter.error(f"{_REPORTER_PREFIX} {err}")
raise err raise err
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dependencies)} actual dependencies from {len(venv.envconfig.deps)} specified env dependencies" f"{_REPORTER_PREFIX} identified {len(dependencies)} total dependencies from {len(env_deps.locked_deps)} locked env dependencies"
) )
reporter.verbosity1( reporter.verbosity1(
@@ -212,21 +247,55 @@ def _install_env_dependencies(venv: ToxVirtualEnv, poetry: Poetry):
_install_to_venv(poetry, venv, dependencies) _install_to_venv(poetry, venv, dependencies)
def _install_package_dependencies(venv: ToxVirtualEnv, poetry: Poetry): def _install_project_dependencies(
venv: ToxVirtualEnv, poetry: Poetry, packages: PackageMap
):
"""Install the dependencies of the project package
Install all primary dependencies of the project package.
:param venv: Tox virtual environment to install the packages to
:param poetry: Poetry object the packages were sourced from
:param packages: Mapping of package names to the corresponding package object
"""
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} performing installation of project dependencies" f"{_REPORTER_PREFIX} performing installation of project dependencies"
) )
primary_dependencies = poetry.locker.locked_repository(False).packages base_dependencies: List[PoetryPackage] = [
packages[item.name]
for item in poetry.package.requires
if not item.is_optional()
]
extra_dependencies: List[PoetryPackage] = []
for extra in venv.envconfig.extras:
try:
extra_dependencies += [
packages[item.name] for item in poetry.package.extras[extra]
]
except KeyError:
raise ExtraNotFoundError(
f"Environment '{venv.name}' specifies project extra '{extra}' which was not found in the lockfile"
) from None
dependencies: List[PoetryPackage] = []
for dep in base_dependencies + extra_dependencies:
try:
dependencies += _find_transients(packages, dep.name.lower())
except ToxPoetryInstallerException as err:
venv.status = "lockfile installation failed"
reporter.error(f"{_REPORTER_PREFIX} {err}")
raise err
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(primary_dependencies)} dependencies of project '{poetry.package.name}'" f"{_REPORTER_PREFIX} identified {len(dependencies)} total dependencies from {len(poetry.package.requires)} project dependencies"
) )
reporter.verbosity0( reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(primary_dependencies)} project dependencies from lockfile" f"{_REPORTER_PREFIX} ({venv.name}) installing {len(dependencies)} project dependencies from lockfile"
) )
_install_to_venv(poetry, venv, primary_dependencies) _install_to_venv(poetry, venv, dependencies)
@hookimpl @hookimpl
@@ -268,39 +337,41 @@ def tox_testenv_install_deps(venv: ToxVirtualEnv, action: ToxAction):
try: try:
poetry = PoetryFactory().create_poetry(venv.envconfig.config.toxinidir) poetry = PoetryFactory().create_poetry(venv.envconfig.config.toxinidir)
except RuntimeError as err: except RuntimeError:
# Support running the plugin when the current tox project does not use Poetry for its # Support running the plugin when the current tox project does not use Poetry for its
# environment/dependency management. # environment/dependency management.
# #
# ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case, # ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case,
# is) raised in many different places for different purposes. This check of the error # is) raised in many different places for different purposes.
# content, while crude and potentially fragile, will hopefully prevent ``RuntimeError``s reporter.verbosity1(
# not caused by this specific condition to be re-raised as genuine errors. This may need f"{_REPORTER_PREFIX} project does not use Poetry for env management, skipping installation of locked dependencies"
# tuning in the future. )
if "[tool.poetry] section not found" in str(err): return
reporter.verbosity1(
f"{_REPORTER_PREFIX} project does not use Poetry for env management, skipping installation of locked dependencies"
)
return
raise err
reporter.verbosity1( reporter.verbosity1(
f"{_REPORTER_PREFIX} loaded project pyproject.toml from {poetry.file}" f"{_REPORTER_PREFIX} loaded project pyproject.toml from {poetry.file}"
) )
package_map: PackageMap = {
package.name: package
for package in poetry.locker.locked_repository(True).packages
}
# Handle the installation of any locked env dependencies from the lockfile # Handle the installation of any locked env dependencies from the lockfile
_install_env_dependencies(venv, poetry) _install_env_dependencies(venv, poetry, package_map)
# Handle the installation of the package dependencies from the lockfile if the package is # Handle the installation of the package dependencies from the lockfile if the package is
# being installed to this venv; otherwise skip installing the package dependencies # being installed to this venv; otherwise skip installing the package dependencies
if not venv.envconfig.skip_install and not venv.envconfig.config.skipsdist: if venv.envconfig.skip_install:
_install_package_dependencies(venv, poetry) reporter.verbosity1(
else: f"{_REPORTER_PREFIX} env specifies 'skip_install = true', skipping installation of project package"
if venv.envconfig.skip_install: )
reporter.verbosity1( return
f"{_REPORTER_PREFIX} env specifies 'skip_install = true', skipping installation of project package"
) if venv.envconfig.config.skipsdist:
elif venv.envconfig.config.skipsdist: reporter.verbosity1(
reporter.verbosity1( f"{_REPORTER_PREFIX} config specifies 'skipsdist = true', skipping installation of project package"
f"{_REPORTER_PREFIX} config specifies 'skipsdist = true', skipping installation of project package" )
) return
_install_project_dependencies(venv, poetry, package_map)