10 Commits
0.1.3 ... 0.2.0

Author SHA1 Message Date
78efd82c82 Add quickstart section 2020-09-27 14:51:12 -04:00
5476f4ab11 Update example errors with new names, messages, and formatting 2020-09-27 14:45:11 -04:00
a4d1c1e4df Add missing drawback about poetry unsafe dependencies
Add item to beta specification to fix/mitigate this somehow
2020-09-27 14:45:11 -04:00
fb1ac3b0de Misc documentation updates
Fix inconsistent env naming in example ini snippets
Fix PS1 in demo commands to make it clearer what they are
Rename 'getting started' section to 'usage'
Add note about main branch and tag usage
2020-09-27 14:44:50 -04:00
c481b7b0bb Update error handling to improve UX
Add discrete exception for version conflict vs not in lockfile errors
Update to set tox venv to failed when conflict happens
Update tox reporting for errors to use proper level
2020-09-27 14:17:15 -04:00
f20e434f2c Overhaul usage documentation
Add better installation documentation
Add configuration examples and usage walk through using new design system
Update roadmap with feature changes for 0.2
Clarify drawbacks section with more useful context
2020-09-27 14:02:39 -04:00
10211bc674 Bump feature version 2020-09-26 10:48:10 -04:00
50c008d054 Require locked dependencies for all envs
Gotta dogfood sometime
2020-09-26 10:47:46 -04:00
476f27943e Fix bug with dependency name modification
Standardize logging messages
2020-09-26 10:47:45 -04:00
8bb9255fc1 Implement new config interface system to expose more options
Default behavior is now to only install project package deps from lockfile
Specific env deps can be locked using @poetry suffix
Entire env can now be forced to use locked deps with require_locked_deps option
2020-09-26 10:46:38 -04:00
4 changed files with 378 additions and 85 deletions

250
README.md
View File

@@ -11,9 +11,11 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) using it
**Documentation**
* [Installation and Usage](#installation-and-usage)
* [Limitations](#limitations)
* [Why would I use this?](#what-problems-does-this-solve) (What problems does this solve?)
* [Installation](#installation)
* [Quick Start](#quick-start)
* [Usage](#usage)
* [Known Drawbacks and Problems](#known-drawbacks-and-problems)
* [Why would I use this?](#why-would-i-use-this) (What problems does this solve?)
* [Developing](#developing)
* [Contributing](#contributing)
* [Roadmap](#roadmap)
@@ -26,62 +28,237 @@ Related resources:
* [Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html)
## Installation and Usage
## Installation
1. Install the plugin from PyPI:
Add the plugin as a development dependency a project using Poetry:
```
poetry add tox-poetry-installer --dev
~ $: poetry add tox-poetry-installer --dev
```
2. Remove all version specifications from the environment dependencies in `tox.ini`:
Confirm that the plugin is installed, and Tox recognizes it, by checking the Tox version:
```
~ $: poetry run tox --version
3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py
registered plugins:
tox-poetry-installer-0.2.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py
```
If using in a CI/automation environment using Pip, ensure that the plugin is installed to the
same environment as Tox:
```
# Calling the virtualenv's 'pip' binary directly will cause pip to install to that virtualenv
~ $: /path/to/my/automation/virtualenv/bin/pip install tox
~ $: /path/to/my/automation/virtualenv/bin/pip install tox-poetry-installer
```
## Quick Start
To require a Tox environment install all it's dependencies from the Poetry lockfile, add the
`require_locked_deps = true` option to the environment configuration and remove all version
specifiers from the dependency list. The versions to install will be taken from the lockfile
directly:
```ini
# This...
[testenv]
description = My cool test environment
description = Run the tests
require_locked_deps = true
deps =
requests >=2.19,<3.0
toml == 0.10.0
pytest >=5.4
# ...becomes this:
[testenv]
description = My cool test environment
deps =
requests
toml
pytest
pytest-cov
black
pylint
mypy
commands = ...
```
3. Run Tox with the `--recreate` flag to rebuild the test environments:
To require specific dependencies be installed from the Poetry lockfile, and let the rest be
installed using the default Tox installation method, add the suffix `@poetry` to the dependencies.
In the example below the `pytest`, `pytest-cov`, and `black` dependencies will be installed using
the lockfile while `pylint` and `mypy` will be installed using the versions specified here:
```
poetry run tox --recreate
```ini
[testenv]
description = Run the tests
require_locked_deps = true
deps =
pytest@poetry
pytest-cov@poetry
black@poetry
pylint >=2.5.0
mypy == 0.770
commands = ...
```
4. 💸 Profit 💸
**Note:** Regardless of the settings outlined above, all dependencies of the project package (the
one Tox is testing) will always be installed from the lockfile.
## Limitations
## Usage
* In general, any command line or INI settings that affect how Tox installs environment
dependencies will be disabled by installing this plugin. A non-exhaustive and untested
list of the INI options that are not expected to work with this plugin is below:
After installing the plugin to a project, your Tox automation is already benefiting from the
lockfile: when Tox installs your project package to one of your environments, all the dependencies
of your project package will be installed using the versions specified in the lockfile. This
happens automatically and requires no configuration changes.
But what about the rest of your Tox environment dependencies?
Let's use an example `tox.ini` file, below, that defines two environments: the main `testenv` for
running the project tests and `testenv:check` for running some other helpful checks:
```ini
[tox]
envlist = py37, static
isolated_build = true
[testenv]
description = Run the tests
deps =
pytest == 5.3.0
commands = ...
[testenv:check]
description = Static formatting and quality enforcement
deps =
pylint >=2.4.4,<2.6.0
mypy == 0.770
black --pre
commands = ...
```
Let's focus on the `testenv:check` environment first. In this project there's no reason that any
of these tools should be a different version than what a human developer is using when installing
from the lockfile. We can require that these dependencies be installed from the lockfile by adding
the option `require_locked_deps = true` to the environment config, but this will cause an error:
```ini
[testenv:check]
description = Static formatting and quality enforcement
require_locked_deps = true
deps =
pylint >=2.4.4,<2.6.0
mypy == 0.770
black --pre
commands = ...
```
Running Tox using this config gives us this error:
```
tox_poetry_installer.LockedDepVersionConflictError: Locked dependency 'pylint >=2.4.4,<2.6.0' cannot include version specifier
```
This is because we told the Tox environment to require all dependencies to be locked, but then also
specified a specific version constraint for Pylint. With the `require_locked_deps = true` setting
Tox expects all dependencies to take their version from the lockfile, so when it got conflicting
information it errors. We can fix this by simply removing all version specifiers from the
environment dependency list:
```ini
[testenv:check]
description = Static formatting and quality enforcement
require_locked_deps = true
deps =
pylint
mypy
black
commands = ...
```
Now all the dependencies will be installed from the lockfile. If Poetry updates the lockfile with
a new version then that updated version will be automatically installed when the Tox environment is
recreated.
Now let's look at the `testenv` environment. Let's make the same changes to the `testenv`
environment that we made to `testenv:check` above; remove the PyTest version and add
`require_locked_deps = true`. Then imagine that we want to add a new (made up) tool the test
environment called `crash_override` to the environment: we can add `crash-override` as a dependency
of the test environment, but this will cause an error:
```ini
[testenv]
description = Run the tests
require_locked_deps = true
deps =
pytest
crash-override
commands = ...
```
Running Tox with this config gives us this error:
```
tox_poetry_installer.LockedDepNotFoundError: No version of locked dependency 'crash-override' found in the project lockfile
```
This is because `crash-override` is not in our lockfile. Tox will refuse to install a dependency
that isn't in the lockfile to an an environment that specifies `require_locked_deps = true`. We
could fix this (if `crash-override` was a real package) by running
`poetry add crash-override --dev` to add it to the lockfile.
Now let's combine dependencies from the lockfile ("locked dependencies") with dependencies that are
specified inline in the environment configuration ("unlocked dependencies").
[This isn't generally recommended of course](#why-would-i-use-this), but it's a valid use case and
fully supported by this plugin. Let's modify the `testenv` configuration to install PyTest from the
lockfile but then install an older version of the
[Requests](https://requests.readthedocs.io/en/master/) library.
The first thing to do is remove the `require_locked_deps = true` setting so that we can install
Requests as an unlocked dependency. Then we can add our version of requests to the dependency list:
```ini
[testenv]
description = Run the tests
deps =
pytest
requests >=2.2.0,<2.10.0
commands = ...
```
However we still want `pytest` to be installed from the lockfile, so the final step is to tell Tox
to install it from the lockfile by adding the suffix `@poetry` to it:
```ini
[testenv]
description = Run the tests
deps =
pytest@poetry
requests >=2.2.0,<2.10.0
commands = ...
```
Now when the `testenv` environment is created it will install PyTest (and all of its dependencies)
from the lockfile while it will install Requests (and all of its dependencies) using the default
Tox installation backend using Pip.
## Known Drawbacks and Problems
* The following `tox.ini` configuration options have no effect on the dependencies installed from
the Poetry lockfile (note that they will still affect unlocked dependencies):
* [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command)
* [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre)
* [`downloadcache`](https://tox.readthedocs.io/en/latest/config.html#conf-downloadcache) (deprecated)
* [`download`](https://tox.readthedocs.io/en/latest/config.html#conf-download)
* [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras)
* When the plugin is enabled all dependencies for all environments will use the Poetry backend
provided by the plugin; this functionality cannot be disabled on a per-environment basis.
* The [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras) setting in `tox.ini`
does not work. Optional dependencies of the project package will not be installed to Tox
environments. (See the [road map](#roadmap))
* Alternative versions cannot be specified alongside versions from the lockfile. All
dependencies are installed from the lockfile and alternative versions cannot be specified
in the Tox configuration.
* The plugin currently depends on `poetry<1.1.0`. This can be a different version than Poetry being
used for actual project development. (See the [road map](#roadmap))
* There are a handful of packages that cannot be installed from the lockfile, whether as specific
dependencies or as transient dependencies (dependencies of dependencies). This is due to
[an ongoing discussion in the Poetry project](https://github.com/python-poetry/poetry/issues/1584);
the list of dependencies that cannot be installed from the lockfile can be found
[here](https://github.com/python-poetry/poetry/blob/cc8f59a31567f806be868aba880ae0642d49b74e/poetry/puzzle/provider.py#L55).
This plugin will skip these dependencies entirely, but log a warning when they are encountered.
## Why would I use this?
@@ -234,6 +411,10 @@ poetry run tox
All project contributors and participants are expected to adhere to the
[Contributor Covenant Code of Conduct, Version 2](CODE_OF_CONDUCT.md).
The `devel` branch has the latest (potentially unstable) changes. The
[tagged versions](https://github.com/enpaul/tox-poetry-installer/releases) correspond to the
releases on PyPI.
* To report a bug, request a feature, or ask for assistance, please
[open an issue on the Github repository](https://github.com/enpaul/tox-poetry-installer/issues/new).
* To report a security concern or code of conduct violation, please contact the project author
@@ -259,7 +440,7 @@ usage in production systems.
correctly using the Poetry backend.
- [ ] Support the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras)
Tox configuration option
- [ ] Add per-environment Tox configuration option to fall back to default installation
- [X] Add per-environment Tox configuration option to fall back to default installation
backend.
- [ ] Add detection of a changed lockfile to automatically trigger a rebuild of Tox
environments when necessary.
@@ -270,6 +451,7 @@ usage in production systems.
- [ ] Update to use [poetry-core](https://github.com/python-poetry/poetry-core)
Tox configuration option) and improve robustness of the Tox and Poetry module imports
to avoid potentially breaking API changes in upstream packages.
- [ ] Find and implement a way to mitigate the [Poetry UNSAFE_DEPENDENCIES bug](https://github.com/python-poetry/poetry/issues/1584).
### Path to Stable

View File

@@ -1,6 +1,6 @@
[tool.poetry]
name = "tox-poetry-installer"
version = "0.1.3"
version = "0.2.0"
license = "MIT"
authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
description = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"

View File

@@ -4,6 +4,7 @@ isolated_build = true
[testenv]
description = Run the tests
require_locked_deps = true
deps =
pytest
pytest-cov
@@ -13,6 +14,7 @@ commands =
[testenv:static]
description = Static formatting and quality enforcement
require_locked_deps = true
basepython = python3.8
ignore_errors = true
deps =
@@ -30,6 +32,7 @@ commands =
[testenv:static-tests]
description = Static formatting and quality enforcement for the tests
require_locked_deps = true
basepython = python3.8
ingore_errors = true
deps =
@@ -47,6 +50,7 @@ commands =
[testenv:security]
description = Security checks
require_locked_deps = true
basepython = python3.8
ignore_errors = true
skip_install = true

View File

@@ -8,7 +8,7 @@ use Poetry's ``PipInstaller`` class to install those packages into the Tox envir
from pathlib import Path
from typing import Dict
from typing import List
from typing import Optional
from typing import NamedTuple
from typing import Sequence
from typing import Tuple
@@ -22,28 +22,101 @@ from poetry.utils.env import VirtualEnv as PoetryVirtualEnv
from tox import hookimpl
from tox import reporter
from tox.action import Action as ToxAction
from tox.config import DepConfig as ToxDepConfig
from tox.config import Parser as ToxParser
from tox.venv import VirtualEnv as ToxVirtualEnv
__title__ = "tox-poetry-installer"
__summary__ = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
__version__ = "0.1.3"
__version__ = "0.2.0"
__url__ = "https://github.com/enpaul/tox-poetry-installer/"
__license__ = "MIT"
__authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
# Valid PEP508 version delimiters. These are used to test whether a given string (specifically a
# dependency name) is just a package name or also includes a version identifier.
_PEP508_VERSION_DELIMITERS: Tuple[str, ...] = ("~=", "==", "!=", ">", "<")
# Prefix all reporter messages should include to indicate that they came from this module in the
# console output.
_REPORTER_PREFIX = f"[{__title__}]:"
# Suffix that indicates an env dependency should be treated as a locked dependency and thus be
# installed from the lockfile. Will be automatically stripped off of a dependency name during
# sorting so that the resulting string is just the valid package name. This becomes optional when
# the "require_locked_deps" option is true for an environment; in that case a bare dependency like
# 'foo' is treated the same as an explicitly locked dependency like 'foo@poetry'
_MAGIC_SUFFIX_MARKER = "@poetry"
class _SortedEnvDeps(NamedTuple):
unlocked_deps: List[ToxDepConfig]
locked_deps: List[ToxDepConfig]
class ToxPoetryInstallerException(Exception):
"""Error while installing locked dependencies to the test environment"""
class NoLockedDependencyError(ToxPoetryInstallerException):
"""Cannot install a package that is not in the lockfile"""
class LockedDepVersionConflictError(ToxPoetryInstallerException):
"""Locked dependencies cannot specify an alternate version for installation"""
class LockedDepNotFoundError(ToxPoetryInstallerException):
"""Locked dependency was not found in the lockfile"""
def _sort_env_deps(venv: ToxVirtualEnv) -> _SortedEnvDeps:
"""Sorts the environment dependencies by lock status
Lock status determines whether a given environment dependency will be installed from the
lockfile using the Poetry backend, or whether this plugin will skip it and allow it to be
installed using the default pip-based backend (an unlocked dependency).
.. note:: A locked dependency must follow a required format. To avoid reinventing the wheel
(no pun intended) this module does not have any infrastructure for parsing PEP-508
version specifiers, and so requires locked dependencies to be specified with no
version (the installed version being taken from the lockfile). If a dependency is
specified as locked and its name is also a PEP-508 string then an error will be
raised.
"""
reporter.verbosity1(
f"{_REPORTER_PREFIX} sorting {len(venv.envconfig.deps)} env dependencies by lock requirement"
)
unlocked_deps = []
locked_deps = []
for dep in venv.envconfig.deps:
if venv.envconfig.require_locked_deps:
reporter.verbosity1(
f"{_REPORTER_PREFIX} lock required for env, treating '{dep.name}' as locked env dependency"
)
dep.name = dep.name.replace(_MAGIC_SUFFIX_MARKER, "")
locked_deps.append(dep)
else:
if dep.name.endswith(_MAGIC_SUFFIX_MARKER):
reporter.verbosity1(
f"{_REPORTER_PREFIX} specification includes marker '{_MAGIC_SUFFIX_MARKER}', treating '{dep.name}' as locked env dependency"
)
dep.name = dep.name.replace(_MAGIC_SUFFIX_MARKER, "")
locked_deps.append(dep)
else:
reporter.verbosity1(
f"{_REPORTER_PREFIX} specification does not include marker '{_MAGIC_SUFFIX_MARKER}', treating '{dep.name}' as unlocked env dependency"
)
unlocked_deps.append(dep)
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(locked_deps)} locked env dependencies for installation from poetry lockfile: {[item.name for item in locked_deps]}"
)
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(unlocked_deps)} unlocked env dependencies for installation using default backend: {[item.name for item in unlocked_deps]}"
)
return _SortedEnvDeps(locked_deps=locked_deps, unlocked_deps=unlocked_deps)
def _install_to_venv(
@@ -66,9 +139,7 @@ def _install_to_venv(
installer.install(dependency)
def _find_locked_dependencies(
poetry: Poetry, dependency_name: str
) -> List[PoetryPackage]:
def _find_transients(poetry: Poetry, dependency_name: str) -> List[PoetryPackage]:
"""Using a poetry object identify all dependencies of a specific dependency
:param poetry: Populated poetry object which can be used to build a populated locked
@@ -88,7 +159,7 @@ def _find_locked_dependencies(
try:
top_level = packages[dependency_name]
def find_transients(name: str) -> List[PoetryPackage]:
def find_deps_of_deps(name: str) -> List[PoetryPackage]:
if name in PoetryProvider.UNSAFE_PACKAGES:
reporter.warning(
f"{_REPORTER_PREFIX} installing '{name}' using Poetry is not supported; skipping"
@@ -96,30 +167,85 @@ def _find_locked_dependencies(
return []
transients = [packages[name]]
for dep in packages[name].requires:
transients += find_transients(dep.name)
transients += find_deps_of_deps(dep.name)
return transients
return find_transients(top_level.name)
return find_deps_of_deps(top_level.name)
except KeyError:
if any(
delimiter in dependency_name for delimiter in _PEP508_VERSION_DELIMITERS
):
message = "specifying a version in the tox environment definition is incompatible with installing from a lockfile"
else:
message = (
"no version of the package was found in the current project's lockfile"
)
raise NoLockedDependencyError(
f"Cannot install requirement '{dependency_name}': {message}"
raise LockedDepVersionConflictError(
f"Locked dependency '{dependency_name}' cannot include version specifier"
) from None
raise LockedDepNotFoundError(
f"No version of locked dependency '{dependency_name}' found in the project lockfile"
) from None
def _install_env_dependencies(venv: ToxVirtualEnv, poetry: Poetry):
env_deps = _sort_env_deps(venv)
reporter.verbosity1(
f"{_REPORTER_PREFIX} updating env config with {len(env_deps.unlocked_deps)} unlocked env dependencies for installation using the default backend"
)
venv.envconfig.deps = env_deps.unlocked_deps
dependencies: List[PoetryPackage] = []
for dep in env_deps.locked_deps:
try:
dependencies += _find_transients(poetry, dep.name)
except ToxPoetryInstallerException as err:
venv.status = "install-locked-deps-failed"
reporter.error(f"{_REPORTER_PREFIX} {err}")
raise err
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dependencies)} actual dependencies from {len(venv.envconfig.deps)} specified env dependencies"
)
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(dependencies)} env dependencies from lockfile"
)
_install_to_venv(poetry, venv, dependencies)
def _install_package_dependencies(venv: ToxVirtualEnv, poetry: Poetry):
reporter.verbosity1(
f"{_REPORTER_PREFIX} performing installation of project dependencies"
)
primary_dependencies = poetry.locker.locked_repository(False).packages
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(primary_dependencies)} dependencies of project '{poetry.package.name}'"
)
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(primary_dependencies)} project dependencies from lockfile"
)
_install_to_venv(poetry, venv, primary_dependencies)
@hookimpl
def tox_testenv_install_deps(
venv: ToxVirtualEnv, action: ToxAction
) -> Optional[List[PoetryPackage]]:
def tox_addoption(parser: ToxParser):
"""Add required configuration options to the tox INI file
Adds the ``require_locked_deps`` configuration option to the venv to check whether all
dependencies should be treated as locked or not.
"""
parser.add_testenv_attribute(
name="require_locked_deps",
type="bool",
default=False,
help="Require all dependencies in the environment be installed using the Poetry lockfile",
)
@hookimpl
def tox_testenv_install_deps(venv: ToxVirtualEnv, action: ToxAction):
"""Install the dependencies for the current environment
Loads the local Poetry environment and the corresponding lockfile then pulls the dependencies
@@ -131,10 +257,13 @@ def tox_testenv_install_deps(
"""
if action.name == venv.envconfig.config.isolated_build_env:
# Skip running the plugin for the packaging environment. PEP-517 front ends can handle
# that better than we can, so let them do their thing. More to the point: if you're having
# problems in the packaging env that this plugin would solve, god help you.
reporter.verbosity1(
f"{_REPORTER_PREFIX} skipping isolated build env '{action.name}'"
)
return None
return
poetry = PoetryFactory().create_poetry(venv.envconfig.config.toxinidir)
@@ -142,36 +271,14 @@ def tox_testenv_install_deps(
f"{_REPORTER_PREFIX} loaded project pyproject.toml from {poetry.file}"
)
dependencies: List[PoetryPackage] = []
for env_dependency in venv.envconfig.deps:
dependencies += _find_locked_dependencies(poetry, env_dependency.name)
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dependencies)} actual dependencies from {len(venv.envconfig.deps)} specified env dependencies"
)
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(dependencies)} env dependencies from lockfile"
)
_install_to_venv(poetry, venv, dependencies)
# Handle the installation of any locked env dependencies from the lockfile
_install_env_dependencies(venv, poetry)
# Handle the installation of the package dependencies from the lockfile if the package is
# being installed to this venv; otherwise skip installing the package dependencies
if not venv.envconfig.skip_install:
reporter.verbosity1(
f"{_REPORTER_PREFIX} env specifies 'skip_install = false', performing installation of dev-package dependencies"
)
primary_dependencies = poetry.locker.locked_repository(False).packages
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(primary_dependencies)} dependencies of dev-package"
)
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(primary_dependencies)} dev-package dependencies from lockfile"
)
_install_to_venv(poetry, venv, primary_dependencies)
_install_package_dependencies(venv, poetry)
else:
reporter.verbosity1(
f"{_REPORTER_PREFIX} env specifies 'skip_install = true', skipping installation of dev-package package"
f"{_REPORTER_PREFIX} env specifies 'skip_install = true', skipping installation of project package"
)
return dependencies