73 Commits
0.4.0 ... 0.6.2

Author SHA1 Message Date
e3b3b19b31 Merge pull request #39 from enpaul/enp/bugfixes
Misc Bugfixes
2021-01-22 19:27:43 -05:00
99d6eedff1 Fix CI running all tox envs instead of targeted envs
Yaml formatting quirks strike again
2021-01-22 19:23:35 -05:00
ff09706648 Update changelog with v0.6.2 2021-01-22 19:18:54 -05:00
9ec64cf98f Bump patch version 2021-01-22 19:13:42 -05:00
59bf9ad26e Add precondition check for skipping tox self-provisioned env
Fixes #35
2021-01-22 19:13:42 -05:00
366c50ac87 Update transient dep resolution to always exclude root package name
Fixes #37
2021-01-22 19:13:42 -05:00
6e535f0f42 Merge pull request #38 from enpaul/enp/fix-ci
Fix CI Error
2021-01-22 18:39:07 -05:00
af66cd6d06 Fix changelog URL formatting 2021-01-22 18:28:12 -05:00
f6c8001546 Update py to 1.10+ to address CVE-2020-29651 2021-01-22 17:55:31 -05:00
b62e13afc7 Add link to changelog 2021-01-16 21:39:34 -05:00
c119e3b9d4 Add changelog
Fixes #36
2021-01-16 21:33:28 -05:00
0614913cc5 Merge pull request #34 from enpaul/enp/fix-unsafe
Fix unsafe dependency handling
2020-12-16 20:37:35 -05:00
f116ffefa2 Fix special handling of unsafe dependencies as primary dependencies 2020-12-16 20:31:41 -05:00
2ce97a5349 Bump patch version 2020-12-16 19:16:38 -05:00
e77c859355 Add additional logging to dependency processing functionality
Hopefully this will help with dependency resolution errors in the future
2020-12-16 19:14:03 -05:00
c1d1ac2de1 Update handling of poetrys unsafe dependencies to avoid keyerror
Fixes #33
2020-12-16 19:01:20 -05:00
c5c5261a80 Update transient processing to more narrowly scope error catching 2020-12-16 18:47:27 -05:00
ff344c2b4b Merge pull request #32 from enpaul/enp/docs
Beta > Alpha
2020-12-05 18:37:24 -05:00
a7d9b25b62 Fix broken links in readme 2020-12-05 18:31:44 -05:00
8356d52c4f Update project project status from alpha to beta 🎉 2020-12-05 18:31:44 -05:00
1941a103d3 Update contributor, devel, and roadmap documentation 2020-12-05 18:31:44 -05:00
0ad5fb7219 Overhaul basic usage documentation to improve clarity
Add intra document links
Update to document new/updated features
Add badge for downloads per month
Update badge order
2020-12-05 18:30:09 -05:00
ea518d1201 Reorder TOC
Add garbage to email to (hopefully) avoid scraping
2020-12-05 18:30:09 -05:00
03f46d34f3 Rewrite usage reference documentation to improve clarity
Add more details to error documentation
Update error and option documentation to be referenceable via slugs
Add docs for missing --require-poetry option
Add more crosslinks to assit with navigation
2020-12-05 18:30:09 -05:00
88ca772111 Merge pull request #31 from enpaul/enp/force-fail
Add option to support forcing tox failure when Poetry is not available
2020-12-05 15:11:31 -05:00
66c1925679 Add poetry requirement to local CI to ensure proper installation 2020-12-05 15:01:39 -05:00
99edc1c24e Add runtime option for forcing error if poetry is not found 2020-12-05 15:01:39 -05:00
39439f132a Merge pull request #30 from enpaul/enp/cleanup
Misc fixes ahead of push to beta classification
2020-12-05 14:19:18 -05:00
52aaeba93c Overhaul CI to improve resilance and efficiency
Add caching for pip and poetry downloads to reduce runtime
Add pinned pip version
Add poetry installation of local project
Remove bare pip install for local project installation
2020-12-05 14:12:50 -05:00
db761d49c1 Add poetry extra install to tox config to support CI 2020-12-05 12:36:09 -05:00
5a23c05f17 Add internally proxied poetry module to support runtime in non-Poetry envs
Add the _poetry submodule to support importing Poetry internals at runtime rather
than import time. This allows the plugin to be run, and either skipped or errored,
without crashing tox
2020-12-04 22:16:36 -05:00
604e60d567 Bump feature version 2020-12-04 17:57:44 -05:00
872f6b0892 Update poetry dependency to be installed as optional extra
When running 'poetry remove tox-poetry-installer' in the same env as
poetry is installed to, poetry will uninstall itself. This is, obviously,
not ideal.

This change makes poetry an optional dependency so that the plugin can be
installed (and uninstalled) alongside poetry in the same env without
breaking the poetry installation. The intention is that the plugin can be
installed with the 'poetry' extra when being installed to an isolated environment
where poetry is not otherwise available.

This is a mitigation of Issue #2 as an alternative to vendorization of the enitre
poetry project 😬
2020-12-04 17:57:38 -05:00
afad7663f0 Fix clean target not deleting pycache 2020-12-04 17:22:52 -05:00
01635c50c7 Update logging integration to improve standardization
Standardize language in logging messages
Move system version to constants

Fixes #3
2020-12-04 17:22:41 -05:00
bd8124dcbf Move non-hook function out of hooks submodule 2020-12-04 16:09:40 -05:00
4de5fea0e3 Merge pull request #29 from enpaul/enp/fix-platform
Fix dependency installation on incompatible platforms and with invalid python versions
2020-11-26 15:08:12 -05:00
33119df752 Add missing tests __init__ file
I'm tired of fighting this. Every tool under the sun wants the tests to
be an importable module, which doesn't make much sense to me, but it's
whatever. I'm just adding this because pylint, mypy, and others get
angry when I try to use them on just a directory without globbing
2020-11-26 15:01:59 -05:00
498cb4a3d2 Bump patch version 2020-11-26 14:54:34 -05:00
0a363d1848 Fix platform and py version compatibility errors
Add check for compatible python versions during dependency determination
Add check for compatible python platforms during dependency determination
Add verbosity2 logging messages for dependency determination
2020-11-26 14:54:34 -05:00
429992cff9 Bump patch version 2020-11-26 13:44:33 -05:00
216ee8b095 Fix exception when not installing project dependencies
Fix bug in precommit args
2020-11-26 13:43:33 -05:00
e96f4fb5e5 Add python3.9 tests to CI 2020-11-26 12:59:19 -05:00
eebd16383d Add python3.9 to test and classifier lists 2020-11-26 12:54:14 -05:00
add4ec62eb Update python requirement to ^3.6.1
Remove python specifiers from all deps that required >=3.6.1
2020-11-26 12:48:40 -05:00
0c25ca965e Update pre-commit config to use poetry environment
Add precommit hook for checking for merge conflicts
Add dev dep for blacken-docs
Add dev dep for pre-commit-hooks
Update all precommit hooks to use system language
Update toxfile to use new dev deps
Fix tox vs. precommit conflicts with reorder-python-imports
2020-11-26 12:41:06 -05:00
6a5e955fce Merge pull request #27 from enpaul/enp/unsafe
Refactor dep processing to improve efficiency of installation
2020-11-14 10:43:03 -05:00
d910b6ee8d Refactor dep processing to improve efficiency of installation
Assemble single list of dependencies to reduce duplication and reduce installation overhead
2020-11-12 22:14:49 -05:00
e37c166a8b Misc meta fixups
Fix dumping pypi API key to stdout when using "make publish" target
Fix outdated version in example output of docs
2020-11-12 21:13:55 -05:00
3d6d39eca8 Add note about installation via tox requires config option
Fixes #13
2020-11-12 21:12:38 -05:00
b9b0eba90f Merge pull request #26 from enpaul/enp/fixups
Minor fixups ahead of 0.5 release
2020-11-12 19:47:09 -05:00
df8312f5ee Bump feature version 2020-11-12 19:42:43 -05:00
1f102b16cb Add blocking functionality when using require_locked_deps
When require_locked_deps is true further processing will be blocked because
a non-null value is returned by this function
2020-11-12 19:42:43 -05:00
a7cde7a9ab Fix missing exception docs 2020-11-12 19:42:43 -05:00
b0bee11272 Merge pull request #25 from enpaul/enp/docs
Update documentation ahead of 0.5 release
2020-11-12 19:42:27 -05:00
accb4c3278 Update drawbacks section with new info 2020-11-12 19:31:31 -05:00
ea183553c4 Rewrite usage examples section into reference section
Fixes #20
2020-11-12 19:31:30 -05:00
5c5536581b Update quickstart docs with new config option usage with locked_deps 2020-11-12 18:45:02 -05:00
b6ef671e67 Merge pull request #23 from enpaul/enp/faster
Update config handling to support tox's native env change detection
2020-11-12 18:19:21 -05:00
99c10482fc Update tox config to use new plugin config option 2020-11-12 00:54:27 -05:00
b32a212e82 Restructure config options to support tox's native change detection
Remove custom handling of deps option
Add locked_deps option
Stop modifying the envconfig.deps option at runtime
2020-11-12 00:54:27 -05:00
b6415888d9 Merge pull request #24 from enpaul/enp/fix-ci
Fix automated CI
2020-11-12 00:52:18 -05:00
eba268a127 Update CI to include installation of this plugin
Not sure how I overlooked this
2020-11-12 00:44:12 -05:00
d8d483e849 Update to use poetry-core for PEP517 builds 2020-11-12 00:41:54 -05:00
c14313b7b0 Fix overlooked roadmap step that didn't get checked off 2020-11-12 00:32:22 -05:00
d1f161e0fa Merge pull request #22 from enpaul/enp/recursive
Fix recursive dependencies causing recursion error
2020-11-12 00:26:22 -05:00
2961b55c9a Fix recursive dependencies causing recursion error
Rewrite the find_dep_of_deps function to handle recursive dependencies
2020-11-11 23:07:14 -05:00
d4fb7046d8 Merge pull request #21 from enpaul/enp/modsplit
Split single-file module out into directory module
2020-11-11 22:49:30 -05:00
8c4e596316 Update tox automation to work with new module structure 2020-11-11 22:37:09 -05:00
50f6e3d151 Update dependencies to get cryptography security fix 2020-11-11 22:37:09 -05:00
106d1bf6cf Update metadata tests to load metadata from new location 2020-11-11 22:37:09 -05:00
b57b78d4e2 Add typing stub file 2020-11-11 22:37:08 -05:00
33e81f742a Split single module file out into directory module 2020-11-11 22:37:08 -05:00
21 changed files with 1655 additions and 983 deletions

72
.github/scripts/setup-env.sh vendored Executable file
View File

@@ -0,0 +1,72 @@
#!/usr/bin/env bash
#
# Environment setup script for the local project. Intended to be used with automation
# to create a repeatable local environment for tests to be run in. The python env
# this script creates can be accessed at the location defined by the CI_VENV variable
# below.
set -e;
# ##### Prereqs #####
#
# Set global vars for usage in the script, create the cache directory so we can rely
# on that existing, then dump some diagnostic info for later reference.
#
CI_VENV=$HOME/ci;
CI_CACHE=$HOME/.cache;
CI_CACHE_GET_POETRY="$CI_CACHE/get-poetry.py";
CI_POETRY=$HOME/.poetry/bin/poetry;
CI_VENV_PIP="$CI_VENV/bin/pip";
CI_VENV_PIP_VERSION=19.3.1;
CI_VENV_TOX="$CI_VENV/bin/tox";
mkdir --parents "$CI_CACHE";
command -v python;
python --version;
# ##### Install Poetry #####
#
# Download the poetry install script to the cache directory and then install poetry.
# After dump the poetry version for later reference.
#
curl https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py \
--output "$CI_CACHE_GET_POETRY" \
--silent \
--show-error \
--location;
python "$CI_CACHE_GET_POETRY" --yes 1>/dev/null;
python "$CI_POETRY" --version --no-ansi;
# ##### Setup Runtime Venv #####
#
# Create a virtual environment for poetry to use, upgrade pip in that venv to a pinned
# version, then install the current project to the venv.
#
# Note 1: Poetry, Tox, and this project plugin all use pip under the hood for package
# installation. This means that even though we are creating up to eight venvs
# during a given CI run they all share the same download cache.
# Note 2: The "VIRTUAL_ENV=$CI_VENV" prefix on the poetry commands below sets the venv
# that poetry will use for operations. There is no CLI flag for poetry that
# directs it to use a given environment, but if it finds itself in an existing
# environment it will use it and skip environment creation.
#
python -m venv "$CI_VENV";
$CI_VENV_PIP install "pip==$CI_VENV_PIP_VERSION" \
--upgrade \
--quiet;
VIRTUAL_ENV=$CI_VENV "$CI_POETRY" install \
--extras poetry \
--quiet \
--no-ansi \
&>/dev/null;
# ##### Print Debug Info #####
#
# Print the pip and tox versions (which will include registered plugins)
#
$CI_VENV_PIP --version;
echo "tox $($CI_VENV_TOX --version)";

View File

@@ -17,25 +17,54 @@ jobs:
toxenv: py37
- version: 3.8
toxenv: py38
- version: 3.9
toxenv: py39
steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python.version }}
- name: Checkout
uses: actions/checkout@v2
- name: Setup:python${{ matrix.python.version }}
uses: actions/setup-python@v1
with:
python-version: ${{ matrix.python.version }}
- name: Install tox
run: pip install "tox>=3.20.0,<3.21.0" --upgrade
- name: Run tests via ${{ matrix.python.toxenv }}
run: tox -e ${{ matrix.python.toxenv }}
- name: Setup:cache
uses: actions/cache@v2
with:
path: |
~/.cache/pip
~/.cache/pypoetry/cache
~/.poetry
# Including the hashed poetry.lock in the cache slug ensures that the cache
# will be invalidated, and thus all packages will be redownloaded, if the
# lockfile is updated
key: ${{ runner.os }}-${{ matrix.python.toxenv }}-${{ hashFiles('**/poetry.lock') }}
- name: Setup:env
run: .github/scripts/setup-env.sh
- name: Run:${{ matrix.python.toxenv }}
run: $HOME/ci/bin/tox -e ${{ matrix.python.toxenv }} --require-poetry
Check:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up Python 3.8
- name: Checkout
uses: actions/checkout@v2
- name: Setup:python3.8
uses: actions/setup-python@v1
with:
python-version: 3.8
- name: Install tox requirements
run: pip install "tox>=3.20.0,<3.21.0" --upgrade
- name: Run meta checks
run: tox -e static -e static-tests -e security
- name: Setup:cache
uses: actions/cache@v2
with:
path: |
~/.cache/pip
~/.cache/pypoetry/cache
~/.poetry
# Hardcoded 'py38' slug here lets this cache piggyback on the 'py38' cache
# that is generated for the tests above
key: ${{ runner.os }}-py38-${{ hashFiles('**/poetry.lock') }}
- name: Setup:env
run: .github/scripts/setup-env.sh
- name: Run:static
run: $HOME/ci/bin/tox -e static --require-poetry
- name: Run:static-tests
run: $HOME/ci/bin/tox -e static-tests --require-poetry
- name: Run:security
run: $HOME/ci/bin/tox -e security --require-poetry

View File

@@ -1,28 +1,48 @@
---
# All of the pre-commit hooks here actually use the `pytyhon` pre-commit language
# setting. However, for the python language setting, pre-commit will create and manage
# a cached virtual environment for each hook ID and do a bare `pip install <repo>` into
# the venv to setup the hook. This can result in conflicting dependency versions between
# the version installed to the pre-commit venv and the version installed to the Poetry
# venv specified in the lockfile.
#
# The solution is to specify `language: system` for all hooks and then install the
# required dependencies to the Poetry venv. The `system` language skips the isolated
# venv creation and looks for the entrypoint specified by the hook in the global
# environment which, if running in the Poetry venv, will find the entrypoint provided
# by the Poetry-managed dependency.
#
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v3.3.0
hooks:
- id: end-of-file-fixer
language: system
- id: fix-encoding-pragma
args:
- "--remove"
language: system
- id: trailing-whitespace
language: system
- id: check-merge-conflict
language: system
- repo: https://github.com/psf/black
rev: 20.8b1
hooks:
- id: black
language_version: python3
language: system
- repo: https://github.com/asottile/blacken-docs
rev: v0.5.0
rev: v1.8.0
hooks:
- id: blacken-docs
additional_dependencies: [black==20.8b1]
language_version: python3
language: system
- repo: https://github.com/asottile/reorder_python_imports
rev: v2.3.5
rev: v2.3.6
hooks:
- id: reorder-python-imports
language_version: python3
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v2.0.0
hooks:
- id: end-of-file-fixer
- id: fix-encoding-pragma
args: [--remove]
- id: trailing-whitespace
args:
- "--unclassifiable-application-module=tox_poetry_installer"
language: system

228
CHANGELOG.md Normal file
View File

@@ -0,0 +1,228 @@
# changelog
See also: [Github Release Page](https://github.com/enpaul/tox-poetry-installer/releases).
## Version 0.6.2
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.6.2/)
* Update locked version of `py` to `1.10.0` to address
[CVE-2020-29651](https://nvd.nist.gov/vuln/detail/CVE-2020-29651)
* Fix dependency identification failing when the package under test is a transient dependency
of a locked dependency specified for installation
* Fix `AttributeError` being raised while creating the Tox self-provisioned environment when
using either the [`minversion`](https://tox.readthedocs.io/en/latest/config.html#conf-minversion)
or [`requires`](https://tox.readthedocs.io/en/latest/config.html#conf-requires) Tox config options
## Version 0.6.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.6.1/)
* Update logging around transient dependency processing to improve debugging of dependency
installation problems
* Fix regression around handling of Poetry's unsafe packages when the unsafe package is a
transient dependency ([#33](https://github.com/enpaul/tox-poetry-installer/issues/33))
* Fix handling of Poetry's unsafe packages when the unsafe package is a primary (environment
or package) dependency
## Version 0.6.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.6.0/)
* Add `poetry` extra to support installing Poetry as a direct dependency of the plugin
* Add `--require-poetry` runtime option to force Tox failure if Poetry is not installed
* Update logging messages to improve UX around non-verbose messaging
* Update error logging to avoid dumping stack traces
* Update integration with Tox's `action` object to better manage internal state at runtime
* Update documentation to more clearly cover more use cases
* Remove `poetry` as a required dependency to support external Poetry installations
First beta release :tada:
## Version 0.5.2
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.5.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.5.2/)
* Fix always attempting to install dependencies with incompatible python version constraints
* Fix always attempting to install dependencies with incompatible python platforms
## Version 0.5.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.5.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.5.1/)
* Add CI/Tox tests for Python-3.9
* Update dependency processing to reduce duplication during installation
* Update minimum python requirement to `3.6.1`
* Fix `UnboundLocal` exception when not installing project dependencies
## Version 0.5.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.5.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.5.0/)
* Add option `locked_deps` to better support both locked and unlocked dependencies in a
single environment
* Add blocking functionality when using `require_locked_deps = true` to prevent other
hooks from running after this one
* Update documentation to include new configuration options and errors
* Update documentation to improve future maintainability
* Update module structure to move from single-file module to multi-file directory module
* Fix `RecursionError` when installing locked dependencies that specify recursive dependencies
* Fix always reinstalling all locked dependencies on every run regardless of update status
## Version 0.4.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.4.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.4.0/)
* Add `install_dev_deps` configuration option for automatically installing all Poetry
dev-dependencies into a Tox testenv
## Version 0.3.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.3.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.3.1/)
* Fix error when installing an environment with no extras specified in the configuration
* Fix problem where only the dependencies of the sequentially last extra would be installed
* Fix regression causing no project dependencies to be installed
## Version 0.3.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.3.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.3.0/)
* Add support for the Tox [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras)
configuration parameter
* Update runtime-skip-conditional checks to improve clarity and ease of future maintenance
* Update lockfile parsing to avoid parsing it multiple times for a single testenv
* Fix missing `poetry-core` dependency when using Poetry<1.1.0
## Version 0.2.4
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.4),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.4/)
* Fix support for Poetry-1.1 ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
* Include tests in sdist ([#8](https://github.com/enpaul/tox-poetry-installer/issues/8))
## Version 0.2.3
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.3),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.3/)
* Fix usage of the plugin in non-Poetry based projects ([#1](https://github.com/enpaul/tox-poetry-installer/issues/1))
* Fix treating dependency names as case sensitive when they shouldn't be ([#7](https://github.com/enpaul/tox-poetry-installer/issues/7))
## Version 0.2.2
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.2/)
* Fix breaking when running Tox in projects that do not use Poetry for their environment/dependency
management ([#1](https://github.com/enpaul/tox-poetry-installer/issues/1))
## Version 0.2.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.1/)
* Fix duplicate installation of transient environment dependencies
* Fix logging error indicating all environments always have zero dependencies
* Fix installing main dependencies when `skip_install` is false but `skipdist` is true
## Version 0.2.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.2.0),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.2.0/)
* Add support for per-environment configuration setting `require_locked_deps`
* Add support for per-dependency lock requirement setting using `@poetry` suffix
* Add support for coexisting locked and unlocked dependencies in a single test environment
* Update documentation to include more usage examples
* Update documentation to improve clarity around problems and drawbacks
* Fix logging messages being inconsistently formatted
* Fix raising the same exception for "locked dependency not found" and "locked dependency
specifies alternate version" errors
* Fix plugin errors not reporting to Tox that they happened
* Fix plugin errors not causing Tox to mark the env as failed
## Version 0.1.3
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.1.3),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.1.3/)
* Fix core functionality of installing dependencies from lockfile for the package-under-development
("dev-package") built by Tox
* Fix log messages not being displayed with Tox output
* Add additional logging output for diagnostics
* Update Poetry requirement to exclude upcoming Poetry-1.1.0 release which will break compatibility
This is the first release where the core functionality actually works as expected :tada:
## Version 0.1.2
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.1.2),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.1.2/)
* Test trivial functionality on Python-3.6 and Python-3.7
* Fix disagreement between `pyproject.toml` and module metadata on what the current version is
* Fix constant named for PEP-440 that should have been named for PEP-508
## Version 0.1.1
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.1.1),
[PyPI](https://pypi.org/project/tox-poetry-installer/0.1.1/)
* Add/update project documentation
* Add static analysis and formatting enforcement automation to toxfile
* Add security analysis to toxfile
* Fix raising `KeyError` for unlocked dependencies
* Fix mishandling of Poetry's "unsafe dependencies"
* Lint, blacken, and generally improve code quality
## Version 0.1.0
View this release on:
[Github](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.1.0),
* Add support for installing Tox environment dependencies using Poetry from the Poetry lockfile

View File

@@ -19,7 +19,7 @@ clean-py:
rm --recursive --force ./dist
rm --recursive --force ./build
rm --recursive --force ./*.egg-info
rm --recursive --force __pycache__/
rm --recursive --force ./**/__pycache__/
clean: clean-tox clean-py; ## Clean temp build/cache files and directories
@@ -33,4 +33,4 @@ test: ## Run the project testsuite(s)
poetry run tox --recreate
publish: wheel source ## Build and upload to pypi (requires $PYPI_API_KEY be set)
poetry publish --username __token__ --password $(PYPI_API_KEY)
@poetry publish --username __token__ --password $(PYPI_API_KEY)

628
README.md
View File

@@ -3,21 +3,28 @@
A plugin for [Tox](https://tox.readthedocs.io/en/latest/) that allows test environment
dependencies to be installed using [Poetry](https://python-poetry.org/) from its lockfile.
⚠️ **This project is alpha software and should not be used in production environments** ⚠️
⚠️ **This project is beta software and is under active development** ⚠️
[![ci-status](https://github.com/enpaul/tox-poetry-installer/workflows/CI/badge.svg?event=push)](https://github.com/enpaul/tox-poetry-installer/actions)
[![license](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT)
[![pypi-version](https://img.shields.io/pypi/v/tox-poetry-installer)](https://pypi.org/project/tox-poetry-installer/)
[![pypi-downloads](https://img.shields.io/pypi/dm/tox-poetry-installer)](https://libraries.io/pypi/tox-poetry-installer)
[![license](https://img.shields.io/pypi/l/tox-poetry-installer)](https://opensource.org/licenses/MIT)
[![python-versions](https://img.shields.io/pypi/pyversions/tox-poetry-installer)](https://www.python.org)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
See the [Changelog](https://github.com/enpaul/tox-poetry-installer/blob/devel/CHANGELOG.md) for release history.
**Documentation**
* [Installation](#installation)
* [Introduction](#introduction)
* [Install](#install)
* [Quick Start](#quick-start)
* [Usage Examples](#usage-examples)
* [Known Drawbacks and Problems](#known-drawbacks-and-problems)
* [Why would I use this?](#why-would-i-use-this) (What problems does this solve?)
* [Reference](#reference)
* [Configuration Options](#configuration-options)
* [Command-line Arguments](#command-line-arguments)
* [Errors](#errors)
* [Advanced Usage](#advanced-usage)
* [Developing](#developing)
* [Contributing](#contributing)
* [Roadmap](#roadmap)
@@ -27,292 +34,155 @@ dependencies to be installed using [Poetry](https://python-poetry.org/) from its
Related resources:
* [Poetry Python Project Manager](https://python-poetry.org/)
* [Tox Automation Project](https://tox.readthedocs.io/en/latest/)
* [Poetry Dev-Dependencies Tox Plugin](https://github.com/sinoroc/tox-poetry-dev-dependencies)
* [Poetry Tox Plugin](https://github.com/tkukushkin/tox-poetry)
* [Other Tox plugins](https://tox.readthedocs.io/en/latest/plugins.html)
Similar projects:
* [Poetry Dev-Dependencies Tox Plugin](https://github.com/sinoroc/tox-poetry-dev-dependencies)
* [Poetry Tox Plugin](https://github.com/tkukushkin/tox-poetry)
## Installation
Add the plugin as a development dependency of a Poetry project:
## Introduction
```
~ $: poetry add tox-poetry-installer --dev
This is a plugin to unify two great projects in the Python ecosystem: the
[Tox](https://tox.readthedocs.io/en/latest/) automation project and the
[Poetry](https://python-poetry.org) project/dependency manager. Specifically it allows
the repeatable dependency resolution and installation tools that Poetry uses to benefit
the isolated environments that Tox uses to run automated tests. The motivation to write
this plugin came from a need for a single source of truth for the versions of all
packages that should be installed to an environment.
When in use this plugin will allow a Tox environment to install its required
dependencies using the versions specified in the Poetry lockfile. This eliminates
needing to specify package versions in multiple places as well as ensures that the Tox
environment has the exact same versions of a given package as the Poetry environment.
This reduces (or hopefully eliminates) hard to debug problems caused by subtle
differences in the dependency graph of the active development environment (the one managed
by Poetry) and the automated test environment(s) created by Tox.
To learn more about the problems this plugin aims to solve jump ahead to
[What problems does this solve?](#why-would-i-use-this).
Otherwise keep reading to get started.
### Install
The recommended way to install the plugin is to add it to a project's `pyproject.toml`
and lockfile using Poetry:
```bash
poetry add tox-poetry-installer[poetry] --dev
```
Confirm that the plugin is installed, and Tox recognizes it, by checking the Tox version:
**WARNING:** The below installation methods are vulnerable to the
[transient dependency issues this plugin aims to avoid](#why-would-i-use-this). It is
always recommended to install dependencies using Poetry whenever possible.
The plugin can also be installed with pip directly, though it is recommended to always
install to a virtual environment and pin to a specific version:
```bash
source my-venv/bi/activate
pip install tox-poetry-installer[poetry] == 0.6.0
```
The plugin can also be installed using the Tox
[`requires`]((https://tox.readthedocs.io/en/latest/config.html#conf-requires))
configuration option. Note however that dependencies installed via the `requires` option
are not handled by the plugin and will be installed the same way as a `pip install ...`
above. For this reason it is also recommended to always pin to a specific version when
using this installation method:
```ini
# tox.ini
[tox]
requires
tox-poetry-installer[poetry] == 0.6.0
```
Check that the plugin is registered by checking the Tox version:
```
~ $: poetry run tox --version
3.20.0 imported from .venv/lib64/python3.8/site-packages/tox/__init__.py
registered plugins:
tox-poetry-installer-0.2.2 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py
tox-poetry-installer-0.6.0 at .venv/lib64/python3.8/site-packages/tox_poetry_installer.py
```
If using Pip, ensure that the plugin is installed to the same environment as Tox:
**NOTE:** Installing the `tox-poetry-installer[poetry]` extra will add the `poetry`
package as a managed environment dependency which can cause problems when the Poetry
installation is externally managed (such as in a CI or container environment). See
[Advanced Usage](#installing-alongside-an-existing-poetry-installation) for more
information on this use case.
```
# Calling the virtualenv's 'pip' binary directly will cause pip to install to that virtualenv
~ $: /path/to/my/automation/virtualenv/bin/pip install tox
~ $: /path/to/my/automation/virtualenv/bin/pip install tox-poetry-installer
```
### Quick Start
Before making any changes to `tox.ini` the project is already benefiting from having
the plugin installed: all dependencies of the root project package are installed using
the Poetry backend to all Tox environments that install the root package without any
configuration changes.
## Quick Start
To require a Tox environment install all it's dependencies from the Poetry lockfile, add the
`require_locked_deps = true` option to the environment configuration and remove all version
specifiers from the dependency list. The versions to install will be taken from the lockfile
directly:
To add dependencies from the lockfile to a Tox environment, add the option
[`locked_deps`](#locked_deps) to the environment configuration and list names of
dependencies (with no version specifier) under it:
```ini
[testenv]
description = Run the tests
require_locked_deps = true
deps =
pytest
pytest-cov
description = Some very cool tests
locked_deps =
black
pylint
mypy
commands = ...
```
To require specific dependencies be installed from the Poetry lockfile, and let the rest be
installed using the default Tox installation backend, add the suffix `@poetry` to the dependencies.
In the example below the `pytest`, `pytest-cov`, and `black` dependencies will be installed from
the lockfile while `pylint` and `mypy` will be installed using the versions specified here:
The standard [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) option
can be used in parallel with the `locked_deps` option to install unlocked dependencies
(dependencies not in the lockfile) alongside locked dependencies:
```ini
[testenv]
description = Run the tests
require_locked_deps = true
description = Some very cool tests
locked_deps =
black
pylint
mypy
deps =
pytest@poetry
pytest-cov@poetry
black@poetry
pylint >=2.5.0
mypy == 0.770
pytest == 6.1.1
pytest-cov >= 2.10, <2.11
commands = ...
```
Alternatively, to quickly install all Poetry dev-dependencies to a Tox environment, add the
`install_dev_deps = true` option to the environment configuration. This option can be used either
with the `require_locked_deps = true` option or without it
**Note:** Regardless of the settings outlined above, all dependencies of the project package (the
one Tox is testing) will always be installed from the lockfile.
## Usage Examples
After installing the plugin to a project your Tox automation is already benefiting from the
lockfile: when Tox installs your project package to one of your environments, all the dependencies
of your project package will be installed using the versions specified in the lockfile. This
happens automatically and requires no configuration changes.
But what about the rest of your Tox environment dependencies?
Let's use an example `tox.ini` file, below, that defines two environments: the main `testenv` for
running the project tests and `testenv:check` for running some other helpful tools:
```ini
[tox]
envlist = py37, static
isolated_build = true
[testenv]
description = Run the tests
deps =
pytest == 5.3.0
commands = ...
[testenv:check]
description = Static formatting and quality enforcement
deps =
pylint >=2.4.4,<2.6.0
mypy == 0.770
black --pre
commands = ...
```
Let's focus on the `testenv:check` environment first. In this project there's no reason that any
of these tools should be a different version than what a human developer is using when installing
from the lockfile. We can require that these dependencies be installed from the lockfile by adding
the option `require_locked_deps = true` to the environment config, but this will cause an error:
```ini
[testenv:check]
description = Static formatting and quality enforcement
require_locked_deps = true
deps =
pylint >=2.4.4,<2.6.0
mypy == 0.770
black --pre
commands = ...
```
Running Tox using this config gives us this error:
```
tox_poetry_installer.LockedDepVersionConflictError: Locked dependency 'pylint >=2.4.4,<2.6.0' cannot include version specifier
```
This is because we told the Tox environment to require all dependencies be locked, but then also
specified a specific version constraint for Pylint. With the `require_locked_deps = true` setting
Tox expects all dependencies to take their version from the lockfile, so when it gets conflicting
information it errors. We can fix this by simply removing all version specifiers from the
environment dependency list:
```ini
[testenv:check]
description = Static formatting and quality enforcement
require_locked_deps = true
deps =
pylint
mypy
black
commands = ...
```
Now all the dependencies will be installed from the lockfile. If Poetry updates the lockfile with
a new version then that updated version will be automatically installed when the Tox environment is
recreated.
Now let's look at the `testenv` environment. Let's make the same changes to the `testenv`
environment that we made to `testenv:check` above; remove the PyTest version and add
`require_locked_deps = true`. Then imagine that we want to add the
[Requests](https://requests.readthedocs.io/en/master/) library to the test environment: we
can add `requests` as a dependency of the test environment, but this will cause an error:
[`install_dev_deps`](#install_dev_deps) option to the environment configuration:
```ini
[testenv]
description = Run the tests
require_locked_deps = true
deps =
pytest
requests
commands = ...
description = Some very cool tests
install_dev_deps = true
```
Running Tox with this config gives us this error:
See the [Reference](#reference) section for more details on available
configuration options and the [Advanced Usage](#advanced-usage) section for some
unusual use cases.
```
tox_poetry_installer.LockedDepNotFoundError: No version of locked dependency 'requests' found in the project lockfile
```
This is because `requests` is not in our lockfile yet. Tox will refuse to install a dependency
that isn't in the lockfile to an an environment that specifies `require_locked_deps = true`. We
can fix this by running `poetry add requests --dev` to add it to the lockfile.
Now let's combine dependencies from the lockfile with dependencies that are
specified in-line in the Tox environment configuration.
[This isn't generally recommended](#why-would-i-use-this), but it is a valid use case and
fully supported by this plugin. Let's modify the `testenv` configuration to install PyTest
from the lockfile but then install an older version of the Requests library.
The first thing to do is remove the `require_locked_deps = true` setting so that we can install
Requests as an unlocked dependency. Then we can add our version specifier to the `requests`
entry in the dependency list:
```ini
[testenv]
description = Run the tests
deps =
pytest
requests >=2.2.0,<2.10.0
commands = ...
```
However we still want `pytest` to be installed from the lockfile, so the final step is to tell Tox
to install it from the lockfile by adding the suffix `@poetry` to the `pytest` entry in the
dependency list:
```ini
[testenv]
description = Run the tests
deps =
pytest@poetry
requests >=2.2.0,<2.10.0
commands = ...
```
Now when the `testenv` environment is created it will install PyTest (and all of its dependencies)
from the lockfile while it will install Requests (and all of its dependencies) using the default
Tox installation backend.
## Known Drawbacks and Problems
* The following `tox.ini` configuration options have no effect on the dependencies installed from
the Poetry lockfile (note that they will still affect unlocked dependencies):
* [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command)
* [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre)
* [`downloadcache`](https://tox.readthedocs.io/en/latest/config.html#conf-downloadcache) (deprecated)
* [`download`](https://tox.readthedocs.io/en/latest/config.html#conf-download)
* [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* Tox environments automatically inherit their settings from the main `testenv` environment. This
means that if the `require_locked_deps = true` is specified for the `testenv` environment then
all environments will also require locked dependencies. This can be overwritten by explicitly
specifying `require_locked_deps = false` on child environments where unlocked dependencies are
needed.
* There are a handful of packages that cannot be installed from the lockfile, whether as specific
dependencies or as transient dependencies (dependencies of dependencies). This is due to
[an ongoing discussion in the Poetry project](https://github.com/python-poetry/poetry/issues/1584);
the list of dependencies that cannot be installed from the lockfile can be found
[here](https://github.com/python-poetry/poetry/blob/cc8f59a31567f806be868aba880ae0642d49b74e/poetry/puzzle/provider.py#L55).
This plugin will skip these dependencies entirely, but log a warning when they are encountered.
## Why would I use this?
**Introduction**
The lockfile is a file generated by a package manager for a project that records what
dependencies are installed, the versions of those dependencies, and any additional metadata that
the package manager needs to recreate the local project environment. This allows developers
to have confidence that a bug they are encountering that may be caused by one of their
dependencies will be reproducible on another device. In addition, installing a project
environment from a lockfile gives confidence that automated systems running tests or performing
builds are using the same environment as a developer.
[Poetry](https://python-poetry.org/) is a project dependency manager for Python projects, and
so it creates and manages a lockfile so that its users can benefit from all the features
described above. [Tox](https://tox.readthedocs.io/en/latest/#what-is-tox) is an automation tool
that allows Python developers to run tests suites, perform builds, and automate tasks within
self-contained [Python virtual environments](https://docs.python.org/3/tutorial/venv.html).
To make these environments useful Tox supports installing dependencies in each environment.
However, since these environments are created on the fly and Tox does not maintain a lockfile,
there can be subtle differences between the dependencies a developer is using and the
dependencies Tox uses.
This is where this plugin comes into play.
By default Tox uses [Pip](https://docs.python.org/3/tutorial/venv.html) to install the
PEP-508 compliant dependencies to a test environment. This plugin extends the default
Tox dependency installation behavior to support installing dependencies using a Poetry-based
installation method that makes use of the dependency metadata from Poetry's lockfile.
### Why would I use this?
**The Problem**
By default Tox uses Pip to install the [PEP-508](https://www.python.org/dev/peps/pep-0508/)
compliant dependencies to a test environment. This plugin extends the default Tox
dependency installation behavior to support installing dependencies using a Poetry-based
installation method that makes use of the dependency metadata from Poetry's lockfile.
Environment dependencies for a Tox environment are usually specified in PEP-508 format, like
the below example:
```ini
# from tox.ini
...
[testenv]
description = Some very cool tests
deps =
foo == 1.2.3
bar >=1.3,<2.0
baz
...
```
Let's assume these dependencies are also useful during development, so they can be added to the
@@ -366,32 +236,254 @@ less stable than the one presented above because it does not specify any version
dependencies:
```ini
# from tox.ini
...
[testenv]
description = Some very cool tests
require_locked_deps = true
deps =
locked_deps =
foo
bar
baz
...
```
However with the `tox-poetry-installer` plugin installed the `require_locked_deps = true`
setting means that Tox will install these dependencies from the Poetry lockfile so that the
version installed to the Tox environment exactly matches the version Poetry is managing. When
`poetry update` updates the lockfile with new versions of these dependencies, Tox will
automatically install these new versions without needing any changes to the configuration.
However with the `tox-poetry-installer` plugin installed Tox will install these
dependencies from the Poetry lockfile so that the version installed to the Tox
environment exactly matches the version Poetry is managing. When `poetry update` updates
the lockfile with new versions of these dependencies, Tox will automatically install
these new versions without needing any changes to the configuration.
## Reference
### Configuration Options
All options listed below are Tox environment options and can be applied to one or more
environment sections of the `tox.ini` file. They cannot be applied to the global Tox
configuration section.
**NOTE:** Environment settings applied to the main `testenv` environment will be
inherited by child environments (i.e. `testenv:foo`) unless they are explicitly
overridden by the child environment's configuration.
#### `locked_deps`
* **Type:** multi-line list
* **Default:** `[]`
Names of packages in the Poetry lockfile to install to the Tox environment. All
dependencies specified here will be installed to the Tox environment using the details
given by the Poetry lockfile.
#### `require_locked_deps`
* **Type:** boolean
* **Default:** `false`
Whether the environment should allow unlocked dependencies (dependencies not in the
Poetry lockfile) to be installed alongside locked dependencies. If `true` then an error
will be raised if the environment specifies unlocked dependencies to install and the
plugin will block any other plugins from using the
[`tox_testenv_install_deps`](https://tox.readthedocs.io/en/latest/plugins.html#tox.hookspecs.tox_testenv_install_deps)
hook.
#### `install_dev_deps`
* **Type:** boolean
* **Default:** `false`
Whether all Poetry dev-dependencies should be installed to the environment. If `true`
then all dependencies specified in the
[`dev-dependencies`](https://python-poetry.org/docs/pyproject/#dependencies-and-dev-dependencies)
section of `pyproject.toml` will be installed automatically.
### Command-line Arguments
All arguments listed below can be passed to the `tox` command to modify runtime behavior
of the plugin.
#### `--require-poetry`
Indicates that Poetry is expected to be available to Tox and, if it is not, then the Tox
run should fail. If provided and the `poetry` package is not installed to the same
environment as the `tox` package then Tox will fail.
**NOTE:** See [Advanced Usage](#installing-alongside-an-existing-poetry-installation)
for more information.
### Errors
If the plugin encounters an error while processing a Tox environment then it will mark
the environment as failed and set the environment status to one of the values below:
**NOTE:** In addition to the reasons noted below, the plugin can encounter errors if the
Poetry lockfile is not up-to-date with `pyproject.toml`. To resynchronize the
lockfile with the `pyproject.toml` run one of
[`poetry update`](https://python-poetry.org/docs/cli/#update) or
[`poetry lock`](https://python-poetry.org/docs/cli/#lock)
#### Poetry Not Installed Error
* **Status value:** `PoetryNotInstalledError`
* **Cause:** Indicates that the `poetry` module could not be imported from the same
environment as the running `tox` module and the runtime flags specified
[`--require-poetry`](#--require-poetry).
* **Resolution options:**
* Install Poetry: ensure that `poetry` is installed to the same environment as `tox`.
* Skip running the plugin: remove the `--require-poetry` flag from the runtime options.
**NOTE:** See [Advanced Usage](#installing-alongside-an-existing-poetry-installation)
for more information.
#### Locked Dependency Version Conflict Error
* **Status value:** `LockedDepVersionConflictError`
* **Cause:** Indicates that a dependency specified in the [`locked_deps`](#locked_deps)
configuration option in `tox.ini` includes a
[PEP-508 version specifier](https://www.python.org/dev/peps/pep-0508/#grammar)
(i.e. `pytest >=6.0, <6.1`).
* **Resolution options:**
* Use the dependency version from the lockfile: remove any/all version specifiers
from the item in the `locked_deps` list in `tox.ini`.
* Do not install the dependency: remove the item from the `locked_deps` list in
`tox.ini`.
#### Locked Dependency Not Found Error
* **Status value:** `LockedDepNotFoundError`
* **Cause:** Indicates that a dependency specified in the [`locked_deps`](#locked_deps)
configuration option in `tox.ini` could not be found in the Poetry lockfile.
* **Resolution options:**
* Add the dependency to the lockfile: run
[`poetry add <dependency>`](https://python-poetry.org/docs/cli/#add).
* Do not install the dependency: remove the item from the `locked_deps` list in
`tox.ini`.
#### Extra Not Found Error
* **Status value:** `ExtraNotFoundError`
* **Cause:** Indicates that the [`extras`](https://tox.readthedocs.io/en/latest/config.html#conf-extras)
configuration option specified a setuptools extra that is not configured by Poetry in
`pyproject.toml`
* **Resolution options:**
* Configure the extra: add a section for the named extra to the
[`extras`](https://python-poetry.org/docs/pyproject/#extras) section of
`pyproject.toml` and optionally assign dependencies to the named extra using the
[`--optional`](https://python-poetry.org/docs/cli/#options_3) dependency setting.
* Remove the extra: remove the item from the `extras` list in `tox.ini`.
#### Locked Dependencies Required Error
* **Status value:** `LockedDepsRequiredError`
* **Cause:** Indicates that an environment with the [`require_locked_deps`](#require_locked_deps)
configuration option also specified unlocked dependencies using
[`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps) option in
`tox.ini`.
* **Resolution options:**
* Remove all unlocked dependencies: remove the `deps` configuration option in
`tox.ini`.
* Allow unlocked dependencies: remove the `require_locked_deps` configuration option
in `tox.ini` or explicitly set `require_locked_deps = false`.
### Advanced Usage
#### Unsupported Tox configuration options
The `tox.ini` configuration options listed below have no effect on the dependencies
installed by this plugin the Poetry lockfile. Note that these settings will still be
applied by the default Tox installation backend when installing unlocked dependencies
using the built-in `deps` option.
* [`install_command`](https://tox.readthedocs.io/en/latest/config.html#conf-install_command)
* [`pip_pre`](https://tox.readthedocs.io/en/latest/config.html#conf-pip_pre)
* [`download`](https://tox.readthedocs.io/en/latest/config.html#conf-download)
* [`indexserver`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
* [`usedevelop`](https://tox.readthedocs.io/en/latest/config.html#conf-indexserver)
All of these options are obsoleted by using the Poetry backend. If a given package
installs successfully using Poetry (using either `poetry add <package>` or
`poetry install`) then the required configuration options are already properly set in
the Poetry configuration and the plugin will automatically use the same settings when
installing the package.
#### Reinstalling locked dependencies to a Tox environment
Updating the `poetry.lock` file will not automatically cause Tox to install the updated
lockfile specifications to the Tox environments that specify them.
The Tox environment(s) with updated locked dependencies must be deleted and recreated
using the [`--recreate`](https://tox.readthedocs.io/en/latest/config.html#cmdoption-tox-r)
runtime flag. Alternatively Tox can be configured to always recreate an environment by
setting the [`recreate`](https://tox.readthedocs.io/en/latest/config.html#conf-recreate)
option in `tox.ini`.
#### Installing Poetry's unsafe dependencies
There are several packages that cannot be installed from the lockfile because they are
excluded by Poetry itself. As a result these packages cannot be installed by this plugin
either as environment dependencies (passed directly to [`locked_deps`](#locked_deps)) or
as transient dependencies (a dependency of a locked dependency).
As of [Poetry-1.1.4](https://github.com/python-poetry/poetry/releases/tag/1.1.4) there
are four packages classified as "unsafe" by Poetry and excluded from the lockfile:
* `setuptools`
* `distribute`
* `pip`
* `wheel`
When one of these packages is encountered by the plugin a warning will be logged and
_**the package will not be installed to the environment**_. If the unsafe package
is required for the environment then it will need to be specified as an unlocked
dependency using the [`deps`](https://tox.readthedocs.io/en/latest/config.html#conf-deps)
configuration option in `tox.ini`, ideally with an exact pinned version.
* The set of packages excluded from the Poetry lockfile can be found in
[`poetry.puzzle.provider.Provider.UNSAFE_DEPENDENCIES`](https://github.com/python-poetry/poetry/blob/master/poetry/puzzle/provider.py)
* There is an ongoing discussion of Poetry's handling of these packages at
[python-poetry/poetry#1584](https://github.com/python-poetry/poetry/issues/1584)
#### Installing alongside an existing Poetry installation
The plugin specifies the `poetry` package as an optional dependency to support an
externally managed Poetry installation such as in a container or CI environment. This
gives greater flexibility when using Poetry arguments like `--no-root`, `--no-dev`, or
`--remove-untracked` which can cause Poetry to uninstall itself if Poetry is specified
as a dependency of one of the packages it is managing (like this plugin).
To have the plugin use the externally-managed Poetry package simply do not install the
`poetry` extra when installing this plugin:
```bash
# Installing Poetry as a dependency with the plugin
poetry add tox-poetry-installer[poetry]
# Relying on an externally managed Poetry installation
poetry add tox-poetry-installer
```
Note that Poetry is an optional dependency to support this use case _only_: Poetry must
be installed to the same environment as Tox for the plugin to function. To check that
the local environment has all of the required modules in scope run the below command:
```bash
python -c '\
import tox;\
import tox_poetry_installer;\
from poetry.poetry import Poetry;\
'
```
**NOTE:** To force Tox to fail if Poetry is not installed, run the `tox` command with
the [`--require-poetry`](#--require-poetry) option.
## Developing
This project requires a developer to have Poetry version 1.0+ installed on their workstation, see
This project requires Poetry version 1.0+ on the development workstation, see
the [installation instructions here](https://python-poetry.org/docs/#installation).
Local environment setup instructions:
```bash
# Clone the repository...
# ...over HTTPS
@@ -401,7 +493,7 @@ git clone git@github.com:enpaul/tox-poetry-installer.git
# Create a the local project virtual environment and install dependencies
cd tox-poetry-installer
poetry install
poetry install -E poetry
# Install pre-commit hooks
poetry run pre-commit install
@@ -410,11 +502,18 @@ poetry run pre-commit install
poetry run tox
```
**NOTE:** Because the pre-commit hooks require dependencies in the Poetry environment it
is recommend to [launch an environment shell](https://python-poetry.org/docs/cli/#shell)
when developing the project. Alternatively, many `git` commands will need to be run from
outside of the environment shell by prefacing the command with
[`poetry run`](https://python-poetry.org/docs/cli/#run).
## Contributing
All project contributors and participants are expected to adhere to the
[Contributor Covenant Code of Conduct, Version 2](CODE_OF_CONDUCT.md).
[Contributor Covenant Code of Conduct, v2](CODE_OF_CONDUCT.md)
([external link](https://www.contributor-covenant.org/version/2/0/code_of_conduct/)).
The `devel` branch has the latest (potentially unstable) changes. The
[tagged versions](https://github.com/enpaul/tox-poetry-installer/releases) correspond to the
@@ -423,7 +522,7 @@ releases on PyPI.
* To report a bug, request a feature, or ask for assistance, please
[open an issue on the Github repository](https://github.com/enpaul/tox-poetry-installer/issues/new).
* To report a security concern or code of conduct violation, please contact the project author
directly at **ethan dot paul at enp dot one**.
directly at **me [at] enp dot one**.
* To submit an update, please
[fork the repository](https://docs.github.com/en/enterprise/2.20/user/github/getting-started-with-github/fork-a-repo)
and
@@ -432,10 +531,10 @@ releases on PyPI.
## Roadmap
This project is under active development and is classified as alpha software, not yet ready
for usage in production environments.
This project is under active development and is classified as beta software, ready for
production environments on a provisional basis only.
* Beta classification will be assigned when the initial feature set is finalized
* Beta classification was assigned with [v0.6.0](https://github.com/enpaul/tox-poetry-installer/releases/tag/0.6.0)
* Stable classification will be assigned when the test suite covers an acceptable number of
use cases
@@ -447,24 +546,25 @@ for usage in production environments.
Tox configuration option ([#4](https://github.com/enpaul/tox-poetry-installer/issues/4))
- [X] Add per-environment Tox configuration option to fall back to default installation
backend.
- [ ] Add warnings when an unsupported Tox configuration option is detected while using the
Poetry backend. ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5))
- [ ] ~Add warnings when an unsupported Tox configuration option is detected while using the
Poetry backend. ([#5](https://github.com/enpaul/tox-poetry-installer/issues/5))~
- [X] Add trivial tests to ensure the project metadata is consistent between the pyproject.toml
and the module constants.
- [X] Update to use [poetry-core](https://github.com/python-poetry/poetry-core) and
improve robustness of the Tox and Poetry module imports
to avoid potentially breaking API changes in upstream packages. ([#2](https://github.com/enpaul/tox-poetry-installer/issues/2))
- [ ] Find and implement a way to mitigate the [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in Poetry.
([#6](https://github.com/enpaul/tox-poetry-installer/issues/6))
- [ ] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3))
- [ ] Add configuration option for installing all dev-dependencies to a testenv ([#14](https://github.com/enpaul/tox-poetry-installer/issues/14))
- [ ] ~Find and implement a way to mitigate the [UNSAFE_DEPENDENCIES issue](https://github.com/python-poetry/poetry/issues/1584) in Poetry.
([#6](https://github.com/enpaul/tox-poetry-installer/issues/6))~
- [X] Fix logging to make proper use of Tox's logging reporter infrastructure ([#3](https://github.com/enpaul/tox-poetry-installer/issues/3))
- [X] Add configuration option for installing all dev-dependencies to a testenv ([#14](https://github.com/enpaul/tox-poetry-installer/issues/14))
### Path to Stable
Everything in Beta plus...
- [ ] Add tests for each feature version of Tox between 2.3 and 3.20
- [ ] Add tests for Python-3.6, 3.7, and 3.8
- [ ] Add comprehensive unit tests
- [ ] Add tests for each feature version of Tox between 3.0 and 3.20
- [ ] Add tests for Python-3.6, 3.7, 3.8, and 3.9
- [X] Add Github Actions based CI
- [ ] Add CI for CPython, PyPy, and Conda
- [ ] Add CI for Linux and Windows

588
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@@ -1,18 +1,21 @@
[tool.poetry]
name = "tox-poetry-installer"
version = "0.4.0"
version = "0.6.2"
license = "MIT"
authors = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
description = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
repository = "https://github.com/enpaul/tox-poetry-installer/"
packages = [
{include = "tox_poetry_installer.py"},
{include = "tox_poetry_installer"},
{include = "tests/*.py", format = "sdist"}
]
include = [
"tox_poetry_installer/py.typed"
]
keywords = ["tox", "poetry", "plugin"]
readme = "README.md"
classifiers = [
"Development Status :: 3 - Alpha",
"Development Status :: 4 - Beta",
"Environment :: Plugins",
"Framework :: tox",
"Intended Audience :: Developers",
@@ -22,32 +25,38 @@ classifiers = [
"Programming Language :: Python :: 3.6",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: Implementation :: CPython",
]
[tool.poetry.plugins.tox]
poetry_installer = "tox_poetry_installer"
[tool.poetry.extras]
poetry = ["poetry"]
[tool.poetry.dependencies]
python = "^3.6"
poetry = "^1.0.0"
python = "^3.6.1"
poetry = {version = "^1.0.0", optional = true}
poetry-core = "^1.0.0"
tox = "^2.3.0 || ^3.0.0"
tox = "^3.0.0"
[tool.poetry.dev-dependencies]
bandit = "^1.6.2"
black = { version = "^20.8b1", allow-prereleases = true }
blacken-docs = "^1.8.0"
ipython = { version = "^7.18.1", python = "^3.7" }
mypy = "^0.782"
pre-commit = {version = "^2.7.1", python = "^3.6.1"}
pre-commit = "^2.7.1"
pre-commit-hooks = "^3.3.0"
pylint = "^2.4.4"
pytest = "^6.0.2"
pytest-cov = "^2.10.1"
reorder-python-imports = {version = "^2.3.5", python = "^3.6.1"}
reorder-python-imports = "^2.3.5"
safety = "^1.9.0"
toml = "^0.10.1"
tox = "^3.20.0"
[build-system]
requires = ["poetry>=1.0.0"]
build-backend = "poetry.masonry.api"
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

0
tests/__init__.py Normal file
View File

View File

@@ -7,7 +7,7 @@ from pathlib import Path
import toml
import tox_poetry_installer
from tox_poetry_installer import __about__
def test_metadata():
@@ -16,16 +16,14 @@ def test_metadata():
with (Path(__file__).resolve().parent / ".." / "pyproject.toml").open() as infile:
pyproject = toml.load(infile, _dict=dict)
assert pyproject["tool"]["poetry"]["name"] == tox_poetry_installer.__title__
assert pyproject["tool"]["poetry"]["version"] == tox_poetry_installer.__version__
assert pyproject["tool"]["poetry"]["license"] == tox_poetry_installer.__license__
assert (
pyproject["tool"]["poetry"]["description"] == tox_poetry_installer.__summary__
)
assert pyproject["tool"]["poetry"]["repository"] == tox_poetry_installer.__url__
assert pyproject["tool"]["poetry"]["name"] == __about__.__title__
assert pyproject["tool"]["poetry"]["version"] == __about__.__version__
assert pyproject["tool"]["poetry"]["license"] == __about__.__license__
assert pyproject["tool"]["poetry"]["description"] == __about__.__summary__
assert pyproject["tool"]["poetry"]["repository"] == __about__.__url__
assert (
all(
item in tox_poetry_installer.__authors__
item in __about__.__authors__
for item in pyproject["tool"]["poetry"]["authors"]
)
is True
@@ -33,7 +31,7 @@ def test_metadata():
assert (
all(
item in pyproject["tool"]["poetry"]["authors"]
for item in tox_poetry_installer.__authors__
for item in __about__.__authors__
)
is True
)

60
tox.ini
View File

@@ -1,66 +1,64 @@
[tox]
envlist = py36, py37, py38, static, static-tests, security
envlist = py36, py37, py38, py39, static, static-tests, security
isolated_build = true
skip_missing_interpreters = true
[testenv]
description = Run the tests
require_locked_deps = true
deps =
extras =
poetry
locked_deps =
pytest
pytest-cov
toml
commands =
pytest --cov tox_poetry_installer --cov-config {toxinidir}/.coveragerc tests/ --cov-report term-missing
pytest --cov {envsitepackagesdir}/tox_poetry_installer --cov-config {toxinidir}/.coveragerc --cov-report term-missing tests/
[testenv:static]
description = Static formatting and quality enforcement
require_locked_deps = true
basepython = python3.8
platform = linux
ignore_errors = true
deps =
require_locked_deps = true
locked_deps =
pre-commit
pre-commit-hooks
black
blacken-docs
reorder-python-imports
pylint
mypy
black
reorder-python-imports
pre-commit
commands =
black {toxinidir}/tox_poetry_installer.py
reorder-python-imports {toxinidir}/tox_poetry_installer.py
pre-commit run --all-files
pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tox_poetry_installer.py
mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tox_poetry_installer.py
pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tox_poetry_installer/
mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tox_poetry_installer/
[testenv:static-tests]
description = Static formatting and quality enforcement for the tests
require_locked_deps = true
basepython = python3.8
platform = linux
ingore_errors = true
deps =
require_locked_deps = true
locked_deps =
pylint
mypy
black
reorder-python-imports
allowlist_externals =
bash
commands =
black {toxinidir}/tests/
bash -c "reorder-python-imports {toxinidir}/tests/*.py --unclassifiable-application-module tox_poetry_installer"
bash -c "pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tests/*.py"
bash -c "mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tests/*.py"
pylint --rcfile {toxinidir}/.pylintrc {toxinidir}/tests/
mypy --ignore-missing-imports --no-strict-optional {toxinidir}/tests/
[testenv:security]
description = Security checks
require_locked_deps = true
basepython = python3.8
ignore_errors = true
skip_install = true
deps =
platform = linux
ingore_errors = true
require_locked_deps = true
locked_deps =
bandit
safety
poetry
allowlist_externals =
bash
commands =
bandit --quiet {toxinidir}/tox_poetry_installer.py
bash -c "bandit --quiet --skip B101 {toxinidir}/tests/*.py"
bash -c "poetry export --format requirements.txt --without-hashes --dev | safety check --stdin --bare"
bandit --recursive --quiet {toxinidir}/tox_poetry_installer/
bandit --recursive --quiet --skip B101 {toxinidir}/tests/
poetry export --format requirements.txt --output {envtmpdir}/requirements.txt --without-hashes --dev
safety check --bare --file {envtmpdir}/requirements.txt

View File

@@ -1,401 +0,0 @@
"""Tox plugin for installing environments using Poetry
This plugin makes use of the ``tox_testenv_install_deps`` Tox plugin hook to augment the default
installation functionality to install dependencies from the Poetry lockfile for the project. It
does this by using ``poetry`` to read in the lockfile, identify necessary dependencies, and then
use Poetry's ``PipInstaller`` class to install those packages into the Tox environment.
Quick definition of terminology:
* "project package" - the package that Tox is testing, usually the one the current project is
is developing; definitionally, this is the package that is built by Tox in the ``.package`` env.
* "project package dependency" or "project dependency" - a dependency required by the project
package for installation; i.e. a package that would be installed when running
``pip install <project package>``.
* "environment dependency" - a dependency specified for a given testenv in the Tox configuration.
* "locked dependency" - a package that is present in the Poetry lockfile and will be installed
according to the metadata in the lockfile.
* "unlocked dependency" - a package that is either not present in the Poetry lockfile or is not
specified to be installed according to the metadata in the lockfile.
* "transiety dependency" - a package not explicitly specified for installation, but required by a
package that is explicitly specified.
"""
from pathlib import Path
from typing import Dict
from typing import List
from typing import NamedTuple
from typing import Sequence
from typing import Set
from typing import Tuple
from poetry.core.packages import Package as PoetryPackage
from poetry.factory import Factory as PoetryFactory
from poetry.installation.pip_installer import PipInstaller as PoetryPipInstaller
from poetry.io.null_io import NullIO as PoetryNullIO
from poetry.poetry import Poetry
from poetry.puzzle.provider import Provider as PoetryProvider
from poetry.utils.env import VirtualEnv as PoetryVirtualEnv
from tox import hookimpl
from tox import reporter
from tox.action import Action as ToxAction
from tox.config import DepConfig as ToxDepConfig
from tox.config import Parser as ToxParser
from tox.venv import VirtualEnv as ToxVirtualEnv
__title__ = "tox-poetry-installer"
__summary__ = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
__version__ = "0.4.0"
__url__ = "https://github.com/enpaul/tox-poetry-installer/"
__license__ = "MIT"
__authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]
# Valid PEP508 version delimiters. These are used to test whether a given string (specifically a
# dependency name) is just a package name or also includes a version identifier.
_PEP508_VERSION_DELIMITERS: Tuple[str, ...] = ("~=", "==", "!=", ">", "<")
# Prefix all reporter messages should include to indicate that they came from this module in the
# console output.
_REPORTER_PREFIX = f"[{__title__}]:"
# Suffix that indicates an env dependency should be treated as a locked dependency and thus be
# installed from the lockfile. Will be automatically stripped off of a dependency name during
# sorting so that the resulting string is just the valid package name. This becomes optional when
# the "require_locked_deps" option is true for an environment; in that case a bare dependency like
# 'foo' is treated the same as an explicitly locked dependency like 'foo@poetry'
_MAGIC_SUFFIX_MARKER = "@poetry"
# Map of package names to the package object
PackageMap = Dict[str, PoetryPackage]
class _SortedEnvDeps(NamedTuple):
unlocked_deps: List[ToxDepConfig]
locked_deps: List[ToxDepConfig]
class ToxPoetryInstallerException(Exception):
"""Error while installing locked dependencies to the test environment"""
class LockedDepVersionConflictError(ToxPoetryInstallerException):
"""Locked dependencies cannot specify an alternate version for installation"""
class LockedDepNotFoundError(ToxPoetryInstallerException):
"""Locked dependency was not found in the lockfile"""
class ExtraNotFoundError(ToxPoetryInstallerException):
"""Project package extra not defined in project's pyproject.toml"""
def _sort_env_deps(venv: ToxVirtualEnv) -> _SortedEnvDeps:
"""Sorts the environment dependencies by lock status
Lock status determines whether a given environment dependency will be installed from the
lockfile using the Poetry backend, or whether this plugin will skip it and allow it to be
installed using the default pip-based backend (an unlocked dependency).
.. note:: A locked dependency must follow a required format. To avoid reinventing the wheel
(no pun intended) this module does not have any infrastructure for parsing PEP-508
version specifiers, and so requires locked dependencies to be specified with no
version (the installed version being taken from the lockfile). If a dependency is
specified as locked and its name is also a PEP-508 string then an error will be
raised.
"""
reporter.verbosity1(
f"{_REPORTER_PREFIX} sorting {len(venv.envconfig.deps)} env dependencies by lock requirement"
)
unlocked_deps = []
locked_deps = []
for dep in venv.envconfig.deps:
if venv.envconfig.require_locked_deps:
reporter.verbosity1(
f"{_REPORTER_PREFIX} lock required for env, treating '{dep.name}' as locked env dependency"
)
dep.name = dep.name.replace(_MAGIC_SUFFIX_MARKER, "")
locked_deps.append(dep)
else:
if dep.name.endswith(_MAGIC_SUFFIX_MARKER):
reporter.verbosity1(
f"{_REPORTER_PREFIX} specification includes marker '{_MAGIC_SUFFIX_MARKER}', treating '{dep.name}' as locked env dependency"
)
dep.name = dep.name.replace(_MAGIC_SUFFIX_MARKER, "")
locked_deps.append(dep)
else:
reporter.verbosity1(
f"{_REPORTER_PREFIX} specification does not include marker '{_MAGIC_SUFFIX_MARKER}', treating '{dep.name}' as unlocked env dependency"
)
unlocked_deps.append(dep)
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(locked_deps)} locked env dependencies: {[item.name for item in locked_deps]}"
)
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(unlocked_deps)} unlocked env dependencies: {[item.name for item in unlocked_deps]}"
)
return _SortedEnvDeps(locked_deps=locked_deps, unlocked_deps=unlocked_deps)
def _install_to_venv(
poetry: Poetry, venv: ToxVirtualEnv, packages: Sequence[PoetryPackage]
):
"""Install a bunch of packages to a virtualenv
:param poetry: Poetry object the packages were sourced from
:param venv: Tox virtual environment to install the packages to
:param packages: List of packages to install to the virtual environment
"""
reporter.verbosity1(
f"{_REPORTER_PREFIX} Installing {len(packages)} packages to environment at {venv.envconfig.envdir}"
)
installer = PoetryPipInstaller(
env=PoetryVirtualEnv(path=Path(venv.envconfig.envdir)),
io=PoetryNullIO(),
pool=poetry.pool,
)
for dependency in packages:
reporter.verbosity1(f"{_REPORTER_PREFIX} installing {dependency}")
installer.install(dependency)
def _find_transients(packages: PackageMap, dependency_name: str) -> Set[PoetryPackage]:
"""Using a poetry object identify all dependencies of a specific dependency
:param poetry: Populated poetry object which can be used to build a populated locked
repository object.
:param dependency_name: Bare name (without version) of the dependency to fetch the transient
dependencies of.
:returns: List of packages that need to be installed for the requested dependency.
.. note:: The package corresponding to the dependency named by ``dependency_name`` is included
in the list of returned packages.
"""
try:
top_level = packages[dependency_name]
def find_deps_of_deps(name: str) -> List[PoetryPackage]:
if name in PoetryProvider.UNSAFE_PACKAGES:
reporter.warning(
f"{_REPORTER_PREFIX} installing package '{name}' using Poetry is not supported; skipping installation of package '{name}'"
)
return []
transients = [packages[name]]
for dep in packages[name].requires:
transients += find_deps_of_deps(dep.name)
return transients
return set(find_deps_of_deps(top_level.name))
except KeyError:
if any(
delimiter in dependency_name for delimiter in _PEP508_VERSION_DELIMITERS
):
raise LockedDepVersionConflictError(
f"Locked dependency '{dependency_name}' cannot include version specifier"
) from None
raise LockedDepNotFoundError(
f"No version of locked dependency '{dependency_name}' found in the project lockfile"
) from None
def _install_env_dependencies(
venv: ToxVirtualEnv, poetry: Poetry, packages: PackageMap
):
"""Install the packages for a specified testenv
Processes the tox environment config, identifies any locked environment dependencies, pulls
them from the lockfile, and installs them to the virtual environment.
:param venv: Tox virtual environment to install the packages to
:param poetry: Poetry object the packages were sourced from
:param packages: Mapping of package names to the corresponding package object
"""
env_deps = _sort_env_deps(venv)
dependencies: List[PoetryPackage] = []
for dep in env_deps.locked_deps:
try:
dependencies += _find_transients(packages, dep.name.lower())
except ToxPoetryInstallerException as err:
venv.status = "lockfile installation failed"
reporter.error(f"{_REPORTER_PREFIX} {err}")
raise err
if venv.envconfig.install_dev_deps:
reporter.verbosity1(
f"{_REPORTER_PREFIX} env specifies 'install_env_deps = true', including Poetry dev dependencies"
)
dev_dependencies = [
dep
for dep in poetry.locker.locked_repository(True).packages
if dep not in poetry.locker.locked_repository(False).packages
]
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dev_dependencies)} Poetry dev dependencies"
)
dependencies = list(set(dev_dependencies + dependencies))
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dependencies)} total dependencies from {len(env_deps.locked_deps)} locked env dependencies"
)
reporter.verbosity1(
f"{_REPORTER_PREFIX} updating env config with {len(env_deps.unlocked_deps)} unlocked env dependencies for installation using the default backend"
)
venv.envconfig.deps = env_deps.unlocked_deps
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(dependencies)} env dependencies from lockfile"
)
_install_to_venv(poetry, venv, dependencies)
def _install_project_dependencies(
venv: ToxVirtualEnv, poetry: Poetry, packages: PackageMap
):
"""Install the dependencies of the project package
Install all primary dependencies of the project package.
:param venv: Tox virtual environment to install the packages to
:param poetry: Poetry object the packages were sourced from
:param packages: Mapping of package names to the corresponding package object
"""
reporter.verbosity1(
f"{_REPORTER_PREFIX} performing installation of project dependencies"
)
base_dependencies: List[PoetryPackage] = [
packages[item.name]
for item in poetry.package.requires
if not item.is_optional()
]
extra_dependencies: List[PoetryPackage] = []
for extra in venv.envconfig.extras:
try:
extra_dependencies += [
packages[item.name] for item in poetry.package.extras[extra]
]
except KeyError:
raise ExtraNotFoundError(
f"Environment '{venv.name}' specifies project extra '{extra}' which was not found in the lockfile"
) from None
dependencies: List[PoetryPackage] = []
for dep in base_dependencies + extra_dependencies:
try:
dependencies += _find_transients(packages, dep.name.lower())
except ToxPoetryInstallerException as err:
venv.status = "lockfile installation failed"
reporter.error(f"{_REPORTER_PREFIX} {err}")
raise err
reporter.verbosity1(
f"{_REPORTER_PREFIX} identified {len(dependencies)} total dependencies from {len(poetry.package.requires)} project dependencies"
)
reporter.verbosity0(
f"{_REPORTER_PREFIX} ({venv.name}) installing {len(dependencies)} project dependencies from lockfile"
)
_install_to_venv(poetry, venv, dependencies)
@hookimpl
def tox_addoption(parser: ToxParser):
"""Add required configuration options to the tox INI file
Adds the ``require_locked_deps`` configuration option to the venv to check whether all
dependencies should be treated as locked or not.
"""
parser.add_testenv_attribute(
name="install_dev_deps",
type="bool",
default=False,
help="Automatically install all Poetry development dependencies to the environment",
)
parser.add_testenv_attribute(
name="require_locked_deps",
type="bool",
default=False,
help="Require all dependencies in the environment be installed using the Poetry lockfile",
)
@hookimpl
def tox_testenv_install_deps(venv: ToxVirtualEnv, action: ToxAction):
"""Install the dependencies for the current environment
Loads the local Poetry environment and the corresponding lockfile then pulls the dependencies
specified by the Tox environment. Finally these dependencies are installed into the Tox
environment using the Poetry ``PipInstaller`` backend.
:param venv: Tox virtual environment object with configuration for the local Tox environment.
:param action: Tox action object
"""
if action.name == venv.envconfig.config.isolated_build_env:
# Skip running the plugin for the packaging environment. PEP-517 front ends can handle
# that better than we can, so let them do their thing. More to the point: if you're having
# problems in the packaging env that this plugin would solve, god help you.
reporter.verbosity1(
f"{_REPORTER_PREFIX} skipping isolated build env '{action.name}'"
)
return
try:
poetry = PoetryFactory().create_poetry(venv.envconfig.config.toxinidir)
except RuntimeError:
# Support running the plugin when the current tox project does not use Poetry for its
# environment/dependency management.
#
# ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case,
# is) raised in many different places for different purposes.
reporter.verbosity1(
f"{_REPORTER_PREFIX} project does not use Poetry for env management, skipping installation of locked dependencies"
)
return
reporter.verbosity1(
f"{_REPORTER_PREFIX} loaded project pyproject.toml from {poetry.file}"
)
package_map: PackageMap = {
package.name: package
for package in poetry.locker.locked_repository(True).packages
}
# Handle the installation of any locked env dependencies from the lockfile
_install_env_dependencies(venv, poetry, package_map)
# Handle the installation of the package dependencies from the lockfile if the package is
# being installed to this venv; otherwise skip installing the package dependencies
if venv.envconfig.skip_install:
reporter.verbosity1(
f"{_REPORTER_PREFIX} env specifies 'skip_install = true', skipping installation of project package"
)
return
if venv.envconfig.config.skipsdist:
reporter.verbosity1(
f"{_REPORTER_PREFIX} config specifies 'skipsdist = true', skipping installation of project package"
)
return
_install_project_dependencies(venv, poetry, package_map)

View File

@@ -0,0 +1,7 @@
# pylint: disable=missing-docstring
__title__ = "tox-poetry-installer"
__summary__ = "Tox plugin to install Tox environment dependencies using the Poetry backend and lockfile"
__version__ = "0.6.2"
__url__ = "https://github.com/enpaul/tox-poetry-installer/"
__license__ = "MIT"
__authors__ = ["Ethan Paul <24588726+enpaul@users.noreply.github.com>"]

View File

@@ -0,0 +1,3 @@
# pylint: disable=missing-docstring
from tox_poetry_installer.hooks import tox_addoption
from tox_poetry_installer.hooks import tox_testenv_install_deps

View File

@@ -0,0 +1,40 @@
"""You've heard of vendoirization, now get ready for internal namespace shadowing
Poetry is an optional dependency of this package explicitly to support the use case of having the
plugin and the `poetry` package installed to the same python environment; this is most common in
containers and/or CI. In this case there are two potential problems that can arise in this case:
* The installation of the plugin overwrites the installed version of Poetry resulting in
compatibility issues.
* Running `poetry install --no-dev`, when this plugin is in the dev-deps, results in poetry being
uninstalled from the environment.
To support these edge cases, and more broadly to support not messing with a system package manager,
the `poetry` package dependency is listed as optional dependency. This allows the plugin to be
installed to the same environment as Poetry and import that same Poetry installation here.
However, simply importing Poetry on the assumption that it is installed breaks another valid use
case: having this plugin installed alongside Tox when not using a Poetry-based project. To account
for this the imports in this module are isolated and the resultant import error that would result
is converted to an internal error that can be caught by callers. Rather than importing this module
at the module scope it is imported into function scope wherever Poetry components are needed. This
moves import errors from load time to runtime which allows the plugin to be skipped if Poetry isn't
installed and/or a more helpful error be raised within the Tox framework.
"""
# pylint: disable=unused-import
import sys
from tox_poetry_installer import exceptions
try:
from poetry.factory import Factory
from poetry.installation.pip_installer import PipInstaller
from poetry.io.null_io import NullIO
from poetry.poetry import Poetry
from poetry.puzzle.provider import Provider
from poetry.utils.env import VirtualEnv
except ImportError:
raise exceptions.PoetryNotInstalledError(
f"No version of Poetry could be imported under the current environment for '{sys.executable}'"
) from None

View File

@@ -0,0 +1,31 @@
"""Static constants for reference
Rule of thumb: if it's an arbitrary value that will never be changed at runtime, it should go
in this module.
All constants should be type hinted.
"""
import sys
from typing import Tuple
from poetry.core.semver.version import Version
from tox_poetry_installer import __about__
# Valid PEP508 version delimiters. These are used to test whether a given string (specifically a
# dependency name) is just a package name or also includes a version identifier.
PEP508_VERSION_DELIMITERS: Tuple[str, ...] = ("~=", "==", "!=", ">", "<")
# Prefix all reporter messages should include to indicate that they came from this module in the
# console output.
REPORTER_PREFIX: str = f"[{__about__.__title__}]:"
# Semver compatible version of the current python platform version. Used for checking
# whether a package is compatible with the current python system version
PLATFORM_VERSION: Version = Version(
major=sys.version_info.major,
minor=sys.version_info.minor,
patch=sys.version_info.micro,
)

View File

@@ -0,0 +1,8 @@
"""Definitions for typehints/containers used by the plugin"""
from typing import Dict
from poetry.core.packages import Package as PoetryPackage
# Map of package names to the package object
PackageMap = Dict[str, PoetryPackage]

View File

@@ -0,0 +1,43 @@
"""Custom plugin exceptions
All exceptions should inherit from the common base exception :exc:`ToxPoetryInstallerException`.
::
ToxPoetryInstallerException
+-- SkipEnvironment
| +-- PoetryNotInstalledError
+-- LockedDepVersionConflictError
+-- LockedDepNotFoundError
+-- ExtraNotFoundError
+-- LockedDepsRequiredError
"""
class ToxPoetryInstallerException(Exception):
"""Error while installing locked dependencies to the test environment"""
class SkipEnvironment(ToxPoetryInstallerException):
"""Current environment does not meet preconditions and should be skipped by the plugin"""
class PoetryNotInstalledError(SkipEnvironment):
"""No version of Poetry could be imported from the current Python environment"""
class LockedDepVersionConflictError(ToxPoetryInstallerException):
"""Locked dependencies cannot specify an alternate version for installation"""
class LockedDepNotFoundError(ToxPoetryInstallerException):
"""Locked dependency was not found in the lockfile"""
class ExtraNotFoundError(ToxPoetryInstallerException):
"""Project package extra not defined in project's pyproject.toml"""
class LockedDepsRequiredError(ToxPoetryInstallerException):
"""Environment cannot specify unlocked dependencies when locked dependencies are required"""

View File

@@ -0,0 +1,150 @@
"""Main hook definition module
All implementations of tox hooks are defined here, as well as any single-use helper functions
specifically related to implementing the hooks (to keep the size/readability of the hook functions
themselves manageable).
"""
from typing import List
from typing import Optional
from poetry.core.packages import Package as PoetryPackage
from tox import hookimpl
from tox import reporter
from tox.action import Action as ToxAction
from tox.config import Parser as ToxParser
from tox.venv import VirtualEnv as ToxVirtualEnv
from tox_poetry_installer import __about__
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer import utilities
from tox_poetry_installer.datatypes import PackageMap
@hookimpl
def tox_addoption(parser: ToxParser):
"""Add required configuration options to the tox INI file
Adds the ``require_locked_deps`` configuration option to the venv to check whether all
dependencies should be treated as locked or not.
"""
parser.add_argument(
"--require-poetry",
action="store_true",
dest="require_poetry",
help="Trigger a failure if Poetry is not available to Tox",
)
parser.add_testenv_attribute(
name="install_dev_deps",
type="bool",
default=False,
help="Automatically install all Poetry development dependencies to the environment",
)
parser.add_testenv_attribute(
name="require_locked_deps",
type="bool",
default=False,
help="Require all dependencies in the environment be installed using the Poetry lockfile",
)
parser.add_testenv_attribute(
name="locked_deps",
type="line-list",
help="List of locked dependencies to install to the environment using the Poetry lockfile",
)
@hookimpl
def tox_testenv_install_deps(venv: ToxVirtualEnv, action: ToxAction) -> Optional[bool]:
"""Install the dependencies for the current environment
Loads the local Poetry environment and the corresponding lockfile then pulls the dependencies
specified by the Tox environment. Finally these dependencies are installed into the Tox
environment using the Poetry ``PipInstaller`` backend.
:param venv: Tox virtual environment object with configuration for the local Tox environment.
:param action: Tox action object
"""
try:
poetry = utilities.check_preconditions(venv, action)
except exceptions.SkipEnvironment as err:
if (
isinstance(err, exceptions.PoetryNotInstalledError)
and venv.envconfig.config.option.require_poetry
):
venv.status = err.__class__.__name__
reporter.error(str(err))
return False
reporter.verbosity1(str(err))
return None
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Loaded project pyproject.toml from {poetry.file}"
)
try:
if venv.envconfig.require_locked_deps and venv.envconfig.deps:
raise exceptions.LockedDepsRequiredError(
f"Unlocked dependencies '{venv.envconfig.deps}' specified for environment '{venv.name}' which requires locked dependencies"
)
package_map: PackageMap = {
package.name: package
for package in poetry.locker.locked_repository(True).packages
}
if venv.envconfig.install_dev_deps:
dev_deps: List[PoetryPackage] = [
dep
for dep in package_map.values()
if dep not in poetry.locker.locked_repository(False).packages
]
else:
dev_deps = []
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Identified {len(dev_deps)} development dependencies to install to env"
)
env_deps: List[PoetryPackage] = []
for dep in venv.envconfig.locked_deps:
env_deps += utilities.find_transients(
package_map, dep.lower(), allow_missing=[poetry.package.name]
)
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Identified {len(env_deps)} environment dependencies to install to env"
)
if not venv.envconfig.skip_install and not venv.envconfig.config.skipsdist:
project_deps: List[PoetryPackage] = utilities.find_project_dependencies(
venv, poetry, package_map
)
else:
project_deps = []
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Skipping installation of project dependencies, env does not install project package"
)
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Identified {len(project_deps)} project dependencies to install to env"
)
except exceptions.ToxPoetryInstallerException as err:
venv.status = err.__class__.__name__
reporter.error(f"{constants.REPORTER_PREFIX} {err}")
return False
except Exception as err:
venv.status = "InternalError"
reporter.error(f"{constants.REPORTER_PREFIX} Internal plugin error: {err}")
raise err
dependencies = list(set(dev_deps + env_deps + project_deps))
action.setactivity(
__about__.__title__,
f"Installing {len(dependencies)} dependencies from Poetry lock file",
)
utilities.install_to_venv(poetry, venv, dependencies)
return venv.envconfig.require_locked_deps or None

View File

View File

@@ -0,0 +1,219 @@
"""Helper utility functions, usually bridging Tox and Poetry functionality"""
# Silence this one globally to support the internal function imports for the proxied poetry module.
# See the docstring in 'tox_poetry_installer._poetry' for more context.
# pylint: disable=import-outside-toplevel
import sys
import typing
from pathlib import Path
from typing import List
from typing import Sequence
from typing import Set
from poetry.core.packages import Package as PoetryPackage
from tox import reporter
from tox.action import Action as ToxAction
from tox.venv import VirtualEnv as ToxVirtualEnv
from tox_poetry_installer import constants
from tox_poetry_installer import exceptions
from tox_poetry_installer.datatypes import PackageMap
if typing.TYPE_CHECKING:
from tox_poetry_installer import _poetry
def install_to_venv(
poetry: "_poetry.Poetry", venv: ToxVirtualEnv, packages: Sequence[PoetryPackage]
):
"""Install a bunch of packages to a virtualenv
:param poetry: Poetry object the packages were sourced from
:param venv: Tox virtual environment to install the packages to
:param packages: List of packages to install to the virtual environment
"""
from tox_poetry_installer import _poetry
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Installing {len(packages)} packages to environment at {venv.envconfig.envdir}"
)
installer = _poetry.PipInstaller(
env=_poetry.VirtualEnv(path=Path(venv.envconfig.envdir)),
io=_poetry.NullIO(),
pool=poetry.pool,
)
for dependency in packages:
reporter.verbosity1(f"{constants.REPORTER_PREFIX} Installing {dependency}")
installer.install(dependency)
def find_transients(
packages: PackageMap, dependency_name: str, allow_missing: Sequence[str] = ()
) -> Set[PoetryPackage]:
"""Using a poetry object identify all dependencies of a specific dependency
:param packages: All packages from the lockfile to use for identifying dependency relationships.
:param dependency_name: Bare name (without version) of the dependency to fetch the transient
dependencies of.
:param allow_missing: Sequence of package names to allow to be missing from the lockfile. Any
packages that are not found in the lockfile but their name appears in this
list will be silently skipped from installation.
:returns: List of packages that need to be installed for the requested dependency.
.. note:: The package corresponding to the dependency named by ``dependency_name`` is included
in the list of returned packages.
"""
from tox_poetry_installer import _poetry
def find_deps_of_deps(name: str, searched: Set[str]) -> PackageMap:
searched.add(name)
if name in _poetry.Provider.UNSAFE_PACKAGES:
reporter.warning(
f"{constants.REPORTER_PREFIX} Installing package '{name}' using Poetry is not supported and will be skipped"
)
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Skip {name}: designated unsafe by Poetry"
)
return dict()
transients: PackageMap = {}
try:
package = packages[name]
except KeyError as err:
if name in allow_missing:
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Skip {name}: package is not in lockfile but designated as allowed to be missing"
)
return dict()
raise err
if not package.python_constraint.allows(constants.PLATFORM_VERSION):
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Skip {package}: incompatible Python requirement '{package.python_constraint}' for current version '{constants.PLATFORM_VERSION}'"
)
elif package.platform is not None and package.platform != sys.platform:
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Skip {package}: incompatible platform requirement '{package.platform}' for current platform '{sys.platform}'"
)
else:
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Including {package} for installation"
)
transients[name] = package
for index, dep in enumerate(package.requires):
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Processing dependency {index + 1}/{len(package.requires)} for {package}: {dep.name}"
)
if dep.name not in searched:
transients.update(find_deps_of_deps(dep.name, searched))
else:
reporter.verbosity2(
f"{constants.REPORTER_PREFIX} Package with name '{dep.name}' has already been processed, skipping"
)
return transients
searched: Set[str] = set()
try:
transients: PackageMap = find_deps_of_deps(
packages[dependency_name].name, searched
)
except KeyError:
if dependency_name in _poetry.Provider.UNSAFE_PACKAGES:
reporter.warning(
f"{constants.REPORTER_PREFIX} Installing package '{dependency_name}' using Poetry is not supported and will be skipped"
)
return set()
if any(
delimiter in dependency_name
for delimiter in constants.PEP508_VERSION_DELIMITERS
):
raise exceptions.LockedDepVersionConflictError(
f"Locked dependency '{dependency_name}' cannot include version specifier"
) from None
raise exceptions.LockedDepNotFoundError(
f"No version of locked dependency '{dependency_name}' found in the project lockfile"
) from None
return set(transients.values())
def check_preconditions(venv: ToxVirtualEnv, action: ToxAction) -> "_poetry.Poetry":
"""Check that the local project environment meets expectations"""
# Skip running the plugin for the provisioning environment. The provisioned environment,
# for alternative Tox versions and/or the ``requires`` meta dependencies is specially
# handled by Tox and is out of scope for this plugin. Since one of the ways to install this
# plugin in the first place is via the Tox provisioning environment, it quickly becomes a
# chicken-and-egg problem.
if action.name == venv.envconfig.config.provision_tox_env:
raise exceptions.SkipEnvironment(
f"Skipping Tox provisioning env '{action.name}'"
)
# Skip running the plugin for the packaging environment. PEP-517 front ends can handle
# that better than we can, so let them do their thing. More to the point: if you're having
# problems in the packaging env that this plugin would solve, god help you.
if action.name == venv.envconfig.config.isolated_build_env:
raise exceptions.SkipEnvironment(
f"Skipping isolated packaging build env '{action.name}'"
)
from tox_poetry_installer import _poetry
try:
return _poetry.Factory().create_poetry(venv.envconfig.config.toxinidir)
# Support running the plugin when the current tox project does not use Poetry for its
# environment/dependency management.
#
# ``RuntimeError`` is dangerous to blindly catch because it can be (and in Poetry's case,
# is) raised in many different places for different purposes.
except RuntimeError:
raise exceptions.SkipEnvironment(
"Project does not use Poetry for env management, skipping installation of locked dependencies"
) from None
def find_project_dependencies(
venv: ToxVirtualEnv, poetry: "_poetry.Poetry", packages: PackageMap
) -> List[PoetryPackage]:
"""Install the dependencies of the project package
Install all primary dependencies of the project package.
:param venv: Tox virtual environment to install the packages to
:param poetry: Poetry object the packages were sourced from
:param packages: Mapping of package names to the corresponding package object
"""
base_dependencies: List[PoetryPackage] = [
packages[item.name]
for item in poetry.package.requires
if not item.is_optional()
]
extra_dependencies: List[PoetryPackage] = []
for extra in venv.envconfig.extras:
reporter.verbosity1(
f"{constants.REPORTER_PREFIX} Processing project extra '{extra}'"
)
try:
extra_dependencies += [
packages[item.name] for item in poetry.package.extras[extra]
]
except KeyError:
raise exceptions.ExtraNotFoundError(
f"Environment '{venv.name}' specifies project extra '{extra}' which was not found in the lockfile"
) from None
dependencies: List[PoetryPackage] = []
for dep in base_dependencies + extra_dependencies:
dependencies += find_transients(
packages, dep.name.lower(), allow_missing=[poetry.package.name]
)
return dependencies