I recently revisited a Python module that I developed called singletons. When I set it up, I tried to follow best practices for lots of things, including using tox with Travis CI to automatically run tests upon push. I used a cookiecutter template called cookiecutter-pylibrary, which set a lot of sensible defaults. And then I took a job where I didn’t do much Python at all.
Well, I’m finally getting back into Python (yay!), and decided to revisit this library. It seems the community is converging on poetry for packaging and depdency management rolled into one elegant tool, and having tried it out a bit, I have to say it’s quite nice. I decided to migrate my project to use this instead of setup.py, and while I was at it I decided to get rid of a lot of extraneous files and make the development and deployment process more streamlined.
I did, however, run into some hiccups getting everything set up to work with the way I do development, so I’m documenting my process here (if only to help my future self).
Python version management with asdf
First of all, there’s Python version management. Once upon a time I used pyenv, but I hated having to install a whole bunch of disparate tools for each programming language I used. Now I use asdf, which lets me use a single command to manage basically every programming language. If you haven’t set up asdf already, here’s a quickstart:
# install asdf and common dependencies
$ brew install asdf \
coreutils automake autoconf openssl \
libyaml readline libxslt libtool unixodbc \
unzip curl
# set up asdf with python
$ asdf plugin add python
$ asdf install python 3.8.0
# install additional versions as necessary
$ asdf install python 3.7.5
$ asdf install python 3.6.9
What asdf does is add itself to your path, so that when you run python (or python3 or python3.8), it will use the version installed by asdf. Awesome! But there’s one caveat – it only uses those versions if you tell it to.
Using asdf versions of Python
asdf does give you the option of specifying a global version of a particular interpreter/compiler to use. However, given that OSX includes a system version of python (and some tools may expect that to function normally), I didn’t want to replace it system-wide. So my solution is to do the following.
In each folder where I’m doing python development, I run an asdf local python command. This creates a file called .tool-versions (which you should probably add to a global gitignore file). asdf refers to this file, and looks up the file hierarchy to find one, to determine which version of python to use.
For example, if I want to use Python 3.8.0, I would run the following:
$ asdf local python 3.8.0
The special trick for tox
tox requires multiple versions of Python to be installed. Using asdf, you have multiple versions installed, but they aren’t normally exposed to the current shell. Enter – multiple versions!
You can use the following command to expose multiple versions of Python in the current directory:
$ asdf local python 3.8.0 3.7.5 3.6.9
This will use 3.8.0 by default (if you just run python), but it will also put python3.7 and python3.6 symlinks in your path so you can run those too (which is exactly what tox is looking for).
Installing tox and poetry
Lastly, just to be safe, you should ensure that each of those asdf versions of python have the bare minimum of dependencies. Namely, tox and poetry.
$ pip3.8 install tox poetry
$ pip3.7 install tox poetry
$ pip3.6 install tox poetry
One other thing – asdf might miss the fact that you’ve installed tox and poetry, so you can run the following to force it to pick up on that:
$ asdf reshim python
Now you should be able to run tox normally!
Travis CI
Last of all, getting Travis to work with all this. It’s actually much simpler than it used to be. With an appropriate tox setup, you can keep your Travis configuration very simple:
.travis.yml
language: python
python:
- "3.6"
- "3.7"
- "3.8"
before_install:
- pip install poetry
install:
- pip install tox-travis
script:
- tox
tox.ini
[tox]
isolated_build = true
envlist = py36,py37,py38
skip_missing_interpreters = true
[testenv]
whitelist_externals = poetry
commands =
poetry install -v --extras "eventlet gevent"
poetry run pytest {posargs} tests/
Also, if you have other build stages, like docs, linting, etc., things will become a little more complicated, but hopefully still manageable!
Note that the poetry install command includes some extras. Chances are your library doesn’t have these, but I have some tests that use them. You can probably just do poetry install -v for most situations.
Bonus:
You can update pip for each environment to hide some annoying warnings:
$ pip3.8 install --upgrade pip
$ pip3.7 install --upgrade pip
$ pip3.6 install --upgrade pip
Also, by default, poetry creates virtualenvs in your user directory (~). I prefer to keep my virtualenvs close to the project files, and poetry has an option to support this.
$ poetry config settings.virtualenvs.in-project true
# or if you are running poetry 1.0
$ poetry config virtualenvs.in-project true