I recently revisited a Python module that I developed called singletons. When I set it up, I tried to follow best practices for lots of things, including using tox with Travis CI to automatically run tests upon push. I used a cookiecutter template called cookiecutter-pylibrary, which set a lot of sensible defaults. And then I took a job where I didn’t do much Python at all.

Well, I’m finally getting back into Python (yay!), and decided to revisit this library. It seems the community is converging on poetry for packaging and depdency management rolled into one elegant tool, and having tried it out a bit, I have to say it’s quite nice. I decided to migrate my project to use this instead of setup.py, and while I was at it I decided to get rid of a lot of extraneous files and make the development and deployment process more streamlined.

I did, however, run into some hiccups getting everything set up to work with the way I do development, so I’m documenting my process here (if only to help my future self).

Python version management with asdf

First of all, there’s Python version management. Once upon a time I used pyenv, but I hated having to install a whole bunch of disparate tools for each programming language I used. Now I use asdf, which lets me use a single command to manage basically every programming language. If you haven’t set up asdf already, here’s a quickstart:

# install asdf and common dependencies
$ brew install asdf \
  coreutils automake autoconf openssl \
  libyaml readline libxslt libtool unixodbc \
  unzip curl

# set up asdf with python
$ asdf plugin add python
$ asdf install python 3.8.0

# install additional versions as necessary
$ asdf install python 3.7.5
$ asdf install python 3.6.9

What asdf does is add itself to your path, so that when you run python (or python3 or python3.8), it will use the version installed by asdf. Awesome! But there’s one caveat – it only uses those versions if you tell it to.

Using asdf versions of Python

asdf does give you the option of specifying a global version of a particular interpreter/compiler to use. However, given that OSX includes a system version of python (and some tools may expect that to function normally), I didn’t want to replace it system-wide. So my solution is to do the following.

In each folder where I’m doing python development, I run an asdf local python command. This creates a file called .tool-versions (which you should probably add to a global gitignore file). asdf refers to this file, and looks up the file hierarchy to find one, to determine which version of python to use.

For example, if I want to use Python 3.8.0, I would run the following:

$ asdf local python 3.8.0

The special trick for tox

tox requires multiple versions of Python to be installed. Using asdf, you have multiple versions installed, but they aren’t normally exposed to the current shell. Enter – multiple versions!

You can use the following command to expose multiple versions of Python in the current directory:

$ asdf local python 3.8.0 3.7.5 3.6.9

This will use 3.8.0 by default (if you just run python), but it will also put python3.7 and python3.6 symlinks in your path so you can run those too (which is exactly what tox is looking for).

Installing tox and poetry

Lastly, just to be safe, you should ensure that each of those asdf versions of python have the bare minimum of dependencies. Namely, tox and poetry.

$ pip3.8 install tox poetry
$ pip3.7 install tox poetry
$ pip3.6 install tox poetry

One other thing – asdf might miss the fact that you’ve installed tox and poetry, so you can run the following to force it to pick up on that:

$ asdf reshim python

Now you should be able to run tox normally!

Travis CI

Last of all, getting Travis to work with all this. It’s actually much simpler than it used to be. With an appropriate tox setup, you can keep your Travis configuration very simple:

.travis.yml

language: python
python:
  - "3.6"
  - "3.7"
  - "3.8"
before_install:
  - pip install poetry
install:
  - pip install tox-travis
script:
  - tox

tox.ini

[tox]
isolated_build = true
envlist = py36,py37,py38
skip_missing_interpreters = true

[testenv]
whitelist_externals = poetry
commands =
  poetry install -v --extras "eventlet gevent"
  poetry run pytest {posargs} tests/

Also, if you have other build stages, like docs, linting, etc., things will become a little more complicated, but hopefully still manageable!

Note that the poetry install command includes some extras. Chances are your library doesn’t have these, but I have some tests that use them. You can probably just do poetry install -v for most situations.

Bonus:

You can update pip for each environment to hide some annoying warnings:

$ pip3.8 install --upgrade pip
$ pip3.7 install --upgrade pip
$ pip3.6 install --upgrade pip

Also, by default, poetry creates virtualenvs in your user directory (~). I prefer to keep my virtualenvs close to the project files, and poetry has an option to support this.

$ poetry config settings.virtualenvs.in-project true
# or if you are running poetry 1.0
$ poetry config virtualenvs.in-project true

Several years back I got into the hobby of arcade game restoration. I even created a blog documenting some of the work I did at 1up-arcade.com. I haven’t had the opportunity to do it for quite some time, though, and I let the website expire (it now appears to redirect to an Australian arcade – which is cool!).

I didn’t want to let the posts disappear off the internet, though, so I set up a subdomain, set up a fresh WordPress installation, and uploaded them.

Check it out here:

1up Arcade – Arcade game restoration

I use a Raspberry Pi as a controller for our X-Carve CNC machine, and it recently stopped booting (my guess is due to SD card corruption from being improperly shut down). I wanted to test out some configuration changes to allow for the SD card to be mounted completely read only, but I didn’t want to have to keep on rebooting the Pi. So I decided to look into emulation (running the Pi in a VM).

The Pi uses an ARM processor, which means that traditional virtual machine approaches like VirtualBox or VMWare wouldn’t be suited. I found a few gists online of people who have done similar things on their Macs that I thought could be useful as a starting point.

I used this as my basis: https://gist.github.com/thomasweng15/af0929114efce3524d55d10f170ff30d

First, one-time setup procedure (I put this in a file setup_qemu.sh)

#!/usr/bin/env bash

# # install qemu
# brew install qemu

# # download the latest stretch kernel and the necessary dtb file
# $ curl -OL https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/kernel-qemu-4.14.79-stretch
# $ curl -OL https://github.com/dhruvvyas90/qemu-rpi-kernel/raw/master/versatile-pb.dtb

# # download and unzip the latest raspbian lite
# $ curl -o raspbian_lite_latest.zip -L https://downloads.raspberrypi.org/raspbian_lite_latest
# $ unzip raspbian_lite_latest.zip

export QEMU=$(which qemu-system-arm)
export RPI_KERNEL=kernel-qemu-4.14.79-stretch
export RPI_FS=2019-04-08-raspbian-stretch-lite.img
export RPI_FS_ZIP=2019-04-08_raspbian_lite_stretch.zip
export PTB_FILE=versatile-pb.dtb

# First time setup
# wipe old img file (uncomment this if you are share that's what you want to do!)
if [ -f $RPI_FS ]; then
    rm $RPI_FS
fi
if [ ! -f $RPI_FS ]; then
    unzip $RPI_FS_ZIP
fi

$QEMU -kernel ${RPI_KERNEL} \
    -cpu arm1176 \
    -m 256 \
    -M versatilepb \
    -dtb ${PTB_FILE} \
    -no-reboot \
    -serial mon:stdio \
    -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw init=/bin/bash" \
    -drive "file=${RPI_FS},index=0,media=disk,format=raw"

# Paste the line below (after the comment). press enter, then ctrl-d when it is done booting
# sed -i -e 's/^/#/' /etc/ld.so.preload; sed -i -e 's/^/#/' /etc/ld.so.conf; sed -i -e 's/^/#/' /etc/fstab

Next, the script to run QEMU (I put this in run_qemu.sh):

#!/usr/bin/env bash

export QEMU=$(which qemu-system-arm)
export RPI_KERNEL=kernel-qemu-4.14.79-stretch
export RPI_FS=2019-04-08-raspbian-stretch-lite.img
export PTB_FILE=versatile-pb.dtb

$QEMU -kernel ${RPI_KERNEL} \
    -cpu arm1176 \
    -m 256 \
    -M versatilepb \
    -dtb ${PTB_FILE} \
    -serial mon:stdio \
    -append "root=/dev/sda2 panic=1 rootfstype=ext4 rw" \
    -drive "file=${RPI_FS},index=0,media=disk,format=raw" \
    -net user,hostfwd=tcp::5022-:22 \
    -net nic

To install the official Docker fish completions:

curl https://raw.githubusercontent.com/docker/cli/master/contrib/completion/fish/docker.fish --create-dirs -sLo ~/.config/fish/completions/docker.fish

To install Docker Compose completions (the official ones appear to be minimal):

fisher add brgmnn/fish-docker-compose

I just spent an inordinate amount of time tracking down an issue with pylint-django that I had a hard time finding any clues on the internet about, so I’m documenting it here.

I use pylint and pylint-django to perform automated checks on my Django projects and ensure a certain code quality is maintained. This combination has proven very useful. Recently, though, I ran into an issue where one project would fail to validate with a strange error:

$ pylint --rcfile pylintrc myproject
Using config file /app/pylintrc
Traceback (most recent call last):
  File "/usr/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File "/usr/lib/python3.6/site-packages/pylint/__init__.py", line 16, in run_pylint
    Run(sys.argv[1:])
  File "/usr/lib/python3.6/site-packages/pylint/lint.py", line 1312, in __init__
    linter.load_plugin_modules(plugins)
  File "/usr/lib/python3.6/site-packages/pylint/lint.py", line 495, in load_plugin_modules
    module.register(self)
  File "/usr/lib/python3.6/site-packages/pylint_django/plugin.py", line 18, in register
    name_checker = get_checker(linter, NameChecker)
  File "/usr/lib/python3.6/site-packages/pylint_plugin_utils/__init__.py", line 30, in get_checker
    raise NoSuchChecker(checker_class)
pylint_plugin_utils.NoSuchChecker: <class 'pylint.checkers.base.NameChecker'>

Investigation

I thought maybe this had to do with some incompatibilities between pylint, pylint-django, and possibly astroid, but strangely, the versions in this project were exactly the same as the other project. So I dug into the code where the exception was being reported.

get_checker is a part of pylint_plugin_utils, and it is used by pylint-django to augment the base PyLint checkers. It was trying to find pylint.checkers.base.NameChecker in the list of registered “checkers” for pylint. The file pylint/checkers/base.py" did exist in thesite-packagesfolder, but strangely, it was being registered assite_packages.pylint.checkers.base.NameChecker`.

Pylint has a register_plugins function that it uses to register all the default plugins. It does this by calling modutils.load_module_from_file on each of the files it finds starting from the checkers directory included with the library.

load_module_from_file figures out the proper import path using a function modpath_from_file, which in turn uses modpath_from_file_with_callback to check that each path it traverses has a __init__.py file. It looks at each path in sys.path in order to determine if the file attempting to be loaded has a valid import path.

What was throwing things off was the presence of __init__.py in the site-packages directory. /usr/lib/python3.6 was in sys.path before /usr/lib/python3.6/site-packages, and because there was a __init__.py file in site-packages, the import mechanism thought that site-packages was itself a module in the /usr/lib/python3.6 directory. It was therefore assigning this as the module name. Now where was that coming from?

Fix

It turns out it was a rogue package that stuck that empty file there (singletons). I removed that file (and thankfully had control over the project where that file originated and removed it from the source), and that fixed the issue.