The tools

The original post listed a bunch of packaging tools, calling fourteen tools at least twelve too many. My idea with that was that most people would be happy with one tool that does everything, but the scientific-Python folks might have special requirements that would work best as a second tool.

Out of the tools named in last year’s post, all of them still seem to be maintained. Except for Flit (zero new commits in the past 30 days) and virtualenv (only automated and semi-automated version bumps), the tools have recent commits, pull requests, and issues.

All of those tools are still in use. Françoise Conil analysed all PyPI packages and checked their PEP 517 build backends: setuptools is the most popular (at 50k packages), Poetry is second at 41k, Hatchling is third at 8.1k. Other tools to cross 500 users include Flit (4.4k), PDM (1.3k), Maturin (1.3k, build backend for Rust-based packages).

There are some new tools, of course. Those that crossed my radar are Posy and Rye. Posy is a project of Nathaniel J. Smith (of trio fame), Rye is a project of Armin Ronacher (of Flask fame). The vision for both of them is to manage Python interpreters and projects, but not have a custom build backend (instead using something like hatchling). Posy is built on top of PyBI (a format for distributing binaries of Python interpreters, proposed by Smith in draft PEP 711), Rye uses Gregory Szorc’s pre-built Pythons. Rye seems to be fairly complete and usable, Posy is right now a PoC of the PyBI format, and only offers a REPL with pre-installed packages.

Both Posy and Rye are written in Rust. On the one hand, it makes sense that the part that manages Python interpreters is not written in Python, because that would require a separate Python, not managed by Posy/Rye, to run those tools. But Rye also has its own pyproject.toml parser in Rust, and many of its commands are implemented mostly or largely using Rust (sometimes also calling one-off Python scripts; although the main tasks of creating venvs, installing packages, and working with lockfiles are handed off to venv, pip, and pip-tools respectively).

Speaking of Rust and Python, there’s been another project in that vein that has grown a lot (and gathered a lot of funding) in the past year. That project is Ruff, which is a linter and code formatter. Ruff formats Python code, and is written in Rust. This means it’s 10–100× faster than existing tools written in Python (according to Ruff’s own benchmarks). Fast is good, I guess, but what does this say about Python? Is the fact that package tools (which aren’t rocket science, maybe except for fast dependency solvers, and which often need access to Python internals to do their job) and code formatters (which require a deep understanding of Python syntax, and parsing Python sources to ASTs, something easy by the ast Python module) are written in another language? Does this trend make Python a toy language (as it is also often considered a glue language for NumPy and friends)? Also, why should contributing to a tool important to many Python developers require learning Rust?

The standards

Last time we looked at packaging standards, we focused on PEP 582. It proposed the introduction of __pypackages__, which would be a place for third-party packages to be installed to locally, on a per-project basis, without involving virtual environments, similarly to what node_modules is for node. The PEP was ultimately rejected in March 2023. The PEP wasn’t perfect, and some of its choices were questionable or insufficient (such as not recursively searching for __pypackages__ in parent directories, or focusing on simple use-cases only). No new standards for something in that vein (with a better design) were proposed to this day.

Another contentious topic is lock files. Lock files for packaging systems are useful for reproducible dependency installations. The lock file records all installed packages (i.e. includes transitive dependencies) and their versions. Lock files often include checksums (like sha512) of the installed packages, and they often support telling apart packages installed via different groups of dependencies (runtime, buildtime, optional, development, etc.).

The classic way of achieving this goal are requirements.txt files. They are specific to pip, and they only contain a list of packages, versions, and possibly checksums. Those files can be generated by pip freeze, or the third-party pip-compile from pip-tools. pip freeze is very basic, pip-compile can’t handle different groups of dependencies other than making multiple requirements.in files, compiling them, and hoping there are no conflicts.

Pipenv, Poetry, and PDM have their own lockfile implementations, incompatible with one another. Rye piggybacks on top of pip-tools. Hatch doesn’t have anything in core; they’re waiting for a standard implementation (there are some plugins though). PEP 665 was rejected in January 2022. Its author, Brett Cannon, is working on a PoC of something that might become a standard (named mousebender).

This is the danger of the working model adopted by the Python packaging world. Even for something as simple as lock files, there are at least four incompatible standards. An attempt at a specification was rejected due to “lukewarm reception”, even though there exist at least four implementations which are achieving roughly the same goals, and other ecosystems also went through this before.

Another thing important to Python are extension modules. Extension modules are written in C, and they are usually used to interact with libraries written in other languages (and also sometimes for performance). Poetry, PDM, and Hatchling don’t really support building extension modules. Setuptools does; SciPy and NumPy migrated from their custom numpy.distutils to Meson. The team behind the PyO3 Rust bindings for Python develops Maturin, which allows for building Rust-based extension modules — but it’s not useful if you’re working with C.

There weren’t many packaging-related standards that were accepted in 2023. A standard worth mentioning is PEP 668, which allows distributors to prevent pip from working (to avoid breaking distro-owned site packages) by adding an EXTERNALLY-MANAGED file. It was accepted in June 2022, but pip only implemented support for it in January 2023, and many distros already have enabled this feature in 2023. Preventing broken systems is a good thing.

But some standards did make it through. Minor and small ones aside, the most prominent 2023 standard would be PEP 723: inline script metadata. It allows to add a comment block at the top of the file, that specifies the dependencies and the minimum Python version in a way that can be consumed by tools. Is it super useful? I don’t think so; setting up a project with pyproject.toml would easily allow things to grow. If you’re sending something via a GitHub gist, just make a repo. If you’re sending something by e-mail, just tar the folder. That approach promotes messy programming without source control.

The future…

Looking at the Python Packaging Discourse, there were some discussions about ways to improve things.

For example, this discussion about porting off setup.py was started by Gregory Szorc, who had a long list of complaints, pointing out the issues with the communication from the packaging world, and documentation mess (his post is worth a read, or at least a skim, because it’s long and full of packaging failures). There’s one page which recommends setuptools, another which has four options with Hatchling as a default, and another still promoting Pipenv. We’ve seen this a year ago, nothing changed in that regard. Some people tried finding solutions, some people shared their opinions… and then the Discourse moderator decided to protect his PyPA friends from having to read user feedback and locked the thread.

Many other threads about visions were had, like the one about 10-year views or about singular packaging tools. The strategy discussions, based on the user survey, had a second part (the first one concluded in January 2023), but it saw less posts than the first one, and discussions did not continue (and there were discussions about how to hold the discussions). There are plans to create a packaging council — design-by-committee at its finest.

But all those discussions, even when not locked by an overzealous moderator, haven’t had any meaningful effect. The packaging ecosystem is still severely fragmented and confusing. The PyPA docs and tutorials still contradict each other. The PyPA-affiliated tools still have less features than the unaffiliated competition (even the upstart Rye has some form of lockfiles, unlike Hatch or Flit), and going by the PEP 517 build backend usage statistics, they are more popular than the modern PyPA tools. The authors of similar yet competing tools have not joined forces to produce the One True Packaging Tool.

Conclusion

I understand the PyPA is a loose association of volunteers. It is sometimes said the name Python Packaging Authority was originally a joke. However, they are also the group that maintains all the packaging standards, so they are the authority when it comes to packaging. For example, PEP 668 starts with a warning block saying it’s a historical document, and the up-to-date version of the specification is on PyPA’s site (as well as a bunch of other packaging specs).

The PyPA should shut down or merge some duplicate projects, and work with the community (including maintainers of non-PyPA projects) to build One True Packaging Tool. To make things easier. To avoid writing code that does largely the same thing 5 times. To make sure thousands of projects don’t depend on tools with a bus factor of 1 or 2. To turn packaging from a problem and an insurmountable obstacle to something that just works™, something that an average developer doesn’t need to think about.

It’s not rocket science. Tons of languages, big and small, have a coherent packaging ecosystem (just read last year’s post for some examples of how simple it can be). Instead of focusing on specifications and governance, focus on producing one comprehensive, usable, user-friendly tool.

Discuss below or on Hacker News.

Footnotes

The plethora of tools

There are many packaging-related tools in Python. All of them with different authors, lineages, and often different opinions, although most of them are now unified under the Python Packaging Authority (PyPA) umbrella. Let’s take a look at them.

Tooling proliferation and the Python Package Authority

The previous sections mentioned 14 (fourteen!) distinct tools. As we’ll discover soon, that’s at least 12 too many. Let’s try to compare them.

First, let’s define nine things that we would expect packaging tools to do:

1. Manage environments

2. Install packages

3. Package/develop apps

4. Package libraries

5. Package C extension modules

6. Install in editable mode

7. Lock dependencies

8. Support pyproject.toml files

9. Upload to PyPI

You should pay close attention to the Maintainer column in the table. The vast majority of them are maintained by PyPA, the Python Packaging Authority. Even more curiously, the two tools that have the most “Yes” values (Poetry and PDM) are not maintained by the PyPA, but instead other people completely independent of them and not participating in the working group. So, is the working group successful, if it cannot produce one fully-featured tool? Is the group successful if it has multiple projects with overlapping responsibilities? Should the group focus their efforts on standards like PEP 517, which is a common API for packaging tools, and which also encourages the creation of even more incompatible and competing tools?

Most importantly: which tool should a beginner use? The PyPA has a few guides and tutorials, one is using pip + venv, another is using pipenv (why would you still do that?), and another tutorial that lets you pick between Hatchling (hatch’s build backend), setuptools, Flit, and PDM, without explaining the differences between them—and without using any environment tools, and without using Hatch’s/PDM’s build and PyPI upload features (instead opting to use python -m build and twine). The concept of virtual environments can be very confusing for beginners, and managing virtual environments is difficult if everyone has incompatible opinions about it.

It is also notable that PEP 20, the Zen of Python, states this:

There should be one-- and preferably only one --obvious way to do it.

Python packaging definitely does not follow it [1]. There are 14 ways, and none of them is obvious or the only good one. All in all, this is an unsalvageable mess. Why can’t Python pick one tool? What does the competition do? We’ll look at this in a minute. But first, let’s talk about the elephant in the room: Python virtual environments.

Does Python really need virtual environments?

Python relies on virtual environments for separation between projects. Virtual environments (aka virtualenvs or venvs) are folders with symlinks to a system-installed Python, and their own set of site-packages. There are a few problems with them:

How everyone else is doing it

We’ll look at two ecosystems. We’ll start with JavaScript/Node.js (with npm), and then we’ll look at the C#/.NET (with dotnet CLI/MSBuild) ecosystem for comparison. We’ll demonstrate a sample flow of making a project, installing dependencies in it, and running things. If you’re familiar with those ecosystems and want to skip the examples, continue with How is Node better than Python? and Are those ecosystems’ tools perfect?. Otherwise, read on.

$ mkdir mynpmproject
$ cd mynpmproject
$ npm init
…answer a few questions…
$ ls
package.json

$ npm install --save is-even

added 5 packages, and audited 6 packages in 2s

found 0 vulnerabilities

$ ls
node_modules/  package.json  package-lock.json
$ ls node_modules
is-buffer/  is-even/  is-number/  is-odd/  kind-of/

{
  "name": "mynpmproject",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "is-even": "^1.0.0"
  }
}

$ cat index.js
var isEven = require('is-even');
console.log(isEven(0));

$ node index.js
true

$ rm -rf node_modules/is-odd
$ node index.js
node:internal/modules/cjs/loader:998
  throw err;
  ^

Error: Cannot find module 'is-odd'
Require stack:
- /tmp/mynpmproject/node_modules/is-even/index.js
- /tmp/mynpmproject/index.js
    at Module._resolveFilename (node:internal/modules/cjs/loader:995:15)
    at Module._load (node:internal/modules/cjs/loader:841:27)
    at Module.require (node:internal/modules/cjs/loader:1061:19)
    at require (node:internal/modules/cjs/helpers:103:18)
    at Object.<anonymous> (/tmp/mynpmproject/node_modules/is-even/index.js:10:13)
    at Module._compile (node:internal/modules/cjs/loader:1159:14)
    at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
    at Module.load (node:internal/modules/cjs/loader:1037:32)
    at Module._load (node:internal/modules/cjs/loader:878:12)
    at Module.require (node:internal/modules/cjs/loader:1061:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/tmp/mynpmproject/node_modules/is-even/index.js',
    '/tmp/mynpmproject/index.js'
  ]
}

Node.js v18.12.1

$ pwd
/tmp/mynpmproject
$ npm install

added 1 package, and audited 6 packages in 436ms

found 0 vulnerabilities
$ node /tmp/mynpmproject/index.js
true
$ cd ~
$ node /tmp/mynpmproject/index.js
true

$ su -
# node /tmp/mynpmproject/index.js
true
# sudo -u nginx node /tmp/mynpmproject/index.js
true

$ cd /tmp
$ mv mynpmproject mynodeproject
$ node /tmp/mynodeproject/index.js
true

$ mkdir mydotnetproject
$ cd mydotnetproject
$ dotnet new console
The template "Console App" was created successfully.

Processing post-creation actions...
Running 'dotnet restore' on /tmp/mydotnetproject/mydotnetproject.csproj...
  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 92 ms).
Restore succeeded.
$ ls
mydotnetproject.csproj  obj/  Program.cs

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="AutoFixture" Version="4.17.0" />
  </ItemGroup>

</Project>

using AutoFixture;
var fixture = new Fixture();
var a = fixture.Create<int>();
var b = fixture.Create<int>();
var result = a + b == b + a;
Console.WriteLine(result ? "Math is working": "Math is broken");

$ dotnet run
Math is working

$ dotnet publish
$ ls bin/Debug/net6.0/publish
AutoFixture.dll*  Fare.dll*  mydotnetproject*  mydotnetproject.deps.json  mydotnetproject.dll  mydotnetproject.pdb  mydotnetproject.runtimeconfig.json
$ du -h bin/Debug/net6.0/publish
424K    bin/Debug/net6.0/publish
$ bin/Debug/net6.0/publish/mydotnetproject
Math is working

$ rm bin/Debug/net6.0/publish/AutoFixture.dll
$ bin/Debug/net6.0/publish/mydotnetproject
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'. The system cannot find the file specified.

File name: 'AutoFixture, Version=4.17.0.0, Culture=neutral, PublicKeyToken=b24654c590009d4f'
[1]    45060 IOT instruction (core dumped)  bin/Debug/net6.0/publish/mydotnetproject

$ rm -rf bin obj  # clean up, just in case
$ dotnet publish --sc -r linux-x64 -p:PublishSingleFile=true -o myoutput
Microsoft (R) Build Engine version 17.0.1+b177f8fa7 for .NET
Copyright (C) Microsoft Corporation. All rights reserved.

  Determining projects to restore...
  Restored /tmp/mydotnetproject/mydotnetproject.csproj (in 4.09 sec).
  mydotnetproject -> /tmp/mydotnetproject/bin/Debug/net6.0/linux-x64/mydotnetproject.dll
  mydotnetproject -> /tmp/mydotnetproject/myoutput/
$ ls myoutput
mydotnetproject*  mydotnetproject.pdb
$ myoutput/mydotnetproject
Math is working
$ du -h myoutput/*
62M     myoutput/mydotnetproject
12K     myoutput/mydotnetproject.pdb
$ file -k myoutput/mydotnetproject
myoutput/mydotnetproject: ELF 64-bit LSB pie executable, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=47637c667797007d777f4322729d89e7fa53a870, for GNU/Linux 2.6.32, stripped, too many notes (256)\012- data
$ file -k myoutput/mydotnetproject.pdb
myoutput/mydotnetproject.pdb: Microsoft Roslyn C# debugging symbols version 1.0\012- data

PEP 582: the future of Python packaging?

Recall that when introducing PDM, I mentioned PEP 582. This PEP defines a __pypackages__ directory. This directory would be taken into consideration by Python when looking for imports. It would behave similarly to node_modules. Since there will be no symlinks to the system Python, it will resolve the issues with moving the virtual environment. Because the packages live in the project, there is no problem with sharing a project directory between multiple system users. It might even be possible for different computers (but with the same Python version and OS) to share the __pypackages__ directory (in some specific cases). The proposed __pypackages__ directory structure has lib/python3.10/site-packages/ subfolders, which still makes the “reinstall on Python upgrade” step mandatory, but it doesn’t apply to minor version upgrades, and if you’re dealing with a pure-Python dependency tree, mv __pypackages__/lib/python3.10 __pypackages__/lib/python3.11 might just work. This structure does make sense for binary dependencies, or for dependencies necessary only on older Python versions, as it allows you to use multiple Python versions with the same project directory. The PEP does not say anything about sharing __pypackages__ between projects, but you could probably solve that problem with symlinks (assuming the tooling doesn’t care if the directory is a symlink, and it shouldn’t care IMO).

While PEP 582 is a great vision, and it would simplify many package-related workflows, it hasn’t seen much care from the powers-that-be. The PEP was proposed in May 2018, and there’s even a usable implementation that’s less than 50 lines of code, there hasn’t been much progress on having it accepted and implemented in Python proper. However, PDM does not care, and it allows you to enable the future on your own machine.

$ eval "$(pdm --pep582)"

$ mkdir mypdmproject
$ cd mypdmproject
$ pdm init
Creating a pyproject.toml for PDM...
Please enter the Python interpreter to use
0. /usr/bin/python (3.11)
1. /usr/bin/python3.11 (3.11)
2. /usr/bin/python2.7 (2.7)
Please select (0): 1
Using Python interpreter: /usr/bin/python3.11 (3.11)
Would you like to create a virtualenv with /usr/bin/python3.11? [y/n] (y): n
You are using the PEP 582 mode, no virtualenv is created.
For more info, please visit https://peps.python.org/pep-0582/
Is the project a library that will be uploaded to PyPI [y/n] (n): n
License(SPDX name) (MIT):
Author name (Chris Warrick):
Author email (…):
Python requires('*' to allow any) (>=3.11):
Changes are written to pyproject.toml.
$ ls
pyproject.toml
$ pdm add requests
Adding packages to default dependencies: requests
🔒 Lock successful
Changes are written to pdm.lock.
Changes are written to pyproject.toml.
Synchronizing working set with lock file: 5 to add, 0 to update, 0 to remove

  ✔ Install charset-normalizer 2.1.1 successful
  ✔ Install certifi 2022.12.7 successful
  ✔ Install idna 3.4 successful
  ✔ Install requests 2.28.1 successful
  ✔ Install urllib3 1.26.13 successful

🎉 All complete!

[tool.pdm]

[project]
name = ""
version = ""
description = ""
authors = [
    {name = "Chris Warrick", email = "…"},
]
dependencies = [
    "requests>=2.28.1",
]
requires-python = ">=3.11"
license = {text = "MIT"}

$ ls
pdm.lock  __pypackages__/  pyproject.toml
$ ls __pypackages__
3.11/
$ ls __pypackages__/3.11
bin/  include/  lib/
$ ls __pypackages__/3.11/lib
certifi/             certifi-2022.12.7.dist-info/
idna/                idna-3.4.dist-info/
charset_normalizer/  charset_normalizer-2.1.1.dist-info/
requests/            requests-2.28.1.dist-info/
urllib3/             urllib3-1.26.13.dist-info/

import requests
print(requests.__file__)
r = requests.get("https://chriswarrick.com/")
print(r.text[:15])

$ python mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

$ rm -rf __pypackages__/3.11/lib/urllib3*
$ python mypdmproject.py
Traceback (most recent call last):
  File "/tmp/mypdmproject/mypdmproject.py", line 1, in <module>
    import requests
  File "/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py", line 43, in <module>
    import urllib3
ModuleNotFoundError: No module named 'urllib3'

$ pdm install
Synchronizing working set with lock file: 1 to add, 0 to update, 0 to remove

  ✔ Install urllib3 1.26.13 successful

🎉 All complete!
$ pwd
/tmp/mypdmproject
$ cd ~
$ python /tmp/mypdmproject/mypdmproject.py
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>
# su -s /bin/bash -c 'eval "$(/tmp/pdmvenv/bin/pdm --pep582 bash)"; python /tmp/mypdmproject/mypdmproject.py' - nobody
su: warning: cannot change directory to /nonexistent: No such file or directory
/tmp/mypdmproject/__pypackages__/3.11/lib/requests/__init__.py
<!DOCTYPE html>

PyPA versus reality: packaging survey results and PyPA reaction

Some time ago, the PSF ran a survey on packaging. Over 8000 people responded. The users have spoken:

• Most people think packaging is too complex.

• An overwhelming majority prefers using just a single tool.

• Most people also think the existence of multiple tools is not beneficial for the Python packaging ecosystem.

• Virtually everyone would prefer a clearly defined official workflow.

• Over 50% of responses think tools for other ecosystems are better at managing dependencies and installing packages.

The next step after this survey was for the packaging community to discuss its results and try to come up with a new packaging strategy. The first post from Shamika Mohanan (the Packaging Project Manager at PSF) that triggered the discussion also focused heavily on the users’ vision to unify packaging tools and to have One True Tool. This discussion was open to people involved with the packaging world; many participants of the discussion are involved with PyPA, and I don’t think I’ve seen a single comment from the people behind Poetry or PDM.

Most of the thread ended up being discussion of binary extensions, including discussions of how to help tool proliferation by making it possible for tools that aren’t setuptools to build binary extensions. There was also a lot of focus on the scientific community’s issues with libraries with native code, heavily rooted in C/C++, and with attempts to replace Conda with new PyPA-approved tools. The “unified tool” for everyone else was mentioned in some posts, but they were certainly the minority.

Some PyPA members talked about a UX analysis, and that they expect the unified tool to be re-exporting functionality from existing tools—which immediately raises the question: which tools should it export functionality from and why? Is pip install unified-packaging-tool going to bring in all fourteen? Is the fact that users are unhappy with what they have, and many of them would be happy with something lke npm/dotnet/cargo, not enough to determine the UX direction of the unified tool?

Some of them are also against breaking existing workflows. Is a unified packaging tool going to work for every single user? Definitely not. But are there that many distinct basic workflows? If we ignore things that border on bikeshedding, such as src vs no-src, or venv locations, are there that many workflows to consider? Someone making a library and someone making an application do have different needs (e.g. with regard to publishing the package or acceptable dependency versions). Someone working with C extensions (or extensions using something like Cython) may have different needs, but their needs would usually be a superset of the needs of someone working on a pure-Python project. The scientific community might have more specialized needs, related to complex non-Python parts, but I am positive many of their points could be solved by the unified tool as well, even if it’s not by the time this tool reaches v1.0. It is also possible that the scientific community might prefer to stay with Conda, or with some evolution of it that brings it closer in line with the Unified Packaging Tool but also solves the scientists’ needs better than a tool also solving the non-scientists’ needs can.

Then there’s a discussion about the existing tools and which one is the tool for the future. The maintainer of Hatch (Ofek Lev) says that Hatch can provide the “unified UX”. But do the maintainers of Poetry or PDM agree? Poetry seems to be far more active than Hatch, going by GitHub issues, and it’s also worth noting that Hatch’s bus factor is 1 (with Ofek Lev responsible for 542 out of 576 commits to the master branch). Russell Keith-Magee from BeeWare has highlighted the fact that tooling aside, the PyPA does a bad job at communicating things. Russell mentioned that one of PyPA tutorials now uses Hatch, but there is no way to know if the PyPA considers Hatch to be the future, are people supposed to migrate onto Hatch, and is Flit, another recent PyPA tool, now useless? Russell also makes good points about focusing efforts: should people focus on helping Hatch support extension modules (which, according to the Hatch maintainer, is the last scenario requiring setuptools; other participants note that you can already build native code without setuptools), or should people focus on improving setuptools compatibility with PEP 517?

There were also some people stating their opinions on unifying things in various ways—and many of them are against unifying things. There were some voices of reason, like that of Russell Keith-Magee, or of Simon Notley, who correctly noticed the thread fails to resolve problems of developers, who are confused about packaging, and don’t understand the different choices available and how they interoperate. Simon does agree that native dependencies are important and happen often in Python projects (and so do I), but the users who responded to the survey had something else in mind — as exemplified by the discussion opening post, mentioning the user expecting the simplicity of Rust’s cargo, and by the survey results. 70% of the survey respondents also use npm, so many Python users have already seen the simpler workflows. The survey respondents were also asked to rank a few focus areas based on importance. “Making Python packaging better serve common use cases and workflows” was ranked first out of the provided options [8] by 3248 participants. “Supporting a wider range of use cases (e.g. edge cases, etc.)” was ranked first by 379 people, and it was the least important in the minds of 2989 people.

One more point that highlights the detachment of packaging folk from reality was mentioned by Anderson Bravalheri. To Anderson, a new unified tool would be disrespectful of the work the maintainers of the existing tools put into maintaining them, and disrespectful of users who had to adapt to the packaging mess. This point is completely absurd. Was the replacement of MS-DOS/Windows 9x and Classic Mac OS with Windows NT and Mac OS X OS X macOS disrespectful to their respective designers, and the users who had to adapt to manually configuring minutiae, figuring out how to get all your software and hardware to run with weird limitations that were necessary in the 1980s, and the system crashing every once in a while? Was the replacement of horses with cars disrespectful to horses, and the people who were removing horse manure from the streets? Was the replacement of the Ford Model T with faster, safer, more efficient, and easier to use cars disrespectful to Henry Ford? Technology comes and goes, and sometimes, getting an improvement means we need to get rid of the old stuff. This applies outside of technology, too—you could come up with many examples of change in the world, which might have put some people out of power, but has greatly improved the lives of millions of people (the fall of communism in Europe, for example). Also, going back to the technology world of today, this sentiment suggests Anderson is far too attached to the software they write—is this a healthy approach?

Nobody raised PEP 582 or the complexity of virtual environments. It might not be visible from the ivory towers of packaging tool maintainers, who have years of experience dealing with them, but it certainly does exist for regular people, for people who think the Python provided by their Linux distro is good enough, and especially for people for whom Python is their introduction to programming.

I would like to once again highlight: that’s not just the opinion of one random rambling Chris. The opinion that Python packaging needs to be simplified and unified is held by about half of the 8774 people who took the survey.

But here’s one more interesting thing: Discourse, the platform that the discussion was held on, shows the number of times a link was clicked. Granted, this count might not be always accurate, but if we assume it is, the link to the results summary was clicked only 14 times (as of 2023-01-14 21:20 UTC). The discussion has 28 participants and 2.2k views. If we believe the link click counter, half of the discussion participants did not even bother reading what the people think.

Summary

Python packaging is a mess, and it always has been. There are tons of tools, mostly incompatible with each other, and no tool can solve all problems (especially no tool from the PyPA). PDM is really close to the ideal, since it can do away with the overhead of managing virtual environments—which is hopefully the future of Python packaging, or the 2010s of Node.js packaging (although it is not going to be the 2023 of Python packaging, considering the Steering Council rejection). Perhaps in a few years, Python developers (and more importantly, Python learners!) will be able to just pip install (or pdm install?) what they need, without worrying about some “virtual environment” thing, that is separate but not quite from a system Python, and that is not a virtual machine. Python needs less tools, not more.

Furthermore, I consider that the PyPA must be destroyed. The strategy discussion highlights the fact that they are unable to make Python packaging work the way the users expect. The PyPA should focus on producing one good tool, and on getting PEP 582 into Python. A good way to achieve this would be to put its resources behind PDM. The issues with native code and binary wheels are important, but plain-Python workflows, or workflows with straightforward binary dependencies, are much more common, and need to be improved. This improvement needs to happen now.

Discuss in the comments below, on Hacker News, or on Reddit.

Footnotes

Revision History

This post got amended in April 2023 with an update about the SC rejection of PEP 582 (in a new subsection and in the Summary section).

Starting point

def old(foo, bar):
    """This is old's docstring."""
    print(foo, bar)
    return foo + bar


def new(prefix, foo, *args, **kwargs):
    return old(prefix + foo, *args, **kwargs)

Let’s test it.

>>> o = old('a', 'b')
a b
>>> n = new('!', 'a', 'b')
!a b
>>> print(o, n, sep=' - ')
ab - !ab
>>> help(old)
Help on function old in module __main__:

old(foo, bar)
    This is old's docstring.

>>> help(new)
Help on function new in module __main__:

new(prefix, foo, *args, **kwargs)

The last line is not exactly informative — it doesn’t tell us that we need to pass bar as an argument. Sure, you could define new as just (prefix, foo, bar) — but that means every change to old requires editing new as well. So, not ideal. Let’s try to fix this.

The existing infrastructure: functools.wraps

First, let’s start with the basic facility Python already has. The standard library already comes with functools.wraps and functools.update_wrapper.

If you’ve never heard of those two functions, here’s a crash course:

def decorator(f):
    @functools.wraps(f)
    def wrapper(*args, **kwargs):
        print("Inside wrapper")
        f(*args, **kwargs)
    return wrapper

@decorator
def square(n: float) -> float:
    """Square a number."""
    return n * n

If we try to inspect the square function, we’ll see the original name, arguments, annotations, and the docstring. If we ran this code again, but with the @functools.wraps(f) line commented out, we would only see wrapper(*args, **kwargs).

This approach gives us a hint of what we need to do. However, if we apply wraps (or update_wrapper, which is what wraps ends up calling) to our function, it will only have foo and bar as arguments, and its name will be displayed as old.

So, let’s take a look at functools.update_wrapper. What does it do? Two things:

• copy some attributes from the old function to the new one (__module__, __name__, __qualname__, __doc__, __annotations__)

• update __dict__ of the new function

• set wrapper.__wrapped__

If we try to experiment with it — by changing the list of things to copy, for example — we’ll find out that the annotations, the docstring, and the displayed name come from the copied attributes, but the signature itself is apparently taken from __wrapped__.

Further investigation reveals this fact about inspect.signature:

inspect.signature(callable, *, follow_wrapped=True)

New in version 3.5: follow_wrapped parameter. Pass False to get a signature of callable specifically (callable.__wrapped__ will not be used to unwrap decorated callables.)

And so, this is our end goal:

Craft a function with a specific signature (that merges old and new) and set it as new.__wrapped__.

But first, we need to talk about parallel universes.

Or actually, code objects.

Defining a function programmatically

Let’s try an experiment.

>>> def foo(bar): pass
>>> foo.__wrapped__ = lambda x, y: None
>>> help(foo)
foo(x, y)

So, there are two ways to do this. The first one would be to generate a string with the signature and just use eval to get a __wrapped__ function. But that would be cheating, and honestly, quite boring. (The inspect module could help us with preparing the string.) The second one? Create code objects manually.

code(argcount, kwonlyargcount, nlocals, stacksize, flags, codestring,
    constants, names, varnames, filename, name, firstlineno,
    lnotab[, freevars[, cellvars]])

Create a code object.  Not for the faint of heart.

>>> def f(a, b=1, c=2, *, d=3): pass
>>> inspect.getfullargspec(f)
FullArgSpec(args=['a', 'b', 'c'], varargs=None, varkw=None, defaults=(1, 2), kwonlyargs=['d'], kwonlydefaults={'d': 3}, annotations={})

Final results

Before I show you the code, let’s test it out:

# old defined as before
@merge_args(old)
def new(prefix, foo, *args, **kwargs):
    return old(prefix + foo, *args, **kwargs)

And the end result — help(new) says:

new(prefix, foo, bar)
    This is old's docstring.

We did it!

The code is available on GitHub and on PyPI (pip install merge_args). There’s also an extensive test suite.

PS. you might be interested in another related post of mine, in which I reverse-engineer the compilation of a function: Gynvael’s Mission 11 (en): Python bytecode reverse-engineering

Install

The best tool that can be used to create virtual environments is the venv module, which is part of the standard library since Python 3.3.

venv is built into Python, and most users don’t need to install anything. However, Debian/Ubuntu users will need to run sudo apt-get install python3-venv to make it work (due to Debian not installing some components that venv needs by default). [1]

The alternative (and original, and previously standard) virtual environment tool is virtualenv. It works with Python 2.7, and has a couple extra fetures (that you generally won’t need). virtualenv can be installed with your system package manager, or pip install --user virtualenv.

Which one to use? Probably venv. Both tools achieve the same goal in similar ways. And if one of them does not work, you can try the other and it might just work better.

(Terminology note: most of the time, the names of both tools are used interchargeably, “venv” was often used as an abbreviation for “virtualenv” before the stdlib tool was created)

Create

To create a virtual environment named env, you need to run the venv tool with the Python you want to use in that environment.

Linux:   $ python3 -m venv env
Windows: > py -m venv env

or, if you’re using virtualenv:

$ python3 -m virtualenv env
> py -m virtualenv env

Afterwards, you will end up with a folder named env that contains folders named bin (Scripts on Windows — contains executables and scripts installed by packages, including python), lib (contains code), and include (contains C headers).

Both tools install pip and setuptools, but venv does not ship with wheel. In addition, the default versions tend to be more-or-less outdated. Let’s upgrade them real quick: [2]

$ env/bin/python -m pip install --upgrade pip setuptools wheel
> env\Scripts\python -m pip install --upgrade pip setuptools wheel

Use

There are three ways of working with virtual environments interactively (in a shell):

• activation (run source env/bin/activate on *nix; env\Scripts\activate on Windows) — it simplifies work and requires less typing, although it can sometimes fail to work properly. (After installing scripts, hash -r may be necessary on *nix to use them.)

• executing env/bin/python (env\Scripts\python) and other scripts directly, as activation only changes $PATH and some helper variables — those variables are not mandatory for operation, running the correct python is, and that method is failsafe.

• in subshells (IMO, it’s bad UX)

Whichever method you use, you must remember that without doing any of these things, you will still be working with the system Python.

For non-interactive work (eg. crontab entries, system services, etc.), activation and subshells are not viable solutions. In these cases, you must always use the full path to Python.

Here are some usage examples (paths can be relative, of course):

## *nix, activation ##
$ source /path/to/env/bin/activate
(env)$ pip install Django
(env)$ deactivate

## *nix, manual execution ##
$ /path/to/env/bin/pip install Django

## Windows, activation ##
> C:\path\to\env\Scripts\activate
(venv)> pip install Django
(venv)> deactivate

## Windows, manual execution ##
> C:\path\to\env\Scripts\pip install Django

## Windows, updating pip/setuptools/wheel ##
> C:\path\to\env\Scripts\python -m pip install -U pip setuptools wheel

The same principle applies to running Python itself, or any other script installed by a package. (With Django’s manage.py, calling it as ./manage.py requires activation, or you can run venv/bin/python manage.py.)

$ oldenv/bin/pip freeze > requirements.txt
$ python3 -m venv newenv
$ newenv/bin/pip install -r requirements.txt
(You may rm -rf oldenv now if you desire)

Frequently Asked Questions

Footnotes

A 2020 update (updated)

This blog post was written in 2018, and it’s still pretty accurate when it comes to the criticisms of Pipenv, but something else happened since then.

No release was made between November 2018 and May 2020. Pipenv was effectively dead for 1.5 years, and the state of Pipenv maintenance is alarming.

A release of Pipenv was made in late 2018 (aptly named v2018.11.26). But then, there was silence. New commits were made (on the order of 600-700 by the end of the year). People asked for new releases, in more or less strong words, in May 2019, then in October, and again in December. Many people — including yours truly, in this post — considered Pipenv dead. On 13th December 2019, the current maintainer claimed a new release is almost finished.

Pipenv 2019/2020 was vaporware for five months. Not much progress was made since the December post until March 2020, when an issue from 2018 was renamed the March 2020 Release Tracking Issue. Some progress was happening, and many release dates were given, but delays stacked up. March became April. The first beta release was promised by 21st April, it was delayed until the 29th. The final release was scheduled for a week from that, but it didn’t happen. Finally, Beta 2 came out on 20th May 2020, and the final release landed as v2020.5.28.

If you read further into the post, you’ll encounter a chapter titled The break-neck pace of Pipenv. Am I being a hypocrite right now? No, not at all. Projects that are being depended on, such as a package manager, should have clear policies about how they’re maintained. A new release when the maintainer feels like adding a feature is too often. A new release every 1.5 years is not often enough. And silence from maintainers, when faced with questions about releases, is simply unacceptable. Pip, for example, has updates every few months in a fairly stable pace (with the exception of emergency bugfix releases), and pip has years of development behind it, unlike the fairly new Pipenv.

And even if the May release was successful, you can’t be sure about any future releases, and what will happen with Pipenv. At the same time, Pipenv isn’t a good tool, as this post tries to explain — those criticisms are still valid, since they are at the core of what Pipenv is. Instead, perhaps consider using pip-tools for locking dependencies? It does one thing, and one thing well. It doesn’t enforce any specific structures on users, and supports any workflow you have. (If you don’t need to lock dependencies, pip + venv will suffice.)

“Officially recommended tool”, or how we got here

“Pipenv — the officially recommended Python packaging tool from Python.org, free (as in freedom).”

Pipenv’s README used to have a version of the above line in their README for many months: it was added on 2017-08-31 and eventually disappeared on 2018-05-19. For a short while (2018-05-16), it was clarified (managing application dependencies, and PyPA instead of Python.org), and for about 15 minutes, the tagline called Pipenv the world’s worst or something along these lines (this coming from the maintainer).

The README tagline claimed that Pipenv is the be-all, end-all of Python packaging. The problem is: it isn’t that. There are some use cases that benefit from Pipenv, but for many others, trying to use that tool will only lead to frustration. We will explore this issue later.

Another issue with this tagline was the Python.org and official parts. The thing that made it “official” was a short tutorial [1] on packaging.python.org, which is the PyPA’s packaging user guide. Also of note is the Python.org domain used. It makes it sound as if Pipenv was endorsed by the Python core team. PyPA (Python Packaging Authority) is a separate organization — they are responsible for the packaging parts (including pypi.org, setuptools, pip, wheel, virtualenv, etc.) of Python. This made the endorsement misleading. Of course, PyPA is a valued part of the Python world; an endorsement by the core team — say, inclusion in official Python distributions — is something far more important.

This tagline has led to many discussions and flamewars, perhaps with this Reddit thread from May being the most heated and most important. The change was the direct result of this Reddit thread. I recommend reading this thread in full.

What pipenv does

We’ve already learned that Pipenv is used to manage application dependencies. Let’s learn what that term really means.

What pipenv doesn’t do

The workflow I mentioned above looks pretty reasonable, right? There are some deficiencies, but other than that, it seems to work well. The main issue with Pipenv is: it works with one workflow, and one workflow only. Try to do anything else, and you end up facing multiple obstacles.

$ ~/virtualenvs/foobar/bin/python ~/Downloads/repro.py

$ cd ~/git/foobar
$ pipenv run python ~/Downloads/repro.py

$ cd ~/git/foobar
$ pipenv shell
(foobar-Mwd1l2m9)$ cd ~/Downloads
(foobar-Mwd1l2m9)$ python repro.py
(foobar-Mwd1l2m9)$ exit  # (not deactivate!)

$ cd ~/Downloads
$ workon foobar
(foobar)$ python repro.py
(foobar)$ deactivate

The part where I try to measure times

Pipenv is famous for being slow. But how slow is it really? I put it to the test. I used two test environments:

• Remote: a DigitalOcean VPS, the cheapest option (1 vCPU), Python 3.6/Fedora 28, in Frankfurt

• Local: my 2015 13” MacBook Pro (base model), Python 3.7, on a rather slow Internet connection (10 Mbps on a good day, and the test was not performed on one of them)

Both were runninng Pipenv 2018.7.1, installed from pip.

And with the following cache setups:

• Removed: ~/.cache/pipenv removed

• Partial: rm -rf ~/.cache/pipenv/depcache-py*.json ~/.cache/pipenv/hash-cache/

• Kept: no changes done from previous run

Well, turns out Pipenv likes doing strange things with caching and locking. A look at the Activity Monitor hinted that there is network activity going on when Pipenv displays its Locking [packages] dependencies... line and hangs. Now, the docs don’t tell you that. The most atrocious example was a local Nikola install that was done in two runs: the first pipenv install Nikola run was interrupted [4] right after it was done installing packages, so the cache had all the necessary wheels in it. The install took 10 minutes and 7 seconds, 9:50 of which were taken by locking dependencies and installing the locked dependencies — so, roughly nine and a half minutes were spent staring at a static screen, with the tool doing something in the background — and Pipenv doesn’t tell you what happens in this phase.

(Updated 2018-07-22: In the pipenv measurements: the first entry is the total time of pipenv executon. The second is the long wait for pipenv to do its “main” job: locking dependencies and installing them. The timing starts when pipenv starts locking dependencies and ends when the prompt appears. The third item is pipenv’s reported installation time. So, pipenv install ⊇ locking/installing ⊇ Pipfile.lock install.)

Alternative tools

Python packaging is something with the state of which nobody seems to be satisfied. As such, there are many new contenders for the role of “best new packaging tool”.

Two popular alternatives packaging tools are pip-tools (by Vincent Driessen and Jazzband) and Poetry (by Sébastien Eustace).

Pip is here to stay!

But in all the talk about new tools, we’re forgetting about the old ones, and they do their job well — so well in fact, that the new tools still need them under the covers.

Pip is fast. It does its job well enough. It lacks support for splitting packages between production and development (as Pipenv and Poetry do). This means that pip freeze and pip install are instant, at the cost of (a) needing two separate environments, or (b) installing development dependencies in production (which should only be a waste of HDD space and nothing more in a well-architected system). But at the same time, pip-tools can help keep the environments separate, as long as you take some time to write separate requirements.in files.

The virtualenv management features can be provided by virtualenvwrapper. That tool’s main advantage is the shell script implementation, which means that workon foo activates the foo virtualenv without spawning a new subshell (an issue with Pipenv and Poetry, that I already covered when describing Pipenv’s operation in the Running scripts (badly) chapter.) An argument often raised by Pipenv proponents is that one does not need to concern itself with creating the virtualenv, and doesn’t need to care where it is. Unfortuntately, many tools require this knowledge from their user, or force a specific location, or require it to be different to the home directory.

And for a reasonable project template with release automation — well, I have my own entry in that category, called (rather unoriginally) the Python Project Template (PyPT).

Yes, setup.py files are not ideal, since they use .py code and a function execution, making access to meta information hard (./setup.py egg_info creates tool-accessible text files). Their main advantage is that they are the only format that is widely supported — pip is the de-facto default Python package manager (which is pre-installed on Windows and Mac), and other tools would require installation/bootstrapping first.

The break-neck pace of Pipenv

A good packaging tool is stable. In other words, it doesn’t change often, and it strives to support existing environments. It wouldn’t be fun to re-download everything on your system, because someone decided that /usr is now called /stuff, and all the files in /usr would become forgotten and not removed. Well, this is what Pipenv did:

Over the course of a month, the location of the virtualenv changed twice. If the user didn’t read the changelog and didn’t manually intervene (also of note, the option name was mentioned in the issue and in v3.3.4’s changelog), they would have a stale .venv directory, since the new scheme was adopted for them. And then, after switching to v3.5.0, they would have a stale virtualenv hidden somewhere in their home directory, because pipenv decided to add hashes.

Also, this is not configurable. One cannot disable the hashes in paths, even though users wanted to. It would also help people who want to mix Pipenv and virtualenvwrapper.

Pipenv is a very opinionated tool, and if the dev team changes their mind, the old way is not supported.

Pipenv moves fast and doesn’t care if anything breaks. As an example, between 2018-03-13 13:21 and 2018-03-14 13:44 (a little over 24 hours), Pipenv had 10 releases, ranging from v11.6.2 to v11.7.3. The changelog is rather unhelpful when it comes to informing users what happened in each of the releases.

Extra reading:

• Kenneth Reitz, A Letter to /r/python (with some notes about bipolar disorder) (replaced with Wayback Machine link on 2020-02-07)

• Reddit comment threads for the letter: first and second

Conclusion

• Pipenv, contrary to popular belief and (now removed) propaganda, is not an officially recommended tool of Python.org. It merely has a tutorial written about it on packaging.python.org (page run by the PyPA).

• Pipenv solves one use case reasonably well, but fails at many others, because it forces a particular workflow on its users.

• Pipenv does not handle any parts of packaging (cannot produce sdists and wheels). Users who want to upload to PyPI need to manage a setup.py file manually, alongside and independently of Pipenv.

• Pipenv produces lockfiles, which are useful for reproducibility, at the cost of installation speed. The speed is a noticeable issue with the tool. pip freeze is good enough for this, even if there are no dependency classes (production vs development) and no hashes (which have minor benefits) [2]

• Poetry supports the same niche Pipenv does, while also adding the ability to create packages and improving over many gripes of Pipenv. A notable issue is the use of a custom all-encompassing file format, which makes switching tools more difficult (vendor lock-in).

• Pip, setup.py, and virtualenv — the traditional, tried-and-true tools — are still available, undergoing constant development. Using them can lead to a simpler, better experience. Also of note, tools like virtualenvwrapper can manage virtualenvs better than the aforementioned new Python tools, because it is based on shell scripts (which can modify the enivironment).

• Since 2018, the packaging scene deteriorated even more. See How to improve Python packaging, or why fourteen tools are at least twelve too many (from January 2023) and Python Packaging, One Year Later: A Look Back at 2023 in Python Packaging (from January 2024).

Other discussion threads: r/Python, Hacker News.

Use the right tool for the job

By calling another process, you introduce a third-party dependency. That dependency isn’t controlled by your code, and your code becomes more fragile. The problems include:

• the program is not installed, or even available, for the user’s OS of choice

• the program is not in the $PATH your process gets

• the hard-coded path is not correct on the end user’s system

• the program is in a different version (eg. GNU vs. BSD, updates/patches), which means different option names or other behaviors

• the program’s output is not what you expected due to user config (including locale)

• error reporting is based on numeric exit codes, and the meaning of those differs between programs (if they have meaning besides 0/1 in the first place)

On the other hand, if your code uses a lot of subprocesses, perhaps you should stay with Bash. You can do the harder parts with Python, Ruby, or some other language by calling them from within your Bash script.

Don’t spawn subprocesses if there’s an alternative

Spawning a subprocess always incurs a (minor) [1] performance hit minor compared to the alternatives. With that in mind, and the resiliency issues listed above, you should always try to find an alternative for the external command.

The simplest ones are the basic Unix utilities. Replace grep, sed and awk with string operations and regular expressions. Filesystem utilities will have equivalents — for Python, in os or shutil. Your language of choice can also handle things like networking (don’t call curl), file compression, working with date/time…

Similarly, you should check if there are packages available that already do what you want — library bindings or re-implementations. And if there isn’t, perhaps you could help the world by writing one of those and sharing it?

One more important thing: if the program uses the same language as your code, then you should try to import the code and run it from the same process instead of spawning a process, if this is feasible.

Security considerations: shells, spaces, and command injection

We come to the most important part of this article: how to spawn subprocesses without compromising your system. When you spawn a subprocess on a typical Unix system, fork() is called, and your process is copied. Many modern Unix systems have a copy-on-write implementation of that syscall, meaning that the operation does not result in copying all the memory of the host process over. Forking is (almost) immediately followed by calling execve() (or a helper function from the exec family) [2] in the child process — that function transforms the calling process into a new process [3]. This technique is called fork-exec and is the typical way to spawn a new process on Unix. [4]

There are two ways to access this API, from the C perspective:

• directly, by calling fork() and exec*() (or posix_spawn()), and providing an array of arguments passed to the process, or

• through the shell (sh), usually by calling system(). As Linux’s manpage for system(3) puts it,

  The system() library function uses fork(2) to create a child process that executes the shell command specified in command using execl(3) as follows:

  execl("/bin/sh", "sh", "-c", command, (char *) 0);
  

If you go through the shell, you pass one string argument, whereas exec*() demands you to specify arguments separately. Let’s write a sample program to print all the arguments it receives. I’ll do it in Python to get a more readable output.

#!/usr/bin/env python3
import sys
print(sys.argv)

Let’s see what appears:

$ ./argv.py foo bar
['./argv.py', 'foo', 'bar']
$ ./argv.py 'foo bar'
['./argv.py', 'foo bar']
$ ./argv.py foo\ bar baz
['./argv.py', 'foo bar', 'baz']

$ ./argv.py $(date)
['./argv.py', 'Sat', 'Sep', '2', '16:54:52', 'CEST', '2017']
$ ./argv.py "$(date)"
['./argv.py', 'Sat Sep  2 16:54:52 CEST 2017']

$ ./argv.py /usr/*
['./argv.py', '/usr/X11', '/usr/X11R6', '/usr/bin', '/usr/include', '/usr/lib', '/usr/libexec', '/usr/local', '/usr/sbin', '/usr/share', '/usr/standalone']
$ ./argv.py "/usr/*"
['./argv.py', '/usr/*']

$ ./argv.py $EDITOR
['./argv.py', 'nvim']

$ $PWD/argv.py foo bar
['/Users/kwpolska/Desktop/blog/subprocess/argv.py', 'foo', 'bar']
$ ./argv.py a{b,c}d
['./argv.py', 'abd', 'acd']

$ python argv.py foo bar | cat
['argv.py', 'foo', 'bar']
$ python argv.py foo bar > foo.txt
$ cat foo.txt
['argv.py', 'foo', 'bar']

$ ./argv.py foo; ls /usr
['./argv.py', 'foo']
X11@        X11R6@      bin/        include/    lib/        libexec/    local/      sbin/       share/      standalone/

As you can see, the following things are handled by the shell (the process is unaware of this occurring):

• quotes and escapes

• expanding expressions in braces

• expanding variables

• wildcards (glob, *)

• redirections and pipes (> >> |)

• command substitution (backticks or $(…))

• running multiple commands on the same line (; && || &)

The list is full of potential vulnerabilities. If end users are in control of the arguments passed, and you go through the shell, they can execute arbitrary commands or even get full shell access. Even in other cases, you’ll have to depend on the shell’s parsing, which introduces an unnecessary indirection.

TL;DR: How to do this properly in your language of choice

To ensure spawning subprocess is done securely, do not use the shell in between. If you need any of the operations I listed above as part of your command — wildcards, pipes, etc. — you will need to take care of them in your code; most languages have those features built-in.

In C (Unix)

Perform fork-exec by yourself, or use posix_spawn(). This also lets you communicate with the process if you open a pipe and make it stdout of the child process. Never use system().

In Python

Use the subprocess module. Always pass shell=False and give it a list of arguments. With asyncio, use asyncio.create_subprocess_exec (and not _shell), but note it takes *args and not a list. Never use os.system and os.popen.

In Ruby

Pass arrays to IO.popen. Pass multiple arguments to system() (system(["ls", "ls"]) or system("ls", "-l")). Never use %x{command} or backticks.

In Java

Pass arrays to Runtime.exec. Pass multiple arguments or list to ProcessBuilder.

In PHP

All the standard methods go through the shell. Try escapeshellcmd(), escapeshellarg() — or better, switch to Python. Or anything, really.

In Go

os/exec and os.StartProcess are safe.

In Node.js

Use child_process.execFile or child_process.spawn with shell set to false.

Elsewhere

You should be able to specify multiple strings (using variadic arguments, arrays, or otherwise standard data structures of your language of choice) as the command line. Otherwise, you might be running into something shell-related.

The part where I pretend I know something about Windows

On Windows, argument lists are always passed to processes as strings (Python joins them semi-intelligently if it gets a list). Redirections and variables work in shell mode, but globs (asterisks) are always left for the called process to handle.

Some useful functions are implemented as shell built-ins — in that case, you need to call it via the shell.

Internals: There is no fork() on Windows. Instead, CreateProcess(), ShellExecute(), or lower-level spawn*() functions are used. cmd.exe /c is called in shell calls.

Reverse-engineering Python bytecode: re-creating the function by hand

I started by recreating the original firmware file. I created an empty function and wrote some code to print out __code__ contents and dis.dis output. I also added color-coding to help me read it:

#!/usr/bin/env python2
import dis
import sys

# Write code here
def check_password(s):
    pass

# Reverse engineering the code
cnames = ('co_argcount', 'co_consts', 'co_flags', 'co_name', 'co_names', 'co_nlocals', 'co_stacksize', 'co_varnames')
cvalues = (1, (None, '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89', 'hex', 89, 255, 115, 50), 67, 'check_password', ('decode', 'len', 'False', 'all', 'zip', 'ord'), 4, 6, ('s', 'good', 'cs', 'cg'))

for n, ov in zip(cnames, cvalues):
    v = getattr(check_password.__code__, n)
    if v == ov:
        sys.stderr.write('\033[1;32m')
    else:
        sys.stderr.write('\033[1;31m')
    sys.stderr.flush()

    sys.stdout.write(str(n) + " " + str(v) + "\n")
    sys.stdout.flush()

    sys.stderr.write('\033[0m')
    sys.stderr.flush()

dis.dis(check_password)

If we run this solver, we get the following output (text in brackets added by me):

co_argcount 1            [OK]
co_consts (None,)        [1/7 match]
co_flags 67              [OK]
co_name check_password   [OK]
co_names ()              [0/6 match]
co_nlocals 1             [should be 4]
co_stacksize 1           [should be 6]
co_varnames ('s',)       [1/4 match]
  7           0 LOAD_CONST               0 (None)
              3 RETURN_VALUE

We can see (with the help of colors, not reproduced here), that we’ve got co_argcount, co_flags, co_name correctly. We also have one constant (None, in every function) and one variable name (s, the argument name). We can also see dis.dis() output. While it looks similar to the assignment, there are a few noticeable differences: there is no 7 (line number) at the start, and LOAD_CONST instructions in the original code did not have anything in parentheses (only comparisions and loops did). This makes reading bytecode harder, but still possible. (I originally thought about using diff for help, but it’s not hard to do it by hand. I did use diff for the final checking after a manual “conversion”)

Let’s stop to look at the constants and names for a second. The long string is followed by hex, and one of the constants is decode. This means that we need to use str.decode('hex') to create a (byte)string of some information. Puzzle answers tend to be human-readable, and this string isn’t — so we need to do some more work.

So, let’s try reproducing the start of the original mission code using what we’ve just discussed. Python’s VM is based on a stack. In the bytecode above, you can see that instructions take 0 or 1 arguments. Some of them put things on the stack, others do actions and remove them. Most instruction names are self-explanatory, but the full list can be found in the dis module documentation.

Instructions like LOAD and STORE refer to indices in the constants/names/varnames tuples. To make it easier, here’s a “table” of them:

constants
 0     1                                                       2      3   4    5    6
(None, '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89', 'hex', 89, 255, 115, 50)

names (globals, attributes)
 0         1      2        3      4      5
('decode', 'len', 'False', 'all', 'zip', 'ord')

varnames (locals, _fast)
 0    1       2     3
('s', 'good', 'cs', 'cg')

In order to improve readability, I will use “new” dis output with names in parentheses below:

 0 LOAD_CONST               1 ('4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89')
 3 LOAD_ATTR                0 (decode)
 6 LOAD_CONST               2 ('hex')
 9 CALL_FUNCTION            1 # function takes 1 argument from stack
12 STORE_FAST               1 (good)

As I guessed before, the first line of our function is as follows:

def check_password(s):
    good = '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89'.decode('hex')  # new

If we run the solver again, we’ll see that the first 12 bytes of our bytecode match the mission text. We can also see that varnames is filled in half, we’ve added two constants, and one name. The next few lines are as follows:

15 LOAD_GLOBAL              1
18 LOAD_FAST                0
21 CALL_FUNCTION            1
24 LOAD_GLOBAL              1
27 LOAD_FAST                1
30 CALL_FUNCTION            1
33 COMPARE_OP               3 (!=)
36 POP_JUMP_IF_FALSE       43
39 LOAD_GLOBAL              2
42 RETURN_VALUE

We can see that we’re putting a global object on stack and calling it with one argument. In both cases, the global has the index 1, that’s len. The two arguments are s and good. We put both lengths on stack, then compare them. If the comparison fails (they’re equal), we jump to the instruction starting at byte 43, otherwise we continue execution to load the second global (False) and return it. This wall of text translates to the following simple code:

def check_password(s):
    good = '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89'.decode('hex')
    if len(s) != len(good):  # new
        return False         # new

Let’s take another look at our names. We can see we’re missing all, zip, ord. You can already see a common pattern here: we will iterate over both strings at once (using zip), do some math based on the character’s codes (ord), and then check if all results (of a comparison, usually) are truthy.

Here’s the bytecode with value annotations and comments, which explain what happens where:

>>   43 LOAD_GLOBAL              3 (all)
     46 BUILD_LIST               0
     49 LOAD_GLOBAL              4 (zip)
     52 LOAD_FAST                0 (s)
     55 LOAD_FAST                1 (good)
     58 CALL_FUNCTION            2           # zip(s, good)
     61 GET_ITER                             # Start iterating: iter()
>>   62 FOR_ITER                52 (to 117)  # for loop iteration start (if iterator exhausted, jump +52 bytes to position 117)
     65 UNPACK_SEQUENCE          2           # unpack a sequence (a, b = sequence)
     68 STORE_FAST               2 (cs)      # cs = item from s
     71 STORE_FAST               3 (cg)      # cg = item from good
     74 LOAD_GLOBAL              5 (ord)
     77 LOAD_FAST                2 (cs)
     80 CALL_FUNCTION            1           # put ord(cs) on stack
     83 LOAD_CONST               3 (89)
     86 BINARY_SUBTRACT                      # - 89   [subtract 89 from topmost value]
     87 LOAD_CONST               4 (255)
     90 BINARY_AND                           # & 255  [bitwise AND with topmost value]
     91 LOAD_CONST               5 (115)
     94 BINARY_XOR                           # ^ 115  [bitwise XOR with topmost value]
     95 LOAD_CONST               6 (50)
     98 BINARY_XOR                           # ^ 50   [bitwise XOR with topmost value]
     99 LOAD_GLOBAL              5 (ord)
    102 LOAD_FAST                3 (cg)
    105 CALL_FUNCTION            1           # put ord(cs) on stack
    108 COMPARE_OP               2 (==)      # compare the two values on stack
    111 LIST_APPEND              2           # append topmost value to the list in topmost-1; pop topmost (append to list created in comprehension)
    114 JUMP_ABSOLUTE           62           # jump back to start of loop
>>  117 CALL_FUNCTION            1           # after loop: call all([list comprehension result])
    120 RETURN_VALUE                         # return value returned by all()

We can now write the full answer.

listings/gynvaels-mission-11-en/mission11.py (Source)

In the end, our dis.dis() output matches the mission text (except the removed values, but their IDs do match), our co_* variables are all green, and we can get to work on solving the puzzle itself!

Side note: this task uses a list comprehension. You might want to optimize it, remove the brackets, and end up with a generator expression. This would make the task harder, since would require working with the internal generator code object as well:

co_consts (None, '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89', 'hex', <code object <genexpr> at 0x104a86c30, file "mission11-genexpr.py", line 11>)

46 LOAD_CONST               3 (<code object <genexpr> at 0x104a86c30, file "mission11-genexpr.py", line 11>)

BINARY_* and ord disappeared from the new listing. You can see the modified code (which differs by two bytes) and solver output.

Solving the real puzzle

I solved the extra credit part of the puzzle. The real aim of the puzzle was to recover the password — the text for which check_password() will return True.

This part is pretty boring. I built a dictionary, where I mapped every byte (0…255) to the result of the calculation done in the check_password() function’s loop. Then I used that to recover the original text.

pass_values = {}
for i in range(256):
    result = i - 89 & 255 ^ 115 ^ 50
    pass_values[result] = i

good = '4e5d4e92865a4e495a86494b5a5d49525261865f5758534d4a89'.decode('hex')
password = ''
for c in good:
    password += chr(pass_values[ord(c)])

print(password)
print(check_password(password))

The password is: huh, that actually worked!.

What was that Paint thing about?

Yesterday’s mission was about Elvish — I mean Paint — I mean Python programming.

Most of my readers were probably puzzled by the mention of Paint. Long-time viewers of Gynvael’s streams in Polish remember the Python 101 video he posted on April Fools last year. See original video, explanation, code (video and explanation are both Polish; you can get the gist of the video without hearing the audio commentary though.) Spoilers ahead.

In that prank, Gynvael taught Python basics. The first part concerned itself with writing bytecode by hand. The second part (starts around 12:00) was about drawing custom Python modules. In Paint. Yes, Paint, the simple graphics program included with Microsoft Windows. He drew a custom Python module in Paint, and saved it using the BMP format. It looked like this (zoomed PNG below; download gynmod.bmp):

How was this done? There are three things that come into play:

• Python can import modules from a ZIP file (if it’s appended to sys.path). Some tools that produce .exe files of Python code use this technique; the old .egg file format also used ZIPs this way.

• BMP files have their header at the start of a file.

• ZIP files have their header at the end of a file.

• Thus, one file can be a valid BMP and ZIP at the same time.

I took the code of check_password and put it in mission11.py (which I already cited above). Then I compiled to .pyc and created a .zip out of it.

listings/gynvaels-mission-11-en/mission11.py (Source)

Since I’m not an expert in any of the formats, I booted my Windows virtual machine and blindly copied the parameters used by Gynvael to open the ZIP file (renamed .raw) in IrfanView and saved as .bmp. I changed the size to 83×2, because my ZIP file was 498 bytes long (3 BPP * 83 px * 2 px = 498 bytes) — by doing that, and through sheer luck with the size, I could avoid adding comments and editing the ZIP archive. I ended up with this (PNG again; download mission11.bmp):

The .bmp file is runnable! We can use this code:

listings/gynvaels-mission-11-en/ziprunner.py (Source)

And we get this:

Resources

• mission11-solver.py (full solver code)

• mission11-genexpr.py, mission11-genexpr.txt (used for side note regarding generator expressions vs. list comprehensions)

• mission11.py code, used in BMP file

• ziprunner.py, file that runs the BMP/ZIP module (adapted from Gynvael’s)

• gynmod.bmp

• mission11.bmp

• dis module documentation.

Thanks for the mission (and BMP idea), Gynvael!