Lincoln Loop Blog

Python Package Manager Shootout

Fri, 22 Jul 2022 11:55:11 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

tldr; I built a benchmark for Python Package managers, you can view it at https://lincolnloop.github.io/python-package-manager-shootout/

When starting a new Python project, you have a few different options for how you want to manage your dependencies. Like Node.js has npm, yarn, pnpm, and bun, Python has pdm, pip-tools, pipenv, and poetry (and others).

They largely do the same things and fans of each one will usually tout why their choice is better. It's easy to compare tools based on features, but there isn't any good way to see how they compare on performance. Slow tooling is a regular source of frustration for developers. How much time are you going to be waiting on packages to install either locally or in CI over the life of the project?

With this in mind, I set out to build an impartial benchmark for these projects.

A Real World Example

The first issue to tackle was finding a sufficiently complex real world set of packages to work against. The pain of slow tooling doesn't really crop up until your dependency list is sufficiently large and resolving the dependency chain is non-trivial. I found a great sample from the folks at Sentry. Their product is largely open source and sufficiently complex that their requirements.txt made a great corpus to run the benchmark on. It is also interesting in that it includes at least one dependency that requires compilation instead of downloading a bunch of binary/pure-Python packages.

Operations to Benchmark

All these tools work slightly differently, but the general concepts are more-or-less identical. I decided to measure the following operations:

Tooling installation: How long does it take to install the tool itself?
Importing requirements: How long does it take to add all the depencencies to the project?
Locking: How long does it take to generate a lock file (resolve dependency chain, pin all the dependencies, and fetch published hashes for the packages)?
Install: How long does it take (with both a cold and warm cache) to install all the dependencies?
Update: How long does it take to update all the dependencies?
Add Package: How long does it take to add a new package to the list of dependencies (including install & re-lock)?

Lies, Damned Lies, and Benchmarks

Whenever you publish a benchmark, people will come out of the woodwork to tell you all the ways it is incorrect. I knew I needed to build something that was repeatable and auditable. I hoped package manager maintainers would contribute to the repo to ensure their projects are represented fairly (which they already have!). I also want people to be able to easily add new tools to the benchmark. With that in mind, here's where I ended up...

GitHub Actions

Publishing the code used to perform the benchmark was a given, but I wanted to take it a step further. I'm lazy and don't want to have to re-run the benchmark every time somebody makes a change or releases a new version of a package manager. I also don't want to update the site with the results manually. So I automated the entire thing.

GitHub Actions provides everything I needed for the automation. It can re-run the benchmarks on a schedule, then parse the results and update the site (published on GitHub Pages). Since these run on shared hardware and depend on external network resources, there is variability in the results. To combat that, we publish the average results of a few runs.

Structuring the Benchmarks

I wanted the GitHub workflow file to be machine generated to make it easy to add new tools without an error-prone copy/paste/replace process. I needed some way to abstract the unique commands of each tool into a shared set of commands that could easily be templated. The tool everyone loves to hate, make, fit the bill perfectly. Our Makefile provides a lightweight abstraction layer over all the differences. With that in place, I could use a generic template and assemble the workflow file using a simple bash script with envsubst.

Timing

The time command (not the built-in shell function) has everything needed to measure the performance of the individual commands. Wrapping them in make may create the tiniest bit of overhead, but I suspect it is so small it gets rounded off. time lets you output to a file in whatever format you want. I made it output as a CSV and collect all the operations into a single file for each package manager.

Aggregation

Each tools benchmark runs as a separate job in parallel on GitHub Action. When each job completes, it uploads its result CSV as an artifact. When all the jobs complete, we run a final job which gathers and combines the CSVs into a single file (thanks sqlite-utils!) and stores it as a new artifact.

I leveraged the newer job summary functionality along with mdtable to show a pretty representation of the CSVs in the GitHub Actions web interface which has been invaluable for debugging.

Building the Site

Once the benchmarks finish, a new workflow is kicked off to rebuild the site with the new data. This involves:

Downloading the aggregated data
Transforming the data in Python into the format expected by Chart.js
Building the static site with parcel
Generating an open graph image with shot-scraper so we get a pretty (and accurate) graph preview when the link is shared.
Pushing the files to gh-pages and letting it get deployed via GitHub Pages

Wrapping Up

So far this all seems to plug together quite well. Is it overengineered? Probably a bit 😄 That being said, I'm happy with the results. It's fully auditable and I don't have to answer questions about my laptop and internet connection and the position of the moon when the tests were run. It doesn't require maintaining any infrastructure and uses a lot of basic Linux constructs which will still work years from now. The external dependencies are minimal, so I don't anticipate it requiring much maintenance.

Interpreting the Results

All these package managers are the result of thousands of hours volunteered by folks in the Python community. Please don't use these results to shame or disparage the projects or their maintainers... there's too much toxicity out there already. My intent with publishing these is that people who are evaluating package managers have access to unbiased performance info. I also hope maintainers find it useful to identify places where these tools have room for improvement (it has already caught one major regression!). A faster Python ecosystem is better for everyone.

If you have suggestions on how to improve this benchmark, please don't hesitate to create an Issue or submit a pull request.

Choosing a Django Version

Mon, 13 Jun 2022 13:57:15 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

One of the first things you need to do when starting a new Django project is to choose which version of Django you are going to use. At any given time, there could be as many as three supported Django versions available to choose from:

previous long-term supported version (LTS)
current LTS
latest official version

I'm excluding pre-releases from this list which should only be used for testing.

What are you optimizing for?

The first question to ask yourself is what are you optimizing for. When we are building applications for our clients we are almost always optimizing to reduce costs, both during the initial development process and in support post-launch. For this reason, we are choosing versions which meet the goals of maximimizing interoperability with the Django ecosystem and minimizing maintenance (aka toil) down-the-road. Unless you are working on a hobby project and want to play with new features, I'd argue these goals are universal.

Notes on Django Versioning

Django does not follow the semantic versioning standard strictly. Versions will always be made up of three numbers (major.minor.patch). Minor version 2 is always an LTS version. After the LTS, the major version is incremented and two minor releases are made on it (x.0 and x.1). Patch versions are released for bugs found on any supported versions. For more info, see DEP-0004.

Check Supported Versions to see the current LTS cycle. As of June, 2022 it looks like this,

Minimizing Maintenance

The best way to minimize future maintenance is to choose an LTS release. The promise of not having to do a major version upgrade for up to three years means fewer upgrade cycles and less maintenance work.

Maximizing Interoperability

It's rare to build a Django application that doesn't depend on other open source Django libraries. Coming across a library that's the perfect fit for your project only to find it's not compatible with the version of Django will slow your progress. Being a good open source community member and submitting a patch is the right thing to do, but it takes time and effort that could otherwise be spent on your application.

Open source projects are usually maintained by volunteers who are working in their free time without compensation. As a result, support can lag behind official Django releases. If you are starting development shortly after a new LTS release, it may make more sense to use the previous LTS which should have wide adoption and still have close to 9 months of support remaining. If you take this route, read the new LTS release notes carefully to future-proof your project for the jump to the next LTS. In all cases, make sure your code runs without any deprecation warnings.

Another reason to avoid jumping on a brand new LTS is to avoid any bugs that weren’t caught during testing. I generally prefer to wait for the first patch release to adopt a new LTS version (4.2.1 instead of 4.2.0). These usually show up about a month after the initial release and contain bug fixes found in the wild.

Exceptions

As with all coding rules-of-thumb, there are exceptions, but avoid getting sucked into the “it’s new-and-shiny” trap. If a newer version of Django has a killer feature that is core to your application, that might be a good reason to jump off an LTS temporarily. An example of this might be the introduction of native JSON support for Postgres in Django 1.9.

If you’re using minimal dependencies or know all your dependencies support a new LTS version, that might be a good reason to start using an LTS that was recently released. It’s easy to paint yourself into a corner with this approach however. As your project comes to life, you may find you actually do need a library you didn’t expect when you started.

Finally, if you're developing as a hobby or to learn something new, use whatever version you want! Not every development project needs to be dictated by cost or what's best for the business.

Tldr; Use the latest LTS which has received at least one patch release. The version number will be in the format x.2.z where z > 0. This usually isn't the latest version you can download from PyPI.

High Performance Django Is Free Online

Thu, 30 Sep 2021 16:55:31 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

In 2014 Yann Malet and I (with the help of the rest of the team here) wrote a book about building and scaling Django websites. It was the culmination of things we'd learned from, at the time, close to a decade of experience building, deploying, and supporting Django sites.

It's been seven years since its release and it's fair to say that the book has run its course. While I didn't feel good about continuing to sell the book given its age, I also believe it still contains a lot of valuable information. As you'd expect, after seven years, many of the technical examples are outdated. The book, however, is largely conceptual. Code snippets are rarely the "secret sauce" when scaling a site. Topics such as caching and reducing/optimizing database queries aren't usually coding challenges. In fact, thanks to the stability of Django, much of that code is still the same today.

So instead of relegating the book to the dustbin, we're releasing it for free online. It's something we've wanted to do for a long time and I'm excited to finally get the book in more people's hands.

You can find it on this site at https://lincolnloop.com/high-performance-django/.

Documenting Django Design Systems with Storybook

Fri, 05 Feb 2021 12:19:23 -0600

Please update to our new feed url: lincolnloop.com/blog/feed/

Our clients come to us with a variety of existing internal teams. Sometimes we work with UI designers or brand-focused graphic designers who deliver pixel-perfect mockups. Other times the stakeholders will describe what their goals are, and we’ll be responsible for more of the design process. In either case, it’s our responsibility to document our work well. This is important both during the course of the project and after our engagement with the client ends.

Component Libraries or Design Systems provide a lean way to deliver documentation alongside project deliverables at every stage of a project. Often, the frontend tooling that we set up for these systems allows us to get client approval on design or UI deliverables in a much more accurate and reusable way than a high-fidelity Figma or Sketch file would. Once approved, the HTML/CSS is already done and ready for integration within the application code.

In the past, we have kept this tooling simple, using a static site generator compiled from SASS comments. It provided some minor features to build well-designed documentation, and allowed us to build complicated HTML mockups quickly in a component-driven way.

Why switch to Storybook?

Depending on the project or the day, I switch between UI/UX tasks and frontend development. That means when I’m doing one task, I’m thinking about the other. For example, when I’m working on frontend, I’ll often form opinions on how best to structure the UI of the CMS for a given component. This is where our static-file-handoff process could use some improvement.

Let's take a standard “hero component” as an example.

If I’m handing off static files to the backend team, I’ll also include a list of modifier classes for various color choices (light or dark text, and an overlay color option for the photo). Then, I’d provide a different list of modifiers to change the position of the text on the photo. If I have opinions on the CMS UI, I’d provide those notes as well. (Toggle the knockout-text class, add this helper text, and use a dropdown for text alignment.)

The Django developers will implement the work. This works well enough for us internally. For client approval, the backend team will push the component to a QA server, stand up an example page, and then we’ll start the feedback loop on the component frontend and the CMS UI at the same time. By necessity, that means the frontend team and backend team both need to be on standby for changes.

This is where Storybook allows us to improve both our internal workflow and our client feedback process. I can work CMS-like UI controls directly into our documentation. The storybook file contains some minor configurations:

import Masthead from  '../app/styleguide-examples/masthead.njk';

export default {
  title: 'Components/Hero',
  argTypes: {
    knockout: { control: 'boolean'},
    modifier_class: {
      control: {
        type: 'select',
        options: [
          '',
          'top-right',
          'Top-left',
          'bottom-right',
          'bottom-left'
        ]
      }
    }  
  }
};

Which compiles to a nice UI in Storybook that live-updates the component example:

Once the frontend and documentation are built, I can push a copy to a static server. The client can give feedback on the component and tinker with some form of the CMS UI before involving the backend team. This improves the fidelity of the initial feedback and allows me to provide instructions to the backend team with more confidence.

Then at the end of the project, the client will own this interactive documentation. It acts as a catalog of the components of their site, complete with inline instructions on best-practices and use-cases for each component. When we work in collaboration with the client on these systems, they help us shape them into a tool that can be used to onboard new content creators, designers, or developers. Sections can be added for editorial style guides, social media assets, or branding guidelines for print, depending on the client needs.

Technical considerations

Storybook is a React app at its core, but it is compatible with any frontend. If a project will live in React or Vue, then the app components can be used with no duplication. The “story” files will add a bit of context and mock data to your existing components. For the sake of this “basic Django” experiment, I used the Storybook HTML starter and piped in a nunjucks loader. Nunjucks templates feel very similar to Django or Jinja templates, which makes handoff to the backend team easy. However there are enough minor differences that I can’t recommend trying to use them together. Here are some more details on the problems that you might encounter.

The Verdict

Every tool has its strengths and tradeoffs. Storybook is certainly more complicated to set up and maintain than a static site, but its benefits and baked-in interactivity add more than enough value to make it an essential part of new projects.

Distributed Locking in Django

Tue, 11 Aug 2020 14:44:19 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

As you start scaling an application out horizontally (adding more servers/instances), you may run into a problem that requires distributed locking. That's a fancy term, but the concept is simple. Sometimes you have to be sure that when a block of code is running (usually modifying data somewhere), no other instances runs that same block of code. Ideally, you can design your code not to require locks, but sometimes it is inevitable.

In general, locks are used when you need to modify state (e.g. the database) in an atomic manner. Some examples of when you might need a distributed lock:

Cron jobs/scheduled tasks whose runtime may exceed the interval with which they are triggered
Flushing a write-back cache to the database.
Bulk processing files

If running the code on multiple servers simultaneously would result in corrupted or duplicate data, you probably need to use a distributed lock. The code in question will acquire the lock before execution. Once it has the lock, any other attempt to acquire it will fail.

Implementations

Lock data must be stored in a location accessible to all application instances. If this is a standard Django site, you probably already have two such systems available to you, the cache and the database. The tricky part about locking is that it must be atomic to avoid a race condition when two processes try to acquire the lock simultaneously. Thankfully, this is already a solved problem in the standard cache and database backends used by Django. For these backends, you can find third-party libraries which expose the functionality in Python.

Redis

SETNX is the Redis primitive that is used for locking. The django-redis package provides a context manager for this functionality:

from django.core.cache import cache

with cache.lock("somekey"):
    do_some_thing()

Memcached

Memcached's ADD works similar to Redis' SETNX. I don't have experience with any libraries that implement a context manager around this, but both django-cache-lock and sherlock appear to provide it.

Postgres

Postgres has a pg_advisory_lock function which is utilized by django-pglocks.

from django_pglocks import advisory_lock

with advisory_lock("somekey"):
    do_some_thing()

MySQL

MySQL provides a GET_LOCK function for distributed locks. It is exposed via a context manager in the django-mysql library.

from django_mysql.locks import Lock

with Lock("somekey"):
    do_some_thing()

Additional Considerations

Timeouts

While context managers should release the lock when they exit, there's always the possibility that your application crashes before that can happen. Without a timeout on the lock, it will be held forever and prevent any similar code from running. Refer to the docs of the library you choose to see how to specify a timeout. You should set this value to something longer than it should ever take the code to execute, but short enough, it doesn't prevent successive runs from executing.

Encountering a Lock

What your application does when it encounters a lock that has already been acquired is going to be specific to its requirements. Most implementations will throw an exception if the lock can't be acquired, so you'll usually wrap this code in a try/except. Possibilities of what to do in the exception case include:

Retry later
Wait for the lock to be released
Log an error
Do nothing

Database Row Locking

If your code needs exclusive access to modify specific rows in the database and you want to prevent any other modification of those rows while the code executes, database row locking may be a better option. Django provides functionality for this with the select_for_update queryset method.

Distributed locking sounds like a difficult technical issue, but in general, it is a solved problem. Any Django app should be able to grab a mature off-the-shelf solution to the problem. The only problems to solve are identifying the code that requires a lock and what should happen if a lock is encountered.

Goodconf: A Python Configuration Library

Wed, 29 Jul 2020 11:02:49 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

I've been working quite a bit lately on streamlining Lincoln Loop's standard deployment systems. One thorn we've always had is how to handle application configuration.

In the past, we would have our configuration management system write the configuration out to a JSON file at a known location on the filesystem. The application would read the JSON and set the necessary variables accordingly. This accomplished a few goals:

Deployments didn't require any sort of modification to the code from the upstream repository.
Production secrets could be encrypted and stored safely away from the code.
Unlike environment variables, the data could have proper types (booleans, lists, etc.)
It avoided using environment variables altogether, which can be problematic from a security perspective in many scenarios.

That being said, the setup had some downfalls as well.

The configuration variables needed were not well documented. You had to read through the code to understand how and where they were used.
We were maintaining a separate Django settings module for local and deployed environments since the file wasn't used locally.
The configuration file was not friendly for other services like Heroku or Docker where environment variables are typically used.

Basically, I want to use our Python projects like I would expect to use any other software. Install the software, create a configuration file, and run.

Introducing Goodconf

I wrote Goodconf (with lots of help from Chris Beaven) to solve some of our issues in a reusable library.

Goodconf is inspired by derpconf and logan which are spun out of Thumbor and Sentry respectively.

With Goodconf, you create a class which defines all the configuration values your application expects to receive. They can have default values and help text which can be used to generate documentation.

from goodconf import GoodConf, Value

class MyConf(GoodConf):
    "Configuration for My App"
    DEBUG = Value(default=False, help="Toggle debugging.")
    DATABASE_URL = Value(
        default='postgres://localhost:5432/mydb',
        help="Database connection.")
    SECRET_KEY = Value(
        initial=lambda: base64.b64encode(os.urandom(60)).decode(),
        help="Used for cryptographic signing. "
        "https://docs.djangoproject.com/en/2.0/ref/settings/#secret-key")

You can then instantiate the class like so:

config = MyConf(
    file_env_var="MYAPP_CONF",
    default_files=["/etc/myapp/myapp.yml", "myapp.yml"])

This lets you define the default location for configuration files (/etc/myapp/myapp.yml or $(pwd)/myapp.yml if that doesn't exist) and also an environment variable that can be used to load a file from a different location.

There is also a Django helper which lets you do manage.py --config=/path/to/config.yml ....

With the configuration defined, you simply need to load it and start using it. Loading the config will try to read from the configuration files and fallback on environment variables (cast to the proper Python type) if none are found.

config.load()

SECRET_KEY = config.SECRET_KEY
...

My favorite feature of Goodconf is the output you can generate from a config class:

config.generate_yaml() boilerplate YAML config with help text as comments
config.generate_json() boilerplate JSON config
config.generate_markdown() Documentation in Markdown format

This lets you quickly generate a config file for local development and documentation to drop into a README.md.

Using Goodconf

Goodconf's README has some examples of how to use the library, but if you're like me, it's easier to see an example. Check out our saltdash repo for an example of using Goodconf in a Django project.

saltdash/__init__.py defines the configuration
saltdash/settings/__init__.py shows how to use it to define Django settings
setup.cfg (under [options.entry_points]) shows how to install scripts for the management command and configuration generator

Try it Out

We're using Goodconf in production now and happy with the results. We hope you'll try it out on your own projects. If you do, please give us feedback here or in GitHub.

Python Dependency Locking with pip-tools

Wed, 22 Jul 2020 13:06:57 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

Two of the biggest benefits pipenv and poetry are dependency locking and hash checking. Dependency locking means you can specify the direct dependencies your code requires, for example, celery==4.4.* and the tooling will lock, not only celery to a specific version, but also every dependency pulled in by celery. Hash checking ensures the package you download matches a hash of the package when it was first downloaded. This ensures that when celery (or any of its dependencies) are installed on a different system, they match the same file hash as when they were set up by the developer. If the package is in some way tampered with between the two installs, hash checking will fail, preventing the new (and potentially malicious) package from being installed.

The combination of these two features makes your project installation deterministic: given the same requirements/lock file, you'll always get the same set of packages installed. This prevents all manner of issues when dependencies can shift underneath the code between developers' machines, CI, and deployment. It's a huge improvement over how we previously managed dependencies.

Pipenv and poetry have many other features, but I find myself rarely using them. Managing Python virtual environments is nice, but for experienced Python developers, skipping python -m venv .venv before installing a project is not game-changing. I have limited experience with poetry, but the performance of pipenv and edge-case bugs have been a regular sore point in the projects we use it on.

Locking Dependencies with pip-compile

If you're willing to give up the bells-and-whistles, pip-compile (provided by pip-tools) is a viable alternative. It provides both locking and hash-pinning for your dependencies. Once installed, you'll typically create a requirements.in file. This is where you define your project's top-level dependencies (similar to pipenv's Pipfile or pyproject.toml in poetry). A basic example might look something like this:

Django==2.2.*
psycopg2
celery>4.4

To "lock" these dependencies, you can run:

pip-compile --generate-hashes --output-file=requirements.txt requirements.in

This generates the standard requirements.txt file with all dependencies and includes package hashes provided by PyPI. Here's a line from that file:

pytz==2019.3 \
    --hash=sha256:1c557d7d0e871de1f5ccd5833f60fb2550652da6be2693c1e02300743d21500d \
    --hash=sha256:b02c06db6cf09c12dd25137e563b31700d3b80fcc4ad23abb7a315f2789819be \
    # via django

Note we didn't have pytz in our requirements.in, but it's included in requirements.txt because it is required by django (which the pip-compile is kind enough to output in the file).

If your project uses a Makefile, this workflow is well suited for it. The following will allow you to run make requirements.txt and it will be updated if and only if the requirements.in file has changed since requirements.txt was last generated:

requirements.txt: requirements.in
    pip-compile --upgrade --generate-hashes --output-file=$@ requirements.in

pip-compile also provides the option for selectively upgrading individual packages with the --upgrade-package argument as noted in their README.md

Installing Dependencies

Installing the dependencies is as simple as:

pip install -r requirements.txt

You can also use pip-sync (included in pip-tools) to reconcile an existing virtual environment with whatever exists in requirements.txt. Not only will this install/upgrade packages as needed, but also remove packages that are no longer in requirements.txt.

pipenv and poetry get a lot of attention in the Python community, and rightfully so. They are great projects that certainly scratch an itch for developers. That being said, if you're just looking for something to handle dependency management, pip-tools may be all you need.

User-Generated Themes with Django and CSS Variables

Wed, 24 Jun 2020 10:37:16 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

Consider a “white labeled” app where a CMS admin can customize the design of their public-facing dashboard. The developer is tasked with saving a handful of user-populated values to the database. Colors, fonts, and perhaps a background image or two now need to override dozens of CSS properties throughout the system.

Django Compressor has historically been great for this type of task. The workflow would look something like this:

Build a SASS file/Django template hybrid that looks like this:

$brand-color: {{ user.brand_color }};
$link-color: {{ user.link_color }};
$font-family: {{ user.font_family }};

Pipe that file into Django Compressor and build a fresh full copy of the app CSS using these variables instead of the defaults.
Inject a link to the new CSS file, if it exists, underneath the base app file to let the CSS cascade override everything properly.

This is sub-optimal in 2020 for several reasons. It creates unnecessary duplicative code. It adds a potential point-of-failure on the server. Using modern CSS, there are ways to make the browser do all of the heavy lifting.

Enter CSS variables

CSS variables are a perfect fit for this task if user themes don’t need to support Internet Explorer. (If IE is mandatory, css-vars-ponyfill may help you get the job done.) With CSS variables, it’s possible to override variables using the cascade, so we can avoid the preprocessor entirely. It’s also possible to make this change in a fairly surgical way, without completely refactoring your stylesheets away from SASS or LESS. Note that this is not how I would start a new project. Instead, I would begin with CSS variables as a first-class citizen. This is simply a safe way to make this change to an older codebase.

Update the base SCSS file

First, we’ll update the base SCSS file to use CSS variables, with a fallback of the original values. The original _variables.scss file looks like this:

$heading-color: #222 !default;
$link-color: #33C !default;
$font-family: Lato, sans-serif !default;

To keep those default values and have them overridden later with CSS variables, we need to inject the CSS variable syntax:

$heading-color: var(--heading-color, #222);
$link-color: var(--link-color, #33C);
$font-family: var(--font-family, "Lato, sans-serif");

If no CSS variables are defined, the original values will be used. With these changes made, it’s time to cross your fingers and hope the CSS compiles.

It might not.

If the SCSS source uses color functions throughout the project, there will be errors. For example, SASS won’t know what to do with mix($black, $heading-color, 20%) since $heading-color no longer resolves to computable color values. These will require manual refactoring. Solutions vary greatly depending on how colors are structured in the SCSS. This could be solved with a clever use of RGBA gradient overlays, going all-in with a CSS-variable approach to colors, or making more variables in _variables.scss to consolidate this work and put all of the color math in one place. Unfortunately, SASS color functions are still much more elegant than trying to do this work in CSS, so every solution has tradeoffs.

Update the Django template

Now that the SASS is ready, the base CSS file can be compiled with modern frontend-only tools, and CSS variables would only be used for the theme overrides. The Django template could be a template file that compiles to a CSS file (which would be best for page-caching reasons), or an inlined <style> tag in the dashboard HTML template:

{% block append_head %}
<style>
:root {
    {% if user.heading_color %} 
    --heading-color: {{ user.heading_color }}; 
    {% endif %}
    {% if user.link_color %} 
    --link-color: {{ user.link_color }}; 
    {% endif %}
    {% if user.font_family %} 
    --font-family: {{ user.font_family }};
    {% endif %}
}
</style>
{% endblock append_head %}

Wrap it up, or keep going

This is a small example to begin working with CSS custom properties. It’s a powerful technology with great browser support, and it’s been a game-changer here at Lincoln Loop. So if it’s not a part of your regular tool belt yet, it’s time to give it a spin.

Dissecting a Python Zipapp Built with Shiv

Mon, 06 Jul 2020 14:34:04 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

In a previous post, we showed how to use shiv to bundle a Django project into a single file for distribution and deployment. Running a large Python project as a single file feels like magic -- which is great until you need to debug a problem. At that point, you need to understand how things work and what is happening under the hood. With that in mind, let's demystify shiv's magic.

Zipapps

Shiv uses a little known feature of Python called a ZIP application or "zipapp". Zipapps provide a way to bundle multiple Python files into a single ZIP archive which the Python interpreter can then execute. Here's a trivial example:

$ mkdir myapp
$ echo 'print("hello world")' > myapp/__main__.py
$ python -m zipapp myapp
$ python myapp.pyz
hello world

Pretty cool, huh? Copy myapp.pyz anywhere you have a compatible version of Python and it just works.

But real-world projects aren't so simple. They have dependencies, non-Python files that need to be included, and complex build steps.

Enter Shiv

The folks at LinkedIn built shiv to work around the shortcomings of traditional zipapps. It includes dependencies in the zipapp and ensures they are on the PYTHONPATH at runtime. Shiv is only needed for creation. The resulting zipapp is executable without any additional tooling. You can think shiv as a wrapper around pip which it uses behind the scenes to download and install dependencies. Here is an example of creating a zipapp for awscli:

$ shiv --output-file=aws.pyz --entry-point=awscli.clidriver.main awscli

This downloads awscli and all its dependencies from PyPI and creates a zipapp (aws.pyz) which will run a Python function specified by --entry-point on execution. That file can be copied to any system with a compatible Python version and executed as if it were a binary executable (./aws.pyz or python aws.pyz).

Under the Hood

The good news is that shiv is surprisingly simple. When we build a new archive, shiv does the following:

Use pip to download all the dependencies to a temporary directory
Write some metadata used during the bootstrap process to environment.json
Create a bootstrap script that is used when the zipapp is executed.
Bundle those files up into zip archive
Insert a shebang at the top of the zip archive so it can be used as an executable.

The bootstrap script's responsibility is to:

Unpack the dependencies to a unique path. This is skipped if it already exists from a previous run.
Insert that path into the PYTHONPATH.
Execute the --entry-point we defined during creation.

The unique path is determined by combining the filename of the zipapp and a UUID generated at build time (stored in environment.json). It is stored in ~/.shiv by default but can be changed by setting the SHIV_ROOT environment variable. After execution, we can see the following:

cd ~/.shiv && find . -maxdepth 2
.
./aws_3e32a16c-6652-44cc-a561-3784814d736e
./aws_3e32a16c-6652-44cc-a561-3784814d736e/site-packages

The site-packages directory looks just like the site-packages directory you'd find for a standard Python installation or a virtualenv. As we can see, this was assigned a UUID of 3e32a16c-6652-44cc-a561-3784814d736e at build time which can be confirmed by inspecting the included metadata:

$ unzip -p aws.pyz environment.json | jq .build_id
"3e32a16c-6652-44cc-a561-3784814d736e"

This directory is added to the PYTHONPATH like so:

$ echo "import sys, pprint; pprint.pprint(sys.path)" | \
  SHIV_INTERPRETER=1 ./aws.pyz
Python 3.7.3 (default, Jun 11 2019, 01:05:09)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> ['./aws.pyz',
 '/usr/local/lib/python37.zip',
 '/usr/local/lib/python3.7',
 '/usr/local/lib/python3.7/lib-dynload',
 '/home/user/.shiv/aws_3e32a16c-6652-44cc-a561-3784814d736e/site-packages',
 '/usr/local/lib/python3.7/site-packages']

Note: The environment variable SHIV_INTERPRETER allows us to drop down into a Python shell using the zipapp's environment.

The fifth item, /home/user/.shiv/... is the one that was injected in via the bootstrap script. The others are Python defaults.

If you prefer to run in an isolated environment without the global site packages, Python's -S flag can be used:

$ echo "import sys, pprint; pprint.pprint(sys.path)" | \
  SHIV_INTERPRETER=1 python -S aws.pyz
Python 3.7.3 (default, Jun 11 2019, 01:05:09)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> ['aws.pyz',
 '/usr/local/lib/python37.zip',
 '/usr/local/lib/python3.7',
 '/usr/local/lib/python3.7/lib-dynload',
 '/home/user/.shiv/aws_3e32a16c-6652-44cc-a561-3784814d736e/site-packages']

Once unpacked, you can inspect the files on disk and even edit them if you're trying to do some tricky debugging. Shiv will not overwrite the files of a previously unpacked zipapp unless the environment variable SHIV_FORCE_EXTRACT is set.

That's It

Turns out shiv is pretty straightforward. One of the things I like most about it is its simplicity. When there's a problem with my code, it's easy to poke around and rule out shiv as a cause. The code is well commented and easy to follow (here's the bootstrap script). It is also stable and appears to be feature complete which means not having to work against a moving target.

Single-file Python/Django Deployments

Wed, 10 Jul 2019 17:29:12 -0500

Please update to our new feed url: lincolnloop.com/blog/feed/

This post covers portions of my talk, Containerless Django, from DjangoCon US 2018.

Deploying Python has improved significantly since I started working with it over a decade ago. We have virtualenv, pip, wheels, package hash verification, and lock files. Despite all the improvements, it still feels harder than it needs to be. Installing a typical large project has many steps, each one easy to trip up on:

Install Python
Install build tools (pip/virtualenv, pipenv, poetry, etc.)
Install build dependencies (C compiler, development libraries, etc.)
Download the code
Run the build tools
If you're using Node to build client-side files for a website, repeat steps 1-5 for that

It's normal for ops teams to repeat this process as part of the automated testing and then again on every server the project is deployed to. It's no wonder Docker has become so popular because of the ease in which you can build-once and deploy-everywhere.

But Docker is a heavy-handed solution and doesn't fit for every project. I envy the simplicity of languages like Go where you can compile your project down to a single binary that runs without any external dependencies. Even Java's JAR file format which requires Java to be preinstalled, but otherwise only requires downloading a single file would be a huge improvement.

A JAR File for Python

Turns out there are already a few projects solving this problem. PEX from Twitter, XAR from Facebook, and more ambitious projects like PyOxidizer, but shiv from LinkedIn hits the sweet spot for us. It is simple, stable, does not require special tooling, and incurs no runtime performance hit. It creates a ZIP file of all your code and dependencies that is executable with only a Python interpreter. In a future post, we'll do a deep-dive into how shiv works under-the-hood, but for brevity's sake, we'll treat it as a black box here.

Using Shiv with Django

Shiv works with any proper Python package. Since most Django developers don't think about packaging their projects, we'll give you a crash course here.

Packaging Your Project

Previously, the only viable packaging tool was setuptools, but since PEP-517 we now have a number of other options including flit and poetry. At the moment, setuptools is still the de-facto standard, so, despite it being a little cruftier than the other options we'll use that in our example.

You can use our post, Using setup.py in Your (Django) Project, as a starting point, but we need to take a couple more steps to ensure non-Python files (static files, templates, etc.) are included.

The easiest way to do this is with a MANIFEST.in file. It might look something like this:

graft your_project/collected_static
graft your_project/templates

Note that these directories need to live inside the top-level Python module to be included. Also note that the static files directory should be your STATIC_ROOT not your STATICFILES_DIRS. If you define templates inside your individual apps, you'll need to include those directories as well.

Dealing with Dependencies

These days, every project should include some sort of a lock file which is machine generated and defines the exact version of every dependency and the hash verifications for them. You can do this via poetry, pip-compile from pip-tools, or pipenv.

I typically let one of these tools handle the install and then pass its site-packages directory to shiv via the --site-packages option. In that case, you'll also pass the pip flags --no-deps . to install your local project, but not include any defined dependencies for it.

Including an Entry Point

We need to provide shiv a Python function that it will run when the zipapp is executed. The most logical one is the equivalent, manage.py. You can use django.core.management.execute_from_command_line directly, but I recommend writing a small wrapper which also sets the default DJANGO_SETTINGS_MODULE environment variable. You could create a __main__.py in your project and include the following:

import os
from django.core.management import execute_from_command_line

def main():
    os.environ.setdefault("DJANGO_SETTINGS_MODULE", "your_project.settings")
    execute_from_command_line()

if __name__ == "__main__":
    main()

Putting it in __main__.py is a Python convention that also allows you to execute management commands via python -m your_project ....

The Shebang

While not necessary, you can customize the shebang of your zipapp to have it executed with a specific version of Python or a Python from a specific location. I typically use /usr/bin/env python3.7 (or whatever version the project expects).

Putting this altogether, you might have something like this in your CI script:

pipenv install --deploy
pipenv run manage.py collectstatic --noinput
shiv --output-file=your_project.pyz \
     --site-packages=$(pipenv --venv)/lib/python3.7/site-packages \
     --python="/usr/bin/env python3.7" \
     --entry-point=your_project.__main__.main \
     --no-deps .

Production Webserver

Astute readers may have noticed there's not an easy way to run uwsgi or gunicorn when we want to deploy. Typically you execute your webserver, then point it at your project instead of the other way around. We created django-webserver so you'll have access to your favorite WSGI server as a management command. We also sponsored work on uWSGI to package it as a wheel making it quick and easy to use in this setup. 🎺

You'll also want to be sure that your zipapp can serve its own static files, either via whitenoise or by letting uwsgi handle your staticfiles (included by default in django-webserver).

Settings

People do all sorts of strange things with Django settings. You can still use the DJANGO_SETTINGS_MODULE environment variable to pick the settings to use at runtime. You can also use environment variables to set different values in your settings file, but that can be tedious and, potentially, a security problem.

Instead, I prefer to use a file that is easily machine readable (JSON or YAML) and also easily generated from configuration management or a secret manager like chamber or Hashicorp Vault.

We built another package, goodconf which lets you use a static configuration file (or environment variables) to adjust settings across environments. This lets us treat our zipapp more like a standard application and less like a special-case Django app. The people who handle your deployments should appreciate this.

Deployment

Once you have your zipapp, deployment is almost trivial:

Install Python
Create the configuration file
Download the zipapp (we store ours in S3)
Start server - ./myproject.pyz gunicorn or ./myproject.pyz pyuwsgi

Caveats

Zipapps created with shiv aren't a perfect solution. You should be aware of a few things before you start using them.

Extraction

On first execution, shiv will cache the zip file contents into a unique directory in ~/.shiv (path is configurable at runtime). This creates a small delay on first run. It also means you may need to periodically clean out the directory if you're doing lots of deploys.

System Dependencies

If you are using libraries which depend on system libraries, they will also need to be installed on the deployment target. For example, mysqlclient will require the MySQL library. Fortunately, the proliferation of the wheel format allows authors to bundle these libraries with their packages as is the case with Pillow, psycopg2-binary, lxml, and many others.

Flexibility

Your zipapp will define a single entry point. While it is possible to override it at runtime or even drop into a Python interpreter, I would save those options for debugging only. If you are used to running arbitrary scripts from your project or loading arbitrary files from your git repository, you'll need to use some more discipline to make management commands for the things you want to run once deployed.

Portability

Pure Python projects should be very portable across different operating systems and potentially different Python versions. As soon as you start compiling dependencies however, your best bet is to build on the same OS and Python as you intend to run the project.

Isolation

Your project will run with the system's site packages in sys.path, effectively the same as creating a virtualenv with --system-site-packages. For better isolation, use the -S flag for Python, e.g. python3.7 -S ./your_project.pyz. See this GitHub Issue for more details.

We've been successfully running this site and many of our client sites using shiv-generated zipapps for a few months now. We're very happy with the simplicity and speed it lends to rolling out new software. If you're using shiv or a similar technique to bundle your application, let us know in the comments below.