Free-Threaded Python Meets HTTP/3: Modernizing the Load Balancer Behind rackit.io

apython_lb is the async reverse proxy and load balancer we built to sit in front of rackit.io’s own backends: round-robin and sticky routing, health checks with a circuit breaker, per-IP rate limiting, a libmodsecurity WAF, and a live management UI, all on top of Quart and Hypercorn. Over the last three releases (v0.3.0 through v0.5.0) we pushed it onto some genuinely new ground for a production Python service: a free-threaded interpreter, HTTP/3, and an independently-versioned base image. Here’s what changed, what broke along the way, and the load test numbers that came out the other end.

v0.3.0: HTTP/3 on free-threaded Python

Python’s free-threaded builds (PEP 703, stable as of 3.13) remove the GIL as a build option. We wanted it for one simple reason: it’s where CPython is headed, and there’s no cost to adopting it early if nothing in the stack actually depends on the GIL for correctness. The catch is that no official Python Docker image — not python:3.14-slim, not any version — ships a free-threaded build. We ended up compiling CPython from source, following the official docker-library/python recipe with --disable-gil added, on debian:trixie-slim.

HTTP/3 was the more interesting problem. Hypercorn supports it through an h3 extra built on aioquic, and aioquic — along with its own QPACK dependency, pylsqpack — hard-codes Python’s Limited API in its build config. Free-threaded CPython doesn’t support the Limited API at all; Python.h itself refuses to compile with an explicit error. Neither library’s upstream wheels nor a stock source build would install. We fetched both from source and rebuilt them against the full API instead — the same fix every other native dependency in the image didn’t need, because cryptography, pycares, and cffi already shipped real free-threaded wheels for that Python version (that stopped being true for one of them a couple releases later — more on that below).

That fix gets the HTTP/3 stack compiling, but not fully free-threaded in practice: aioquic’s own C extensions don’t declare themselves safe to run without the GIL, so CPython’s default behavior is to silently re-enable it the moment they load — which, since HTTP/3 is always on, means immediately at startup. We override that with PYTHON_GIL=0, but only after checking it was actually safe to: Hypercorn’s QUIC implementation (UDPServer/QuicProtocol) processes every datagram through a single asyncio.Queue drained by one coroutine on one event-loop thread. No thread pool, no run_in_executor. As long as that stays true, nothing ever touches aioquic’s unsynchronized internal state from two threads at once — which is what actually matters, independent of whether the GIL is nominally on or off.

Smoke-testing the new image against a real WAF ruleset also turned up a bug that had nothing to do with any of the above: a URI containing a null byte sailed straight past rule 910001 and crashed the proxy with a 500 instead of a clean 400. The root cause was in libmodsecurity itself — msc_process_uri() takes the URI as a plain null-terminated C string with no length argument, unlike the length-prefixed calls used for headers and body, so the byte silently disappears before ModSecurity ever inspects it. We fixed it the only place it could be fixed: enforcing the null-byte check in Python before the request ever reaches the C API.

v0.4.0: workers and event loops, and the race they exposed

Hypercorn supports running multiple worker processes and swapping its event loop implementation, but neither was wired up to anything — so we added HYPERCORN_WORKERS and HYPERCORN_WORKER_CLASS, both read from environment variables in Docker and Kubernetes alike.

The event loop choice mattered for the same reason PYTHON_GIL=0 did in v0.3.0: we had to re-verify the safety argument for every option we were about to offer, not just the default. uvloop turned out to be the easy case — it reuses Hypercorn’s exact same single-threaded asyncio code path, and it ships real free-threaded wheels that don’t trigger CPython’s GIL-reenable at all. trio was the opposite: it’s a fully separate async runtime, and our SQLite layer (aiosqlite) calls asyncio.get_event_loop() internally, which simply doesn’t exist under trio. The app crashed on startup on any Python, free-threaded or not — nothing to do with the rest of this work. We left it out entirely rather than ship an option that doesn’t function.

Turning on multiple worker processes surfaced a real bug of its own: every worker opens the same SQLite file at startup, and they race to switch a brand-new database into WAL mode. That transition takes a brief exclusive lock that — confirmed by actually triggering it, not just reading the SQLite docs — doesn’t honor the connection’s busy_timeout. A losing worker used to crash outright on database is locked instead of just waiting its turn. We added a targeted retry around that specific transition. It’s the kind of bug that’s easy to miss because a single-worker deployment never exercises the code path at all.

v0.5.0: a base image with its own release cycle

By the third release, the from-source Python build had grown into roughly 130 lines of apt-mark bookkeeping and configure flags that had nothing to do with apython_lb itself. We pulled it out into its own repo, rackitio/pythont, published independently on Docker Hub. apython_lb’s own Dockerfile now just does FROM rackitio/pythont:3.15.0, pinned to an exact version rather than a floating channel — pythont also ships stable/latest tags that track the newest Python release automatically, which is exactly the kind of surprise a production pin shouldn’t be exposed to.

That pin turned out to matter immediately: Python 3.15.0 shipped upstream while this was in flight, and pythont’s own release automation picked it up the same day, freezing the 3.14 line at its final patch. We moved straight to 3.15.0 rather than land on a version that was about to stop receiving updates. The jump surfaced one real, if mundane, break: pycares (aiodns’s C extension) didn’t have a free-threaded wheel yet for the new version, so pip fell back to a source build — and pycares’s own build now vendors c-ares via CMake, which we simply didn’t have installed. Everything else, including our own from-source aioquic/pylsqpack builds, compiled against the new headers without any changes.

Load testing the result

To see what all of this actually bought us, we ran k6 against a live deployment — not a synthetic health check, but the full request path: WAF inspection, rate limiting, IP tracking, backend selection, and a real proxied response. 1,500 virtual users, ramped up over 30 seconds, held for a minute, ramped back down.

Every one of the 113,550 checks run across those two minutes passed. The k6 script we used is in the repo if you want to point it at your own deployment.

Takeaway

None of these three releases were about chasing a benchmark. They were about being honest with ourselves about what a given piece of new ground — a free-threaded interpreter, an HTTP/3 stack still catching up to it, multi-process scaling — actually required us to verify before we trusted it, and fixing what that verification turned up. The load test numbers are the result of that discipline, not a substitute for it: a load balancer that holds a clean p95 under 1,500 concurrent users with zero failed requests is doing its job precisely because the pieces underneath it — the WAF, the backend selection, the SQLite layer, the HTTP/3 stack — were each checked under real load, not just assumed to work.

apython_lb is open source at github.com/rackitio/apython_lb, with the free-threaded Python base image at github.com/rackitio/pythont. If you’re running Python infrastructure and curious about free-threading or HTTP/3 in practice, the Dockerfile comments in both repos are written to be read — they’re where we kept the actual reasoning, not just the commands.