Testcontainers reuse: 172 zombie containers ate my laptop

A dark warehouse wall stacked with 172 identical glowing containers while a small reaper stands locked outside the door

172 running containers. 160 of them postgres:18, twelve Mailpit, the oldest up for eight days. All inside the Podman machine on my Mac, all left behind by Quarkus test runs, and nothing was ever going to clean them up.

This is the companion to the Docker 29 post. That one was Testcontainers breaking loudly in CI. This one is Testcontainers succeeding quietly, and leaking, on a laptop. The cause is one line:

# ~/.testcontainers.properties
testcontainers.reuse.enable=true

Reusable containers are excluded from Ryuk, Testcontainers' cleanup mechanism, by design. The rest of this post is how 172 of them piled up unnoticed, why reuse behaves this way, and the settings that stop it.

The symptom

A 32 GB Apple Silicon Mac, sluggish under normal load. The usual suspects (browser tabs, chat apps) were each a few hundred MB. The standout in ps was krunkit, the libkrun-based VM behind podman machine. So my first instinct: the Podman machine is too fat, shrink it.

The number that gave it away was in sysctl vm.swapusage:

vm.swapusage: total = 23552.00M  used = 22397.38M  free = 1154.62M

22 GB of swap in use on a 32 GB machine. Something had been demanding far more memory than the machine has, repeatedly. krunkit's resident size at that moment was only 2.8 GB, but the machine was configured with an 11 GB ceiling, and libkrun ratchets toward that ceiling under load without handing memory back.

The reveal

Before shrinking the VM, count what's actually running inside it:

$ podman ps -q | wc -l
172
$ podman ps --format '{{.Image}}' | sort | uniq -c | sort -rn
 160 docker.io/library/postgres:18
  12 docker.io/axllent/mailpit:v1.30.0

172 containers, the oldest "Up 8 days". A label check on one of them:

$ podman inspect <id> --format '{{json .Config.Labels}}'
{
  "org.testcontainers": "true",
  "io.quarkus.devservice.launch-mode": "TEST",
  "io.quarkus.devservice.process-uuid": "2e57977c-...",
  "restart": "no"
}

Every single one carried org.testcontainers=true. These are Quarkus Dev Services: when a Quarkus test starts and the datasource has no explicit URL, Quarkus uses Testcontainers under the hood to start a real PostgreSQL, and with quarkus-mailpit-testing, a Mailpit. The process-uuid is unique per JVM run. 160 different ones meant 160 separate test runs had each left their database behind.

No Java process was running. They were pure zombies.

Why Ryuk didn't save me

Testcontainers ships Ryuk, the resource reaper: a small companion container with access to the Docker socket. It watches the JVM's connection and removes every labelled container about ten seconds after that JVM disconnects. That's the safety net that makes leaks impossible, even when a test run dies on kill -9 or the IDE stop button.

Reusable containers are deliberately excluded from Ryuk. That's the entire point of reuse: the container is meant to survive the JVM exiting, so the next run can attach to it instead of paying the cold start again. With testcontainers.reuse.enable=true, two things happen at once for any container marked reusable. Ryuk never reaps it. And Quarkus doesn't stop it on shutdown either; Dev Services leaves it running on purpose. The Testcontainers docs state it plainly: reused containers "won't stop after all tests are finished".

Quarkus opts its database Dev Services into reuse by default once that global flag is set (quarkus.datasource.devservices.reuse defaults to true), and reuse applies to test runs, not just dev mode. The Quarkus docs warn about exactly this: old containers "might be left running indefinitely" and you "need to stop and remove these containers manually".

In steady state, reuse keeps one warm container per service and reattaches to it. But it reattaches only when the container configuration matches exactly. The moment anything changes (a new branch, a migration tweak, a different test setup) Testcontainers finds no match, and it starts a fresh reusable container. The old one keeps running, because nothing is allowed to reap it.

Multiply that by a week of branch-switching and you get 160 orphaned databases.

The fix

Reuse is a convenience for quarkus:dev: keep a warm database across restarts. In the test profile it's a trap. Tests run constantly, configs churn, and every config mismatch leaks a container. I turned it off globally:

# ~/.testcontainers.properties
testcontainers.reuse.enable=false

With reuse off, Ryuk reaps test containers about ten seconds after the JVM disconnects, Quarkus tears its Dev Services down on a clean shutdown, and a hand-written GenericContainer calling withReuse(true) becomes a no-op, because the flag is honoured only when the global switch is on. The only thing lost is a warm dev database between quarkus:dev restarts. With a migration-seeded schema that costs a few seconds.

If you want to keep reuse for dev and only disable it for tests, there's a surgical version for databases in application.properties:

%test.quarkus.datasource.devservices.reuse=false

That covers the datasource. Not every dev service exposes a switch like this; the Mailpit extension doesn't. The global off is the only setting that covers them all, which is why I went with it.

Then reap the survivors. Scope the removal to the label, so it can't touch anything else:

podman rm -f $(podman ps -aq --filter label=org.testcontainers=true)

The moment those 172 were gone, krunkit's resident size dropped from 2.8 GB to 66 MB and the guest went from 6.8 GB used to 1.5 GB.

While you're in there: right-size the machine

The leak was the cause, but the oversized machine let it do damage. Two krunkit behaviours worth knowing on Apple Silicon. Freed guest RAM doesn't come back to macOS on its own; in my testing it came back only on podman machine stop. And the disk image is the same story for storage, a raw image that grows and never shrinks. That one is a separate post.

So cap the machine as a guardrail. Resizing needs a stop and start, but it does not recreate the machine; your images and volumes are untouched:

podman machine stop
podman machine set --memory 6144 --cpus 4
podman machine start

6 GB and 4 vCPUs is plenty for a couple of Dev Services containers, and the next leak can't blow past 6 GB and take the whole laptop with it.

Lessons

"Podman is heavy" is a symptom, not a diagnosis. Run podman ps | wc -l before you shrink the VM. The fat process was real; the cause was 172 things inside it.
Reuse opts out of Ryuk by design. Enabling testcontainers.reuse.enable removes the safety net, and Quarkus applies it to tests by default once the flag is set. Use reuse for dev, never globally for test.
Reuse only reuses on an exact config match. Branch and config churn produce a fresh reusable container each time, and reusable containers never die. The failure mode is silent accumulation, not an error.
krunkit gave RAM back only on podman machine stop. Right-size the machine so a leak is bounded, and stop it when you're not using it.
Scope destructive cleanup to a label. --filter label=org.testcontainers=true means a mass rm -f can only ever hit the things you mean.

References

Testcontainers reuse docs: the reuse feature, including its experimental status and the Ryuk exclusion.
Quarkus database Dev Services guide: where quarkus.datasource.devservices.reuse and the manual-cleanup warning are documented.
moby-ryuk: the resource reaper itself.

This is one of a small series of Java-on-containers traps we've written up. See also how Docker 29 breaks Testcontainers 1.x in CI, why JAVA_OPTS_APPEND is a fallback, not an append on Red Hat UBI images, and the JDK 25 AOT cache portability trap.

Running Quarkus and tired of walking into this kind of thing alone? Talk to us about managed Java hosting.