The database was deleting your progress every time we restarted
Six destructive operations ran on every server boot. One of them sat exactly on its own trigger, waiting for a single row to change.
For about three months, starting the DealArena server could drop tables, null out columns, and delete rows. Not once, during a planned migration, with someone watching. Every single time. Every deploy, every crash restart, every time a worker recycled at three in the morning because a cloud host felt like it.
We knew about none of it until we went looking.
Here is the mechanism, because it is embarrassingly ordinary. There was a function called init_db() whose job was to make sure the database had the right shape. It ran at import time, which means it ran whenever anything imported the app. Over the months, people added things to it. A table drop here, when a feature got cut. A cleanup delete there, when an agent got retired. Each one was correct on the day it was written, and each one was written as though it would run once.
They ran forever.
The one that could have hurt
Five of the six were harmless in practice, in the specific sense that a bullet is harmless if it misses. Three were dropping tables belonging to a feature we cut on 2026-05-15, when The Sauce went back to being a rotating feed instead of a custom query builder. Those tables were already gone. The drops kept firing anyway, for nearly three months, on live customer data. The genuine risk there was subtle and nasty: if anyone had ever tried to bring one of those tables back, it would have vanished on the next restart, silently, and the person debugging it would have lost a very bad afternoon.
One of them was worse in a more interesting way. When we retired a couple of agents in July, the cleanup deleted their rows by name. One of those names was enterprise_brain. Nothing writes that name today, so the delete is a no-op that has been running for a month. But it is exactly what a future Brain agent would call itself. The day somebody shipped that feature, it would have worked beautifully all day and lost its entire history overnight, on a schedule, with no error anywhere. We found a live time bomb with a delay fuse measured in product roadmap.
And then there was the challenges table, which is the one that could have reached customers.
The code read, in effect: if there are fewer than 20 challenges, delete all challenge progress, delete all challenges, and re-insert the 20 seed rows. The seed list has exactly 20 entries. The threshold was exactly 20. So the system sat precisely on its own trigger, balanced, forever. Delete one challenge row for any reason and the next server restart wipes every user's gamification progress in every workspace. The DELETE FROM challenge_progress had no WHERE clause at all.
It was also, and I enjoy this part, redundant as well as dangerous. The progress table cascades on delete already, so that statement's only unique effect was destroying the progress rows that had not lost their challenge. It was doing nothing useful and one catastrophic thing.
Nobody wrote this maliciously. Somebody wrote a reseeder, gave it a sensible-looking guard, and the guard happened to be the same number as the thing it was guarding.
What we built instead
Numbered migrations that run before the app boots. Twenty-one of them now, numbered up to 0023 because two numbers got burned in collisions between branches. Not a new idea, which is rather the point.
We looked hard at Alembic and did not use it, and the reasoning is worth stating because "we rolled our own" is usually a confession. Alembic drags a whole ORM stack into the production install for a codebase with no ORM models, which makes its autogenerate feature useless to us. More decisively, it cannot express the one migration we actually needed first: "migration 0001 is, call this existing 2,700-line function."
That baseline decision was the interesting one. The obvious move is to stamp the current schema as migration zero and start numbering from there. We ran it instead. Stamping would have frozen whatever drift production had quietly accumulated behind a hundred-odd schema alterations, each wrapped in an exception handler that logged a warning and carried on. Running it costs exactly one slow boot. What we bought is that "fresh database" and "five-month-old production database" become the same chain by construction, rather than two things we hope are similar.
The other decision that matters is where migrations run. They run from gunicorn's on_starting hook, which fires before the socket is created. A failed migration therefore exits the master process with the port never bound, the health check never passes, the deploy rolls back, and the previous instance keeps serving customers on a schema it understands. That failure mode is the entire argument for that placement. A migration system that fails after you start accepting traffic has handed you a half-migrated schema and a live audience.
The proof was pleasingly boring. First run applied the baseline in fourteen seconds and the next migration in twenty-one milliseconds. Second run reported "applied 0, skipped 0, pending 0" and returned instantly. That second line is the whole product.
The part where we were wrong in public
CI failed all three of the new gate tests on the very first run, which was humbling given that the entire point of the work was reliability.
The gates were meant to prove that importing the app creates no tables. Proving that requires a database where the schema is genuinely absent, so we used a throwaway schema with the default one off the search path. Except the vector extension is installed per database, not per schema, and it lives in the default one. So the column type stopped resolving and the baseline died inside the test while the real migration step in the same job succeeded.
Both halves of that tradeoff are real and they are mutually exclusive on a single database. The fix was to use a separate database. The interesting part is why local testing could never have caught it: the development machine has no vector extension at all, so the code falls back to a plain text column and the schema builds fine either way. The test fixture now documents both failed approaches in a comment, so the next person does not get to rediscover them.
We also found an unrelated bug while confirming that a different one was not a regression, which is how most bugs are found. It went into the handoff pile rather than into this commit, because fixing everything you notice on the way to fixing something else is how a two-day change becomes a two-week change.
Starting the server can no longer drop, truncate, or rewrite anything. That is the least exciting sentence in this entire post and it is the one we are proudest of.
— DealArena Team