Four numbers were keeping the database awake: 5, 30, 5, and 10
Serverless Postgres bills you for being awake. Four background workers were politely asking it a question every few seconds, forever.
The database this product runs on is serverless, which means it goes to sleep when nobody is using it and you stop paying for the compute. That is the entire pitch, and it is a good one for a product with a small number of users doing their work during the day.
Ours never slept. Not once. Not at three in the morning on a Sunday.
The cause was four numbers, sitting in four different files, each of them individually reasonable.
The bulk jobs worker checked for new work every five seconds. The sequence executor checked every thirty. The enrichment bulk worker, every five. The webhook delivery worker, every ten. Four background daemons, each waking up on a timer, each asking the database whether anything had happened, each being told no, each going back to sleep for a few seconds to try again.
The idle window on the compute is measured in minutes. The longest gap any of these four left was thirty seconds. So the machine was never idle by its own definition, and it ran continuously, twenty-four hours a day, serving four workers asking a question whose answer was almost always nothing.
Polling is a shape, not a bug
I want to be fair to the code that did this, because polling is not a mistake. It is the simplest possible correct design for a background worker, and it has one enormous virtue: it cannot get stuck. If a job appears by any mechanism at all, whether through the application, a script, a manual database insert, or something you have not thought of yet, the worker finds it within one interval. There is no wiring to forget.
The trouble is that polling has a cost structure that is invisible on a server you rent by the month and extremely visible on one you rent by the second. On a fixed monthly box, a query every five seconds is free. It is noise. The exact same code, moved onto compute that bills for wakefulness, is a standing order to never stop paying.
Nothing about the code changed. The economics underneath it did, and no test tells you that.
The fix, which is 250 lines across five files, is to invert the flow. Instead of asking repeatedly whether work has arrived, the workers now block on an event and wait to be told. The places that create work, enqueuing a job, enrolling a prospect in a sequence, emitting a webhook, poke the relevant worker directly. An idle worker issues zero queries, because an idle worker is not doing anything at all.
The two workers that run on a schedule rather than on demand got a different treatment, and it is the more elegant half. The sequence executor does not need to check every thirty seconds whether a step is due. The database already knows when the next step is due, because that timestamp is a column. So the worker asks once for the minimum of that column, sleeps exactly that long, and wakes up when there is genuinely something to do. The same trick works for the webhook worker and its retry schedule.
That is the part I find satisfying. The information required to sleep correctly was already in the database the whole time. The old design threw it away every five seconds and asked again.
The backstops
Four new configuration values came with it, one per worker, each an upper bound on how long that worker is allowed to sleep.
Those exist because event-driven code has exactly one failure mode, and it is worse than polling's. If a notification is ever missed, whether through a crash between the write and the poke, a code path someone adds later that forgets to notify, or a race, then a polling worker recovers in five seconds and an event-driven worker waits forever. Silently. With a queue filling up behind it.
The maximum sleep is the admission that this will eventually happen. It converts "stuck until someone notices" into "late by at most this long," which is the difference between an outage and a delay. It is a deliberate, small reintroduction of the thing we just removed, sized so that it costs almost nothing while the notifications are working correctly.
Any time you replace polling with events, you should be able to point at the line that catches the missed notification. If you cannot, you have not built an event system, you have built a bet.
There is a broader point about serverless infrastructure hiding in here. The pitch is that you stop thinking about capacity, and that is true for the request path, where traffic genuinely is bursty and idle genuinely is idle. It is false for everything else your application runs. Background workers, health checks, cron jobs, keepalives and connection pools all have opinions about how often to touch the database, those opinions were formed in an era when touching the database was free, and none of them will tell you they are the reason your bill has no floor.
Four numbers, in four files, none of them wrong when they were written.
— DealArena Team