It Worked in Staging. Production Destroyed It. Queues Saved It.

On a random Tuesday in March, I got assigned a task to build a complete feature end to end. The goal was straightforward. When a user uploads an image of a water quality test sample, the system should notify them on WhatsApp and ping the admins or test reviewers. Once a reviewer finishes the analysis, the system generates an AI summary comparing the parameters to global safety standards, compiles a branded PDF report, stores it in cloud storage, and sends the customer a notification with a download link. Finally, if dangerous contaminant levels are detected and a physical device is linked to the test, an emergency support ticket gets raised automatically.

Sounds like a long trip. And it is. A single test completion triggers multiple sequential external API calls. OpenAI alone takes 5 to 10 seconds, and the PDF service adds another 2 to 3 seconds on top. End to end, processing one test can easily consume 15 or more seconds of wall clock time.

What I Built First

With my understanding of the architecture at the time, I went with the most straightforward approach I knew. The pipeline was sequential and synchronous:

1. Receive the test payload from the database trigger
2. Call OpenAI to generate the AI summary
3. Generate the PDF report on the server using that summary
4. Save the report to BunnyCDN
5. Update the database with the CDN URL
6. Send out WhatsApp and push notifications
7. Evaluate results against safety thresholds and raise emergency tickets if needed

It worked perfectly on my local machine. I pushed it to staging, ran through the test cases, and everything looked solid. I shipped it.

A few days later, my manager called. Reports were not generating. I checked locally—everything still worked fine. I assumed it was an environment mismatch, tightened up the config, and pushed again. Same thing happened a few days later, except this time the server actually crashed.

I opened Grafana to look at the logs and finally saw the real issue. Postgres was throwing connection limit errors. The whole pipeline was completely synchronous and it was choking the database.

Why the Synchronous Approach Failed

The root of the problem was something I had not thought about carefully enough. Our application uses a PostgreSQL LISTEN client to receive database change notifications. This client holds an open database connection for the entire lifetime of its session. The moment a waterquality_changed notification came in, my listener would await the entire pipeline, all 5–10 seconds of it while that database connection stayed open.

Now imagine five testers submitting results at the same time during a field session. That is five listeners each holding a Postgres connection open for 15 seconds simultaneously. Our connection pool has a hard limit. Once it is full, every other part of the application that needs a database connection—API endpoints, cron jobs, other listeners—starts timing out and throwing errors. Unhandled rejections in Node.js are fatal to the process. The server goes down and returns 502 errors to everyone until it restarts.

Even in scenarios that never hit full pool exhaustion, the synchronous approach had zero resilience. If OpenAI was temporarily unavailable or the PDF service timed out halfway through, the entire job would fail silently. There was no retry. The customer would never receive their report or notification, and from the outside the feature looked like it was working because the test data was saved correctly to the database. The downstream processing had just evaporated without a trace.

What made it worse was that everything had looked fine in staging. Staging has light, predictable traffic. Production has field testers submitting results in bursts. That mismatch is what hid the problem until it blew up in a live environment.

The Architecture We Already Had

Here is the part I wish I had noticed earlier. pg-boss was not something we needed to introduce—it was already the backbone of our background processing for sensor data streams, device status events, emergency handling, and more. It is a durable job queue that persists jobs directly to Postgres, manages worker polling, enforces concurrency limits, automates retries with configurable backoff, and routes permanently failed jobs to Dead Letter Queues for human review.

The water quality pipeline had simply never been wired into it. The fix was to route this heavy, fault-prone processing chain through the same infrastructure that was already handling our most critical real-time workloads.

The Fix: Separating the Trigger from the Processing

The core architectural change was clean. Instead of the trigger listener doing all the work, it now does only one thing: receive the database notification and hand the payload off to a background queue immediately. That handoff takes milliseconds. The database connection is released right away, regardless of how long the downstream processing takes.

On the other side, a new dedicated worker continuously polls the wq-report-pipeline queue for jobs. When it picks one up, it runs the full processing chain—AI summary, PDF generation, notification dispatch, and safety checks—entirely in the background, completely detached from the trigger lifecycle. The worker is configured with a hard concurrency limit of three simultaneous jobs, which prevents a spike of uploads from overwhelming OpenAI or the PDF service at the same time.

Two queues were registered: the main wq-report-pipeline for active jobs, and wq-report-pipeline_dlq as its Dead Letter Queue. The DLQ has to be registered first because pg-boss validates the foreign key relationship at creation time. Any job that exhausts its retry limit gets automatically moved to the DLQ, where it can be inspected and manually replayed once the root cause is resolved. No report is permanently lost due to a transient infrastructure failure.

The Hidden Bugs That Came After

Wiring the pipeline into pg-boss was not the end of it. Deploying the integration revealed several bugs—some from our own logic, some from breaking changes in pg-boss v10 that were not obvious until they hit production.

The Queue That Never Existed

The first symptom was boss.send() consistently returning null as the job ID. In pg-boss, a null return from send() means the queue row could not be found in the database during the job insertion. The queue simply did not exist.

The reason was subtle. Our queue setup script initialized over fifteen queues inside a single shared error handler. If any queue earlier in that list encountered a conflict or a foreign key violation during startup, the exception would be caught at the top level and the entire remaining initialization sequence would be abandoned silently. Our wq-report-pipeline queue added in the middle of the list never made it into the database.

The fix was to give every queue its own isolated error handler. One queue failing during startup now produces a single log warning for that specific queue and nothing else. We also added an explicit updateQueue call for wq-report-pipeline on every startup, because pg-boss uses ON CONFLICT DO NOTHING in its queue creation SQL. If the queue row already existed from a previous deployment with stale or missing configuration, createQueue would silently skip updating it. The updateQueue call ensures the retry limits, expiry, and dead letter configuration are always in sync with the current code.

pg-boss v10 Breaking Changes

We had recently upgraded to pg-boss v10 and hit two breaking changes that were not immediately obvious.

First, deadLetter can no longer be passed inside the send() options. In older versions this was tolerated. In v10 it is strictly a queue-level policy. Passing it in send() caused the internal job insertion to fail silently which is exactly why send() kept returning null even after the queue initialization issue was fixed.

Second, the worker handler signature changed. In earlier versions, the .work() callback received a single job object as its argument. In v10 it receives an array of jobs, even when only one job is being processed at a time. Our worker was trying to read properties like id and data directly off an array, which returned undefined. The logs showed Starting wq-report-pipeline job undefined followed immediately by a crash reading device_id from an undefined value. Updating the handler to iterate over the received array fixed this completely.

Before vs. After

Why This Actually Holds Under Load

Ten simultaneous uploads now result in ten near-instant trigger handoffs. The database connection pool is never pressured by the processing pipeline. Jobs wait safely in Postgres, durable across server restarts, and are processed in a controlled, throttled manner. Transient failures at any external service result in an automatic retry, not silent data loss. The main API thread is never blocked.

The honest takeaway for me was that the synchronous approach did not feel wrong when I was building it. It worked locally. It worked in staging. It only betrayed me when real, concurrent load hit it in production. That gap between “works on my machine” and “works under pressure” is exactly what asynchronous job queues are designed to close.