The site would work all day. Then, first thing the next morning, the first person to load a page got an error, and the one after that got a page. Reload and it was fine. By ten o'clock nobody could reproduce it and everybody had a theory.
The theory everyone reaches first is the connection pool. It fits: intermittent, database-shaped, worse after quiet periods. It was wrong, and the way it was wrong is worth writing down because the correct explanation looks identical from the outside.
What the symptoms said
Errors were 503s and they came from the application, not from the reverse proxy. They clustered after idle stretches: overnight, and after lunch on a quiet day. They never appeared under load, which is precisely backwards from what pool exhaustion does.
That last detail is the one that should have redirected the investigation two hours earlier. Pool exhaustion is a busy-hour problem. It happens when demand exceeds the number of connections. A failure that only occurs when nothing has happened for a while is not contention. It is decay.
What was actually happening
MySQL and MariaDB close idle client connections after wait_timeout. The default is eight hours; plenty of managed configurations set it far lower. When the server closes a connection, it does not notify the client in any way the client is obliged to notice. The socket is simply gone.
The application's connection pool does not know this. It is holding what it believes are healthy, idle connections. The next request arrives, the pool hands one over, the driver writes to a closed socket, and the query fails. The pool then discards it and opens a fresh one, which is why the immediate retry succeeds and why the problem is impossible to reproduce deliberately.
So the sequence is: quiet period, server reaps the idle connections, first request afterwards gets a dead one, error, pool heals itself, everything looks fine. Repeat tomorrow.
The detail that made it hurt more than it should have
The box is small. One vCPU, under two gigabytes of memory, a single application instance. Prisma sizes its pool from the CPU count, which on that machine computes to three connections.
Three is a perfectly reasonable number for the load. It is a terrible number for this failure mode, because after an idle period all three are stale, and the first three requests after a quiet stretch all fail before the pool is healthy again. On a bigger pool the same bug produces one unlucky user and gets ignored for a year. On a pool of three it produces a visible morning outage.
That is worth remembering generally: small pools do not cause this bug, they amplify it into something you can see. Which, on balance, is a favour.
Why the logs were no help
Two reasons, and the second one is a lesson.
The application logged the query failure, but at a level that was not being collected, and the message named a broken pipe rather than anything about connections or timeouts.
And Caddy, sitting in front, had no log directive at all. Access logging was off. So the reverse proxy's logs showed TLS certificate maintenance and nothing else, and the absence of 5xx entries in them was read, briefly, as evidence that there were no 5xx responses.
It was not evidence of anything. A log that is not configured to record something is not testimony that the thing did not happen. That sounds obvious written down and it is remarkably persuasive at two in the morning.
The fixes, in order of how much they help
Test the connection before handing it out, or after taking it from the pool. Most pools have an option for this. It costs a round trip on checkout, which on a local database is negligible, and it eliminates the entire class of failure. This is the actual fix.
Set the pool's own idle timeout below the server's wait_timeout. If the client retires connections at four minutes and the server reaps at eight, the client never holds one long enough to be surprised. Cheap, effective, and it depends on knowing the server value rather than assuming the default.
Keep a keepalive query on a timer. Works, and it is the least pleasant of the three: you are paying continuous traffic to avoid a check you could have done on checkout.
Retry once on a connection-level error, not on a query error. Worth having as a backstop, and worth being narrow about. Retrying a failed query blindly is how you get duplicates. Retrying because the socket was dead before the query was sent is safe, because nothing was sent.
The general shape
The reason this incident is worth an article is not the fix, which is a configuration line. It is that every visible symptom pointed at capacity, and the cause was liveness.
It is the same trap as a feed that answers 200 with the wrong thing: the signal you are reading is real, and it is not measuring what you think.
Those two look the same from the request side and are opposites underneath. Capacity problems get worse with traffic. Liveness problems get worse with idleness. If your failures cluster in the quiet hours, stop looking at how many of something you have and start asking how long it has been since anybody checked whether it still works.