Why Is My Node.js App Hanging During Database Reconnects?

Software, Cloud and Development Coding & Programming
1

Hello

I am building a Node.js application that connects to a SQL database & it usually works fine until the database connection drops (perhaps due to idle timeout or server restart). :slightly_smiling_face: After the disconnect; the app tries to reconnect, but this causes the entire application to hang / become unresponsive. :upside_down_face:

Restarting the process helps temporarily but that’s not a robust solution for a long-running production setup.:innocent:

I have tried using libraries like node-postgres with built-in retry options & even implemented a manual reconnect loop but I still face long delays or blocked event loops during reconnection attempts. :innocent:

I wonder if using pattern like exponential backoff, connection pooling / streaming reconnections can help. :thinking: I would love to know best practices / patterns that seasoned developers use to handle this more gracefully. Checked mysql - Managing database connections in Node.js, best practices? - Stack Overflow guide for reference. I was also wondering what is a DevOps Engineer supposed to handle in cases like this?:thinking:

If anyone has experience handling these cases especially in production-grade Node.js services I would appreciate examples / advice. It’s crucial to keep the app responsive even during temporary database outages.

Thank you !!:slightly_smiling_face:

2

Just based on the information provided, connection pooling is the way to go (use a standard library, don’t do DIY unless you know what you’re doing). Also worth checking the connection limits on the DB.

As you haven’t shared the code, I’m just listing a bunch of common issues I’ve come across, might be worth checking.

1, Not handling the connection_close() logic properly. This is by far the most common issue when not using a connection manager. DB usually holds on to old connections and chokes up.

2, Check if you’re actually using reusing the connection and not doing a connect() call for every query, this might sound silly but have seen it happen plenty.

3, Check the number of simultaneous connections allowed in DB, sometimes the defaults are too low.

4, Check if the DB has some rate limiting mechanism. Not very common but worth checking.

The only weird thing about your scenario is restarting the app. In most of the cases mentioned above you’d have to restart the DB as that’s the easiest way to clear the zombie connections. So either your app is somehow closing all the connections on restart or it’s a different issue than the ones mentioned above.

1 Like