Connection pooling in microservices deployed in Kubernetes

Although so many articles are available on microservices and using the database connections with microservices, lot of ambiguity still exists in using connection pools for connecting to database and disastrous impact it creates due to this antipattern especially when deployed in Kubernetes environment as pods. (Concurrent requests can be handled in Kubernetes by so many techniques like increased replicas, introducing a micro second delay which I will right a separate article).

Googling and making a lot of search has provided little results and therefore I like to share some fundamentals on this subject and shed some light, as most developers tend to breach this, ignoring the consequences it brings and even experienced architects are overlooking this, creating nightmares and induce fire fighting for the operations team to resolve this.

When it comes to database, two types of practice is being followed widely in the microservice world. Database per microservice pattern which suggest creating and using one database per microservice. The other option is to have a common database and having an access for all microservices on top of it. This approach is widely known as shared database approach.

Both the approaches has its own advantage and disadvantage and to choose which option varies among use cases. Also there are additional challenges on implementing these approaches, as it depends on the database chosen. Traditional RDBMS provide less comfort on per service per database and there are some new databases has come recently in the market exclusively for microservices to address this issue and that discussion is out of scope for this topic.

As the industry is moving towards Gitops paradigm from devops and started providing new meaning for CI/CD, the executable jar files seldom deployed as plain vanilla executable. Every microservices are containerized and deployed in an orchestrated environment like Kubernetes as pods. Even though the microservice and the database are deployed as independent containers inside a pod, single connection is more than suffice to access the database.

If the pods are replicated, each pod will internally connect to the database and the connection happens within the boundary of the pod. So this ensures that the memory consumed for each connection is well within control and the resource limits requested for the pod does not cause any hindrance to the running node when the deployed pods in a node increases in count. Kubernetes will be comfortably manage this and deploy or redeploy the pods to other nodes without creating any node failures due to resource crunch.

There is no point in creating a connection pool inside a microservice when deployed in Kubernetes adopting per database per microservice strategy as only one connection is used mostly unless increase in userbase. Infact a direct connection is more than enough. The data will be stored in persistent volumes and the application might be deployed as a statefulset based on the use case and will write a separate article for that later.

When a shared database for all microservice approach is made, we can use connection pool here with the no of database connections can be created in a connection pool for reuse. The queries are received as a string or a number (as decided by the architecture) from other pods (no database containers, only microservice container embedded on these pods), the query is executed and a resultset is returned as response. The connections in this database service can be reused. This pod can be deployed in kubernetes using taints and toleration, where other pods which has no tolerations cannot be deployed in this node.

When nosql database like scylla are used on per database per microservice strategy, the no of sessions it created due to increased connections in the pool sitting without any use caused sleepless nights to our team and this story continued for several months.