Socket timeout exception with no pending queries

Working with Data Java Frameworks
1

Hi,
I am facing socket timeout exception very frequently in my development environment even no pending running quries in mongodb.
Mongodb and Application server running on different subnet with inbetween firewall.
Application not running properly due to this problem. Please help to solve this issue and it is very critical now. Thanks.
Server Details:
App Server IP : 10.10.6.7
Mongodb Server IP : 10.9.7.7
Mongodb Version : 4.2.2

Stack Trace :

    org.springframework.data.mongodb.UncategorizedMongoDbException: Timeout while receiving message; nested exception is com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
        at org.springframework.data.mongodb.core.MongoExceptionTranslator.translateExceptionIfPossible(MongoExceptionTranslator.java:107)
        at org.springframework.data.mongodb.core.MongoTemplate.potentiallyConvertRuntimeException(MongoTemplate.java:2114)
        at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:464)
        at org.springframework.data.mongodb.core.MongoTemplate.saveDBObject(MongoTemplate.java:1080)
        at org.springframework.data.mongodb.core.MongoTemplate.doSave(MongoTemplate.java:1015)
        at org.springframework.data.mongodb.core.MongoTemplate.save(MongoTemplate.java:961)
        at org.springframework.data.mongodb.core.MongoTemplate.save(MongoTemplate.java:949)
        at in.org.db.access.DBOpMongo.save(DBOpMongo.java:306)
        at in.org.db.access.DBOperation.save(DBOperation.java:186)
        at in.org.gen.objectStore.ObjectStore.saveObj(ObjectStore.java:847)
        at in.org.gen.objectStore.ObjectStore.saveObject(ObjectStore.java:204)
        at in.org.fms.objects.FMObjectAbstract.saveOrUpdate(FMObjectAbstract.java:351)
        at in.org.fms.messaging.Messaging.saveAndSendMsg(Messaging.java:78)
        at in.org.fms.messaging.Messaging.saveAndSendMsg(Messaging.java:71)
        at in.org.fms.process.Impl.FMExpiryProcess$ExpirySubProcess.call(FMExpiryProcess.java:226)
        at in.org.fms.process.Impl.FMExpiryProcess$ExpirySubProcess.call(FMExpiryProcess.java:167)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Caused by: com.mongodb.MongoSocketReadTimeoutException: Timeout while receiving message
        at com.mongodb.internal.connection.InternalStreamConnection.translateReadException(InternalStreamConnection.java:563)
        at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:448)
        at com.mongodb.internal.connection.InternalStreamConnection.receiveCommandMessageResponse(InternalStreamConnection.java:299)
        at com.mongodb.internal.connection.InternalStreamConnection.sendAndReceive(InternalStreamConnection.java:259)
        at com.mongodb.internal.connection.UsageTrackingInternalConnection.sendAndReceive(UsageTrackingInternalConnection.java:99)
        at com.mongodb.internal.connection.DefaultConnectionPool$PooledConnection.sendAndReceive(DefaultConnectionPool.java:450)
        at com.mongodb.internal.connection.CommandProtocolImpl.execute(CommandProtocolImpl.java:72)
        at com.mongodb.internal.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:226)
        at com.mongodb.internal.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:269)
        at com.mongodb.internal.connection.DefaultServerConnection.command(DefaultServerConnection.java:131)
        at com.mongodb.operation.MixedBulkWriteOperation.executeCommand(MixedBulkWriteOperation.java:435)
        at com.mongodb.operation.MixedBulkWriteOperation.executeBulkWriteBatch(MixedBulkWriteOperation.java:261)
        at com.mongodb.operation.MixedBulkWriteOperation.access$700(MixedBulkWriteOperation.java:72)
        at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:205)
        at com.mongodb.operation.MixedBulkWriteOperation$1.call(MixedBulkWriteOperation.java:196)
        at com.mongodb.operation.OperationHelper.withReleasableConnection(OperationHelper.java:501)
        at com.mongodb.operation.MixedBulkWriteOperation.execute(MixedBulkWriteOperation.java:196)
        at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:148)
        at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:52)
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:213)
        at com.mongodb.client.internal.MongoClientDelegate$DelegateOperationExecutor.execute(MongoClientDelegate.java:182)
        at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:356)
        at com.mongodb.DBCollection.replaceOrInsert(DBCollection.java:436)
        at com.mongodb.DBCollection.save(DBCollection.java:425)
        at org.springframework.data.mongodb.core.MongoTemplate$11.doInCollection(MongoTemplate.java:1086)
        at org.springframework.data.mongodb.core.MongoTemplate.execute(MongoTemplate.java:462)
        ... 19 more
Caused by: java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:170)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at com.mongodb.internal.connection.SocketStream.read(SocketStream.java:109)
        at com.mongodb.internal.connection.InternalStreamConnection.receiveResponseBuffers(InternalStreamConnection.java:580)
        at com.mongodb.internal.connection.InternalStreamConnection.receiveMessage(InternalStreamConnection.java:445)
        ... 43 more
2

Check TCP keepalives on the server.

Ensure that the system default TCP keepalive is set correctly. A value of 300 often provides better performance for replica sets and sharded clusters. See: Does TCP keepalive time affect MongoDB Deployments? in the Frequently Asked Questions for more information.

ref: https://docs.mongodb.com/manual/administration/production-checklist-operations/
https://docs.mongodb.com/manual/faq/diagnostics/#faq-keepalive

1 Like
3

Thank you Chris.
I made all settings and as you mentiontion in the link. But still threads are waiting on socket and finally got timing out. What else I suppose to verify?

4

By waiting I suppose you mean idle ?

The firewall would be the next thing I would look into. Some will have a maximum socket lifetime. The TCP keepalive would generally keep a connection marked as active.

5

There is no firewall in between the mongodb and application server. There is one switch which spliting the network with vlan id. No other component.