Hi,
I’m running a sharded cluster and I recently upgraded mongo from 3.0 to 4.2. Some programs that run previously without errors now raise CursorNotFound error in loops like
for rec in aggregation_query:
process(rec)
where process can be quite time consuming and aggregation_query can return > 10000 values…
Errors come around 65mn from the beginning of the job (so much lower than the cursor timeout parameter, see below)
I don’t use explicit sessions
DB parameters cursorTimeoutMillis and localLogicalSessionTimeoutMinutes have already be “ugraded” to the equivalent of 2 hours (7200000 and 120 respectively).
How can I get more information (which systemLog component should I put to a debug level) ?
Any idea of how to solve that ?
Messages are like CursorNotFound: Cursor not found (namespace: 'my_db.my_collection', id: 5454438319793971081)
In which log (mongod or mongos) can I find this id ?
That seems to be quite a long time to iterate, is it possible to refactor the application code to reduce the process iteration time ? Perhaps utilise $out to a temporary collection and spawn multiple processes.
Cursor timeout is one of the possible reasons why the cursor could no longer be found. Could you ensure that the options were set correctly ? If an iteration of a cursor batch takes longer than the default cursor timeout of 10 minutes, the server deemed the cursor idle and will close it.
Could you check on logs whether there’s anything happening on the shard (i.e. replica set or config servers election) around the 65 minutes ?
I had a look in the log. Actually, it comes from the sessions handler that kills the session after 30mn and not the 2 hours as specified by localLogicalSessionTimeoutMinutes=120. All the cursors attached to the session are deleted at same time.
I updated my program to issue sessions refresh every 10mn so sessions don’t expire before the end of processing.
But now, when cursor in the main “for” loop is a find and not an aggregate, I still get cursor timeout due to sharding… (cursor is active in (one) shard but expires in others)
Before starting my request, I start a session and a thread that wakeup periodically (10mn) to refresh this session.
Then I use this session in all the long requests.
When the requests are done, I end the session.
(using mongodb 4.2)
Any idea on how to do this with the C# driver? When you start a session using db.client.StartSession(), you get an IClientSessionHandle, but no hint of documentation on how to call RunCommand with “RefreshSessions” and a session ID. There isn’t any property on the IClientSessionHandle that seems to map to the session ID.
You are correct that you can create explicit sessions in the .NET/C# Driver using client.StartSession(). The session handle contains a wrapped session, which allows you to access the sessionId. You can then use the sessionId (which must be in GuidRepresentation.Standard format) to construct a refreshSessions command.
BsonDefaults.GuidRepresentation = GuidRepresentation.Standard;
var client = new MongoClient();
using var session = client.StartSession();
Console.WriteLine(session.WrappedCoreSession.Id);
var db = client.GetDatabase("test");
var pingResult = await db.RunCommandAsync<BsonDocument>(session, "{ping:1}");
var refreshResult = await db.RunCommandAsync<BsonDocument>(session, new BsonDocument { { "refreshSessions", new BsonArray{session.WrappedCoreSession.Id} }});
Console.WriteLine(pingResult);
Console.WriteLine(refreshResult);
You would want to do something smarter whereby you’d call refreshSessions periodically while the other running command is still pending. Hope that helps.