MongoDB secondary node in a data shard does not start with invalid argument wired tiger error

About the Community Welcome
, , , ,
1

Hi,

I am trying to restore a sharded cluster from backup on a 7.0 server. All the nodes were started on the config shard. When I tried to restore the data shard, primary node came up fine. But when I started the secondary node, it crashed with below error. I wasn’t seeing this issue before. Please can someone help?

{"t":{"$date":"2024-07-15T20:52:06.562+05:30"},"s":"I",  "c":"INITSYNC", "id":21195,   "ctx":"ReplCoordExtern-2","msg":"Writing to the oplog and applying operations until stopTimestamp before initial sync can complete","attr":{"stopTimestamp":{"":{"$timestamp":{"t":1721056925,"i":2}}},"beginFetchingTimestamp":{"":{"$timestamp":{"t":1721045730,"i":1}}},"beginApplyingTimestamp":{"":{"$timestamp":{"t":1721045730,"i":1}}}}}
{"t":{"$date":"2024-07-15T20:52:06.562+05:30"},"s":"E",  "c":"WT",       "id":22435,   "ctx":"ReplCoordExtern-0","msg":"WiredTiger error message","attr":{"error":22,"message":{"ts_sec":1721056926,"ts_usec":562853,"thread":"31219:0x7f86f0c93700","session_name":"WT_CONNECTION.set_timestamp","category":"WT_VERB_DEFAULT","category_id":9,"verbose_level":"ERROR","verbose_level_id":-3,"msg":"__wt_txn_global_set_timestamp:393:set_timestamp: oldest timestamp (1721045730, 1) must not be later than stable timestamp (1721036910, 1)","error_str":"Invalid argument","error_code":22}}}
{"t":{"$date":"2024-07-15T20:52:06.562+05:30"},"s":"F",  "c":"ASSERT",   "id":23083,   "ctx":"ReplCoordExtern-0","msg":"Invariant failure","attr":{"expr":"_conn->set_timestamp(_conn, oldestTSConfigString.c_str())","error":"BadValue: 22: Invalid argument","file":"src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp","line":2311}}
{"t":{"$date":"2024-07-15T20:52:06.562+05:30"},"s":"F",  "c":"ASSERT",   "id":23084,   "ctx":"ReplCoordExtern-0","msg":"\n\n***aborting after invariant() failure\n\n"}
{"t":{"$date":"2024-07-15T20:52:06.562+05:30"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"ReplCoordExtern-0","msg":"Writing fatal message","attr":{"message":"\n"}}
{"t":{"$date":"2024-07-15T20:52:06.563+05:30"},"s":"F",  "c":"CONTROL",  "id":6384300, "ctx":"ReplCoordExtern-0","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}}

Thanks,
Akshaya

1 Like
2

I’m encountering the same error with Mongo 7.0.12.

When I start the primary and add a replica, mongod crashes on the replica with this error.

I’m actually starting each member of the replica set from an EBS snapshot with mongo 4 data that is migrated through all major releases successfully, before I start mongo 7 (also successfully). And then it crashes when the primary attempts to initialize the replica set.