MongoDB uses 100%CPU on RHEL9

Other MongoDB Topics
1

Hello,

I have ReplSet - 3 nodes installed on EC2.
Currently it is installed on Centos 7, but I need to migrate to RedHat 9.
I did following:

  • created 3 new EC2 (the same instance type as is) with RedHat 9 AMI
  • installed MongoDB 6.0.20
  • connected to ReplSet and synched (initial sync)
  • removed 3 old nodes (Centos 7) from replSet
  • performed load tests
    Day after tests, there is no any traffic, mongo nodes are using 100% CPU, PRIMARY and SECONDARY too.
    Previous configuration (Centros 7) used up to 60% CPU during load, and lower than 10% CPU after tests. I’m confused, why it is…

Appreciate for any comments or suggestions.

2

I can confirm downgrading openssl to version 3.0.7-28 avoids this issue.

However, note that there is a fix for it in mongo version 8.0.5
https://jira.mongodb.org/browse/SERVER-97842

3

Ran across this:

The solution is to downgrade openssl to version 3.0.7-28. I had to run sudo yum downgrade openssl twice on our RHEL9 servers to get to that point. Restarted them and by tomorrow I’ll know if it the solutions was successful.

4

I am seeing very similar symptoms on RHEL9.

I currently have a 3 node cluster in Mongo 8.0.4 on RHEL8.
I have three new RHEL9 VMs, added them as secondary nodes (6 nodes in cluster now, 1 primary, 5 secondaries). At first everything is fine, sync’d up, all green. Can run load tests on the application which uses the servers, no problems and all nodes have similar traffic and CPU stats. Many hours later, the RHEL9 nodes start to step up CPU usage 100% per core at a time until all cores on each of the new server are maxed out. (all usage from the mongod process) Forcing a re-sync on one of the new replicas causes the CPU to go back to normal for a time.

While a node has all CPUs pegged, I don’t see anything out of the ordinary in the logs, and if I look at the db.Ops(); on a maxed out replica, it doesn’t appear any different than the RHEL8 replicas that are using minimal CPU in the same cluster. No long ops or anything suspicious looking. This leads me to believe it’s something internal to mongod I don’t have visibility into the performance of.

5

Thank you !
I did downgrade OpenSSL to version 3.0.7 on RHEL 9 nodes.
I’m waiting to tomorrow, to see results.
I would expect that the same issue with OpenSSL was on my configuration: RHEL 9 and MongoDB 6.0.
I will let you know.