Hi!
I have this problem, my MongoDB container always exceeds my k8s memory limits after a while.
For example after working an hour on load, it can reach even 4/6GiB of ram.
My k8s yaml contains:
resources:
requests:
memory: "1Gi"
limits:
memory: "3Gi
Also, i specify MongoDB to use config file: command:
["mongod","--config","/etc/mongod.conf"]
Which has:
storage:
wiredTiger:
engineConfig:
cacheSizeGB: 1
On start up i even get log that shows limitations are working on k8s: “availableMemSizeMB”:3072 and on “cacheSizeGB”:1.0…
{"t":{"$date":"2020-11-20T10:20:54.897+00:00"},"s":"W", "c":"CONTROL", "id":20720, "ctx":"initandlisten","msg":"Available memory is less than system memory","attr":{"availableMemSizeMB":3072,"systemMemSizeMB":15847}}
{"t":{"$date":"2020-11-20T10:20:54.897+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"4.4.0","gitVersion":"563487e100c4215e2dce98d0af2a6a5a2d67c5cf","openSSLVersion":"OpenSSL 1.1.1 11 Sep 2018","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu1804","distarch":"x86_64","target_arch":"x86_64"}}}}
{"t":{"$date":"2020-11-20T10:20:54.897+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"18.04"}}}
{"t":{"$date":"2020-11-20T10:20:54.897+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"config":"/etc/mongod.conf","net":{"bindIp":"*"},"storage":{"wiredTiger":{"engineConfig":{"cacheSizeGB":1.0}}}}}}
On Grafana i can see fallowing use of RAM:
Extra info that i think might be useful:
MongoDb version: 4.4.0
db.stats()
{
"db" : "YYY",
"collections" : 2,
"views" : 1,
"objects" : 370687,
"avgObjSize" : 31445.03214302093,
"dataSize" : 11656264630,
"storageSize" : 2233819136,
"indexes" : 33,
"indexSize" : 929951744,
"totalSize" : 3163770880,
"scaleFactor" : 1,
"fsUsedSize" : 10017492992,
"fsTotalSize" : 161033994240,
"ok" : 1
}
db.serverStatus().wiredTiger.cache
{
"application threads page read from disk to cache count" : 3321,
"application threads page read from disk to cache time (usecs)" : 9010883,
"application threads page write from cache to disk count" : 958,
"application threads page write from cache to disk time (usecs)" : 363838,
"bytes allocated for updates" : 7305804,
"bytes belonging to page images in the cache" : 170520520,
"bytes belonging to the history store table in the cache" : 554,
"bytes currently in the cache" : 193956579,
"bytes dirty in the cache cumulative" : 173682490,
"bytes not belonging to page images in the cache" : 23436058,
"bytes read into cache" : 44543321,
"bytes written from cache" : 206773479,
"cache overflow score" : 0,
"checkpoint blocked page eviction" : 0,
"eviction calls to get a page" : 7724,
"eviction calls to get a page found queue empty" : 87,
"eviction calls to get a page found queue empty after locking" : 189,
"eviction currently operating in aggressive mode" : 0,
"eviction empty score" : 0,
"eviction passes of a file" : 3626,
"eviction server candidate queue empty when topping up" : 23,
"eviction server candidate queue not empty when topping up" : 421,
"eviction server evicting pages" : 0,
"eviction server slept, because we did not make progress with eviction" : 1870,
"eviction server unable to reach eviction goal" : 0,
"eviction server waiting for a leaf page" : 0,
"eviction state" : 64,
"eviction walk target pages histogram - 0-9" : 1940,
"eviction walk target pages histogram - 10-31" : 1310,
"eviction walk target pages histogram - 128 and higher" : 0,
"eviction walk target pages histogram - 32-63" : 292,
"eviction walk target pages histogram - 64-128" : 84,
"eviction walk target strategy both clean and dirty pages" : 0,
"eviction walk target strategy only clean pages" : 0,
"eviction walk target strategy only dirty pages" : 3626,
"eviction walks abandoned" : 316,
"eviction walks gave up because they restarted their walk twice" : 1084,
"eviction walks gave up because they saw too many pages and found no candidates" : 820,
"eviction walks gave up because they saw too many pages and found too few candidates" : 41,
"eviction walks reached end of tree" : 3153,
"eviction walks started from root of tree" : 1955,
"eviction walks started from saved location in tree" : 1671,
"eviction worker thread active" : 4,
"eviction worker thread created" : 0,
"eviction worker thread evicting pages" : 7448,
"eviction worker thread removed" : 0,
"eviction worker thread stable number" : 0,
"files with active eviction walks" : 0,
"files with new eviction walks started" : 2069,
"force re-tuning of eviction workers once in a while" : 0,
"forced eviction - history store pages failed to evict while session has history store cursor open" : 0,
"forced eviction - history store pages selected while session has history store cursor open" : 0,
"forced eviction - history store pages successfully evicted while session has history store cursor open" : 0,
"forced eviction - pages evicted that were clean count" : 0,
"forced eviction - pages evicted that were clean time (usecs)" : 0,
"forced eviction - pages evicted that were dirty count" : 0,
"forced eviction - pages evicted that were dirty time (usecs)" : 0,
"forced eviction - pages selected because of too many deleted items count" : 0,
"forced eviction - pages selected count" : 12,
"forced eviction - pages selected unable to be evicted count" : 0,
"forced eviction - pages selected unable to be evicted time" : 0,
"forced eviction - session returned rollback error while force evicting due to being oldest" : 0,
"hazard pointer blocked page eviction" : 7,
"hazard pointer check calls" : 7460,
"hazard pointer check entries walked" : 40419,
"hazard pointer maximum array length" : 2,
"history store key truncation calls that returned restart" : 0,
"history store key truncation due to mixed timestamps" : 0,
"history store key truncation due to the key being removed from the data page" : 9,
"history store score" : 0,
"history store table insert calls" : 0,
"history store table insert calls that returned restart" : 0,
"history store table max on-disk size" : 0,
"history store table on-disk size" : 4096,
"history store table out-of-order resolved updates that lose their durable timestamp" : 0,
"history store table out-of-order updates that were fixed up by moving existing records" : 0,
"history store table out-of-order updates that were fixed up during insertion" : 0,
"history store table reads" : 0,
"history store table reads missed" : 0,
"history store table reads requiring squashed modifies" : 0,
"history store table remove calls due to key truncation" : 0,
"history store table writes requiring squashed modifies" : 0,
"in-memory page passed criteria to be split" : 24,
"in-memory page splits" : 12,
"internal pages evicted" : 0,
"internal pages queued for eviction" : 3,
"internal pages seen by eviction walk" : 1153,
"internal pages seen by eviction walk that are already queued" : 29,
"internal pages split during eviction" : 0,
"leaf pages split during eviction" : 255,
"maximum bytes configured" : 1073741824,
"maximum page size at eviction" : 0,
"modified pages evicted" : 7440,
"modified pages evicted by application threads" : 0,
"operations timed out waiting for space in cache" : 0,
"overflow pages read into cache" : 0,
"page split during eviction deepened the tree" : 0,
"page written requiring history store records" : 32,
"pages currently held in the cache" : 5169,
"pages evicted by application threads" : 0,
"pages queued for eviction" : 36108,
"pages queued for eviction post lru sorting" : 66063,
"pages queued for urgent eviction" : 2,
"pages queued for urgent eviction during walk" : 2,
"pages read into cache" : 3368,
"pages read into cache after truncate" : 1,
"pages read into cache after truncate in prepare state" : 0,
"pages requested from the cache" : 458124,
"pages seen by eviction walk" : 408760,
"pages seen by eviction walk that are already queued" : 68447,
"pages selected for eviction unable to be evicted" : 7,
"pages selected for eviction unable to be evicted as the parent page has overflow items" : 0,
"pages selected for eviction unable to be evicted because of active children on an internal page" : 0,
"pages selected for eviction unable to be evicted because of failure in reconciliation" : 0,
"pages walked for eviction" : 911582,
"pages written from cache" : 10154,
"pages written requiring in-memory restoration" : 32,
"percentage overhead" : 8,
"tracked bytes belonging to internal pages in the cache" : 1770576,
"tracked bytes belonging to leaf pages in the cache" : 192186003,
"tracked dirty bytes in the cache" : 380,
"tracked dirty pages in the cache" : 1,
"unmodified pages evicted" : 0
}