ERROR child process failed, exited with error number 51

First of all: hi everyone, my 1 topic here :slight_smile:
I have a problem because I cannot run mongo service on Linux.
I tried changing ownership of folders, checking are folders ok with config.
Removing .lock files. I dont know what else I can do. Can I ask for some feedback? Thanks.

This is my console

> [aaa@ddd mongodb]$ sudo systemctl status mongod.service
> ● mongod.service - MongoDB Database Server
>    Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
>    Active: failed (Result: exit-code) since Mon 2023-06-12 16:55:03 CEST; 3min 39s ago
>      Docs: https://docs.mongodb.org/manual
>   Process: 13729 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=51)
>   Process: 13726 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
>   Process: 13723 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
>   Process: 13720 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
>  Main PID: 1703 (code=killed, signal=ABRT)
> 
> Jun 12 16:55:03 ddd systemd[1]: Starting MongoDB Database Server...
> Jun 12 16:55:03 ddd mongod[13729]: about to fork child process, waiting until server is ready for connections.
> Jun 12 16:55:03 ddd mongod[13729]: forked process: 13732
> Jun 12 16:55:03 ddd mongod[13729]: ERROR: child process failed, exited with error number 51
> Jun 12 16:55:03 ddd mongod[13729]: To see additional information in this output, start without the "--fork" option.
> Jun 12 16:55:03 ddd systemd[1]: mongod.service: control process exited, code=exited status=51
> Jun 12 16:55:03 ddd systemd[1]: Failed to start MongoDB Database Server.
> Jun 12 16:55:03 ddd systemd[1]: Unit mongod.service entered failed state.
> Jun 12 16:55:03 ddd systemd[1]: mongod.service failed.
> [aaa@ddd mongodb]$ sudo systemctl start mongod.service
> Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.

and this is my mongod.conf:

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /mongo
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
  fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.


#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:

Issue could be your dbpath directory
Can mongod write to this directory?
Change it to some other dir say your home dir and see if mongod comes up

2 Likes

Before changing the directory, I did

sudo mongod --repair --dbpath /path/to/data/db

and my permissions seems to be broken. What is the exact permissions I should have to db? Is it 755?

And how change the directory of mongo? Should I backup my original folder or just leave it and create new folder somewhere and change only dbPath: /mongo in /etc/mongod.conf?

I am using separate volume for mongo db directory, so I need to use that.
Is there a way to just fix it? Its a prod environment. Is switching directories safe?

If it is prod do not change dbpath.You have to identify the issue and fix it
I have suggested to see if it works with new dbpath suspecting some issue with it
Is the mount point ok?
Apart from permissions you have to check ownership also.It should be owned by mongod
What has changed from working status to now?
Check your mongod.log and compare when it was working fine to now
Does /mongo have mongo related files?

2 Likes

Hi @Brian_Bell,
As suggested from the status of service, try to restart it without option fork for understand better what Is the error.

BR

1 Like

I tried other db path to run, but I had the same problem.

No I switched back to my main mongo dir and I am stuck with this situation.
Maybe you will have any idea of what to do next? I think there is a problem with lock file?

logfile:

-----  END BACKTRACE  -----
2023-06-13T21:04:54.150+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] MongoDB starting : pid=94579 port=27017 dbpath=/mongo 64-bit host=ddd
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] db version v3.6.17
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] modules: none
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] build environment:
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten]     distmod: rhel70
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-13T21:04:54.344+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-13T21:04:54.345+0200 I STORAGE  [initandlisten] exception in initAndListen: Location28596: Unable to determine status of lock file in the data directory /mongo: boost::filesystem::status: Permission denied: "/mongo/mongod.lock", terminating
2023-06-13T21:04:54.345+0200 F -        [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272
2023-06-13T21:04:54.345+0200 F -        [initandlisten]

***aborting after invariant() failure


2023-06-13T21:04:54.370+0200 F -        [initandlisten] Got signal: 6 (Aborted).

listing of few /mongo dir files (mongo.lock example):

-rwxr-xr-x. 1 mongod mongod      94208 Jun 13 20:50 _mdb_catalog.wt
drwxr-xr-x. 5 mongod mongod      16384 Jun 19  2020 mongo19.bck
-rw-------. 1 mongod mongod          0 Jun 13 20:50 mongod.lock
-rwxr-xr-x. 1 mongod mongod      65536 Jun 13 20:50 sizeStorer.wt
-rwxr-xr-x. 1 mongod mongod        114 Apr  8  2020 storage.bson
drwx------. 2 mongod mongod          6 Jun 13 20:50 _tmp
-rwxr-xr-x. 1 mongod mongod         45 Apr  8  2020 WiredTiger
-rw-------. 1 mongod mongod       4096 Jun 13 20:50 WiredTigerLAS.wt
-rwxr-xr-x. 1 mongod mongod         21 Apr  8  2020 WiredTiger.lock
-rw-------. 1 mongod mongod       1140 Jun 13 20:50 WiredTiger.turtle
-rwxr-xr-x. 1 mongod mongod     786432 Jun 13 20:50 WiredTiger.wt

and here is some command output:

[aaa@ddd tmp]$ sudo systemctl restart mongod.service
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[aaa@ddd tmp]$ sudo journalctl -xe
-- Unit mongod.service has begun starting up.
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=AVC msg=audit(1686683540.311:492556): avc:  denied  { search } for  pid=96187 comm="mongod"
Jun 13 21:12:20 ddd mongod[96187]: about to fork child process, waiting until server is ready for connections.
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686683540.311:492556): arch=c000003e syscall=2 success=no exit=-13 a0=55
Jun 13 21:12:20 ddd mongod[96187]: forked process: 96190
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=CWD msg=audit(1686683540.311:492556):  cwd="/"
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PATH msg=audit(1686683540.311:492556): item=0 name="/sys/fs/cgroup/memory/memory.limit_in_b
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686683540.311:492556): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=AVC msg=audit(1686683540.351:492557): avc:  denied  { getattr } for  pid=96190 comm="mongod
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686683540.351:492557): arch=c000003e syscall=4 success=no exit=-13 a0=55
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=CWD msg=audit(1686683540.351:492557):  cwd="/"
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PATH msg=audit(1686683540.351:492557): item=0 name="/mongo/mongod.lock" inode=67 dev=fd:02
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686683540.351:492557): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=ANOM_ABEND msg=audit(1686683540.366:492558): auid=4294967295 uid=776 gid=598 ses=4294967295
Jun 13 21:12:20 ddd mongod[96187]: ERROR: child process failed, exited with error number 51
Jun 13 21:12:20 ddd mongod[96187]: To see additional information in this output, start without the "--fork" option.
Jun 13 21:12:20 ddd systemd[1]: mongod.service: control process exited, code=exited status=51
Jun 13 21:12:20 ddd systemd[1]: Failed to start MongoDB Database Server.
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongod.service has failed.
--
-- The result is failed.
Jun 13 21:12:20 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 13 21:12:20 ddd systemd[1]: mongod.service failed.
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SERVICE_START msg=audit(1686683540.379:492559): pid=1 uid=0 auid=4294967295 ses=4294967295
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686683540.385:492560): arch=c000003e syscall=62 success=yes exit=0 a0=17
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686683540.385:492560): opid=96177 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686683540.385:492560): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686683540.386:492561): arch=c000003e syscall=62 success=yes exit=0 a0=17
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686683540.386:492561): opid=96178 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686683540.386:492561): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686683540.386:492562): arch=c000003e syscall=62 success=yes exit=0 a0=17
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686683540.386:492562): opid=96178 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686683540.386:492562): proctitle="(null)"
Jun 13 21:12:20 ddd polkitd[1028]: Unregistered Authentication Agent for unix-process:96176:453232162 (system bus name :1.51400, object path /org/fre
Jun 13 21:12:20 ddd sudo[96174]: pam_unix(sudo:session): session closed for user root
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=USER_END msg=audit(1686683540.391:492563): pid=96174 uid=0 auid=303693 ses=25537 subj=uncon
Jun 13 21:12:20 ddd audispd[988]: node=ddd type=CRED_DISP msg=audit(1686683540.393:492564): pid=96174 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:12:24 ddd audispd[988]: node=ddd type=USER_ACCT msg=audit(1686683544.047:492565): pid=96197 uid=303693 auid=303693 ses=25537 subj
Jun 13 21:12:24 ddd audispd[988]: node=ddd type=USER_CMD msg=audit(1686683544.048:492566): pid=96197 uid=303693 auid=303693 ses=25537 subj=
Jun 13 21:12:24 ddd audispd[988]: node=ddd type=CRED_REFR msg=audit(1686683544.049:492567): pid=96197 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:12:24 ddd sudo[96197]: pam_unix(sudo:session): session opened for user root by aaa(uid=0)
Jun 13 21:12:24 ddd audispd[988]: node=ddd type=USER_START msg=audit(1686683544.057:492568): pid=96197 uid=0 auid=303693 ses=25537 subj=unc
[aaa@ddd tmp]$

Hi @Brian_Bell,
Delete this mongod.lock file and restart the service.

BR

1 Like

Hi Fabio,
But I am not restarting it with fork option…

Hi @Brian_Bell,
See my previous answer

1 Like

I deleted it and got this output:

[aaa@ddd mongo]$ sudo rm -fr mongod.lock
[aaa@ddd mongo]$ sudo systemctl restart mongod.service
Job for mongod.service failed because the control process exited with error code. See "systemctl status mongod.service" and "journalctl -xe" for details.
[aaa@ddd mongo]$ sudo journalctl -xe
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=AVC msg=audit(1686684211.349:492648): avc:  denied  { search } for  pid=98411 comm="mongod"
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686684211.349:492648): arch=c000003e syscall=2 success=no exit=-13 a0=55
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=CWD msg=audit(1686684211.349:492648):  cwd="/"
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PATH msg=audit(1686684211.349:492648): item=0 name="/sys/fs/cgroup/memory/memory.limit_in_b
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686684211.349:492648): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:23:31 ddd mongod[98411]: about to fork child process, waiting until server is ready for connections.
Jun 13 21:23:31 ddd mongod[98411]: forked process: 98414
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=AVC msg=audit(1686684211.392:492649): avc:  denied  { write } for  pid=98414 comm="mongod"
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686684211.392:492649): arch=c000003e syscall=2 success=no exit=-13 a0=55
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=CWD msg=audit(1686684211.392:492649):  cwd="/"
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PATH msg=audit(1686684211.392:492649): item=0 name="/mongo/" inode=64 dev=fd:02 mode=040755
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PATH msg=audit(1686684211.392:492649): item=1 name="/mongo/mongod.lock" objtype=CREATE cap_
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686684211.392:492649): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=ANOM_ABEND msg=audit(1686684211.407:492650): auid=4294967295 uid=776 gid=598 ses=4294967295
Jun 13 21:23:31 ddd mongod[98411]: ERROR: child process failed, exited with error number 51
Jun 13 21:23:31 ddd mongod[98411]: To see additional information in this output, start without the "--fork" option.
Jun 13 21:23:31 ddd systemd[1]: mongod.service: control process exited, code=exited status=51
Jun 13 21:23:31 ddd systemd[1]: Failed to start MongoDB Database Server.
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongod.service has failed.
--
-- The result is failed.
Jun 13 21:23:31 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 13 21:23:31 ddd systemd[1]: mongod.service failed.
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SERVICE_START msg=audit(1686684211.416:492651): pid=1 uid=0 auid=4294967295 ses=4294967295
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686684211.418:492652): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686684211.418:492652): opid=98399 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686684211.418:492652): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686684211.418:492653): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686684211.418:492653): opid=98400 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686684211.418:492653): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686684211.419:492654): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686684211.419:492654): opid=98400 oauid=303693 ouid=0 oses=25537 obj=unc
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686684211.419:492654): proctitle="(null)"
Jun 13 21:23:31 ddd sudo[98397]: pam_unix(sudo:session): session closed for user root
Jun 13 21:23:31 ddd polkitd[1028]: Unregistered Authentication Agent for unix-process:98398:453299266 (system bus name :1.51413, object path /org/fre
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=USER_END msg=audit(1686684211.422:492655): pid=98397 uid=0 auid=303693 ses=25537 subj=uncon
Jun 13 21:23:31 ddd audispd[988]: node=ddd type=CRED_DISP msg=audit(1686684211.422:492656): pid=98397 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:23:41 ddd audispd[988]: node=ddd type=USER_ACCT msg=audit(1686684221.080:492657): pid=98431 uid=303693 auid=303693 ses=25537 subj
Jun 13 21:23:41 ddd audispd[988]: node=ddd type=USER_CMD msg=audit(1686684221.081:492658): pid=98431 uid=303693 auid=303693 ses=25537 subj=
Jun 13 21:23:41 ddd audispd[988]: node=ddd type=CRED_REFR msg=audit(1686684221.081:492659): pid=98431 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:23:41 ddd sudo[98431]: pam_unix(sudo:session): session opened for user root by aaa(uid=0)
Jun 13 21:23:41 ddd audispd[988]: node=ddd type=USER_START msg=audit(1686684221.087:492660): pid=98431 uid=0 auid=303693 ses=25537 subj=unc
[aaa@ddd mongo]$ sudo systemctl status mongod.service
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Tue 2023-06-13 21:23:31 CEST; 2min 2s ago
     Docs: https://docs.mongodb.org/manual
  Process: 98411 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=51)
  Process: 98410 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 98406 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 98404 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
 Main PID: 1703 (code=killed, signal=ABRT)

Jun 13 21:23:31 ddd systemd[1]: Starting MongoDB Database Server...
Jun 13 21:23:31 ddd mongod[98411]: about to fork child process, waiting until server is ready for connections.
Jun 13 21:23:31 ddd mongod[98411]: forked process: 98414
Jun 13 21:23:31 ddd mongod[98411]: ERROR: child process failed, exited with error number 51
Jun 13 21:23:31 ddd mongod[98411]: To see additional information in this output, start without the "--fork" option.
Jun 13 21:23:31 ddd systemd[1]: mongod.service: control process exited, code=exited status=51
Jun 13 21:23:31 ddd systemd[1]: Failed to start MongoDB Database Server.
Jun 13 21:23:31 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 13 21:23:31 ddd systemd[1]: mongod.service failed.
[aaa@ddd mongo]$ sudo tail /var/log/mongodb/mongod.log -n 60
 mongod(+0x2296F2D) [0x55735dd75f2d]
 libpthread.so.0(+0xF630) [0x7f2bb77f9630]
 libc.so.6(gsignal+0x37) [0x7f2bb7452387]
 libc.so.6(abort+0x148) [0x7f2bb7453a78]
 mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x55735c47affe]
 mongod(_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj+0x158) [0x55735c722878]
 mongod(_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE+0x41) [0x55735dc2af31]
 mongod(_ZN5mongo6Client20makeOperationContextEv+0x27) [0x55735dc27017]
 mongod(+0xA12863) [0x55735c4f1863]
 mongod(+0x2292BF5) [0x55735dd71bf5]
 mongod(_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE+0x364) [0x55735c47c1d7]
 mongod(_ZZN5mongo13duration_castINS_8DurationISt5ratioILl1ELl1000EEEES2_ILl1ELl1EEEET_RKNS1_IT0_EEENKUlvE_clEv+0x0) [0x55735c414cbc]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x87A) [0x55735c4f91ea]
 mongod(main+0x9) [0x55735c47d129]
 libc.so.6(__libc_start_main+0xF5) [0x7f2bb743e555]
 mongod(+0xA0235F) [0x55735c4e135f]
-----  END BACKTRACE  -----
2023-06-13T21:23:31.360+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-13T21:23:31.393+0200 I CONTROL  [initandlisten] MongoDB starting : pid=98414 port=27017 dbpath=/mongo 64-bit host=ddd
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] db version v3.6.17
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] modules: none
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] build environment:
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten]     distmod: rhel70
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-13T21:23:31.394+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-13T21:23:31.394+0200 I STORAGE  [initandlisten] exception in initAndListen: IllegalOperation: Attempted to create a lock file on a read-only directory: /mongo, terminating
2023-06-13T21:23:31.394+0200 F -        [initandlisten] Invariant failure globalStorageEngine src/mongo/db/service_context_d.cpp 272
2023-06-13T21:23:31.394+0200 F -        [initandlisten]

***aborting after invariant() failure


2023-06-13T21:23:31.408+0200 F -        [initandlisten] Got signal: 6 (Aborted).

 0x55d61a997831 0x55d61a996a49 0x55d61a996f2d 0x7f59393ad630 0x7f5939006387 0x7f5939007a78 0x55d61909bffe 0x55d619343878 0x55d61a84bf31 0x55d61a848017 0x55d619112863 0x55d61a992bf5 0x55d61909d1d7 0x55d619035cbc 0x55d61911a1ea 0x55d61909e129 0x7f5938ff2555 0x55d61910235f
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"55D618700000","o":"2297831","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55D618700000","o":"2296A49"},{"b":"55D618700000","o":"2296F2D"},{"b":"7F593939E000","o":"F630"},{"b":"7F5938FD0000","o":"36387","s":"gsignal"},{"b":"7F5938FD0000","o":"37A78","s":"abort"},{"b":"55D618700000","o":"99BFFE","s":"_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j"},{"b":"55D618700000","o":"C43878","s":"_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj"},{"b":"55D618700000","o":"214BF31","s":"_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE"},{"b":"55D618700000","o":"2148017","s":"_ZN5mongo6Client20makeOperationContextEv"},{"b":"55D618700000","o":"A12863"},{"b":"55D618700000","o":"2292BF5"},{"b":"55D618700000","o":"99D1D7","s":"_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE"},{"b":"55D618700000","o":"935CBC","s":"_ZZN5mongo13duration_castINS_8DurationISt5ratioILl1ELl1000EEEES2_ILl1ELl1EEEET_RKNS1_IT0_EEENKUlvE_clEv"},{"b":"55D618700000","o":"A1A1EA","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"55D618700000","o":"99E129","s":"main"},{"b":"7F5938FD0000","o":"22555","s":"__libc_start_main"},{"b":"55D618700000","o":"A0235F"}],"processInfo":{ "mongodbVersion" : "3.6.17", "gitVersion" : "3d6953c361213c5bfab23e51ab274ce592edafe6", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1160.81.1.el7.x86_64", "version" : "#1 SMP Thu Nov 24 12:21:22 UTC 2022", "machine" : "x86_64" }, "somap" : [ { "b" : "55D618700000", "elfType" : 3, "buildId" : "F5A6F048CC3D823882D3FDB7A4A386EE2E0BC1D4" }, { "b" : "7FFD6F784000", "elfType" : 3, "buildId" : "EFDC01C543E3027D760D04B6BFDD53C3F48C6798" }, { "b" : "7F593A5B3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "E0CD0DD5466E6B9E5FB10BFAFF13B1BB50F08EAA" }, { "b" : "7F593A150000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "622F79C1AB7612F082403F4987CE1DAC287775C3" }, { "b" : "7F5939EDE000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "7CBCB0322F585236D81B557ED95C708F98A20C33" }, { "b" : "7F5939CDA000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7F2E9CB0769D7E57BD669B485A74B537B63A57C4" }, { "b" : "7F5939AD2000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "3E44DF7055942478D052E40FDD1F5B7862B152B0" }, { "b" : "7F59397D0000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7615604EAF4A068DFAE5085444D15C0DEE93DFBD" }, { "b" : "7F59395BA000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDF51350C7F71496149D064AA8B1441F786DF88A" }, { "b" : "7F593939E000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B" }, { "b" : "7F5938FD0000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "FC4FA58E47A5ACC137EADB7689BCE4357C557A96" }, { "b" : "7F593A7CD000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "62C449974331341BB08DCCE3859560A22AF1E172" }, { "b" : "7F5938DBA000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "E69C3975164331DF84F4E8955CC3F7A0836B05D0" }, { "b" : "7F5938B6D000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0CAEC124D97114DA40DDEB0FED1FAD5D14C3D626" }, { "b" : "7F5938884000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "52C5B6279DA9CD210E5D58D1B1E0E080E5C8B232" }, { "b" : "7F5938680000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "E4C7298B74FEEADC4DDE40CDD8C4D6B85FE09ADE" }, { "b" : "7F593844D000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5FF9D1075A8D5D62F77F5CE56C935FCD92C62EFA" }, { "b" : "7F593823D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "779381063DAECC27E8480C8F79F0651162586478" }, { "b" : "7F5938039000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "8CA73C16CFEB9A8B5660015B9223B09F87041CAD" }, { "b" : "7F5937E12000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "805AB866A4573EFEC4D8EA95123E8349B2B9D349" }, { "b" : "7F5937BB0000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "F5B144F9F5D9BE451C80211B34DB2CE348E039B6" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55d61a997831]
 mongod(+0x2296A49) [0x55d61a996a49]
 mongod(+0x2296F2D) [0x55d61a996f2d]
 libpthread.so.0(+0xF630) [0x7f59393ad630]
 libc.so.6(gsignal+0x37) [0x7f5939006387]
 libc.so.6(abort+0x148) [0x7f5939007a78]
 mongod(_ZN5mongo22invariantFailedWithMsgEPKcS1_S1_j+0x0) [0x55d61909bffe]
 mongod(_ZN5mongo20ServiceContextMongoD9_newOpCtxEPNS_6ClientEj+0x158) [0x55d619343878]
 mongod(_ZN5mongo14ServiceContext20makeOperationContextEPNS_6ClientE+0x41) [0x55d61a84bf31]
 mongod(_ZN5mongo6Client20makeOperationContextEv+0x27) [0x55d61a848017]
 mongod(+0xA12863) [0x55d619112863]
 mongod(+0x2292BF5) [0x55d61a992bf5]
 mongod(_ZN5mongo8shutdownENS_8ExitCodeERKNS_16ShutdownTaskArgsE+0x364) [0x55d61909d1d7]
 mongod(_ZZN5mongo13duration_castINS_8DurationISt5ratioILl1ELl1000EEEES2_ILl1ELl1EEEET_RKNS1_IT0_EEENKUlvE_clEv+0x0) [0x55d619035cbc]
 mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x87A) [0x55d61911a1ea]
 mongod(main+0x9) [0x55d61909e129]
 libc.so.6(__libc_start_main+0xF5) [0x7f5938ff2555]
 mongod(+0xA0235F) [0x55d61910235f]
-----  END BACKTRACE  -----

Hi @Brian_Bell ,
Commet the fork option in the config file and restart the service, then paste the output of systemctl status mongod & journalctl -xe

BR

1 Like

@Fabio_Ramohitaj thanks for helping me out

I did what you say, I guess there is no mongod process runinng (should there be one?)

my output:

[aaa@ddd mongo]$ cat /etc/mongod.conf
# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# Where and how to store data.
storage:
  dbPath: /mongo
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# how the process runs
processManagement:
# fork: true  # fork and run in background
  pidFilePath: /var/run/mongodb/mongod.pid  # location of pidfile
  timeZoneInfo: /usr/share/zoneinfo

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.


#security:

#operationProfiling:

#replication:

#sharding:

## Enterprise-Only Options

#auditLog:

#snmp:
[aaa@ddd mongo]$ sudo systemctl restart mongod.service
Job for mongod.service failed because a fatal signal was delivered to the control process. See "systemctl status mongod.service" and "journalctl -xe" for details.
[aaa@ddd mongo]$ sudo systemctl status mongod.service
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: signal) since Tue 2023-06-13 21:40:38 CEST; 9s ago
     Docs: https://docs.mongodb.org/manual
  Process: 101600 ExecStart=/usr/bin/mongod $OPTIONS (code=killed, signal=ABRT)
  Process: 101596 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 101594 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 101592 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)
 Main PID: 1703 (code=killed, signal=ABRT)

Jun 13 21:40:38 ddd systemd[1]: Starting MongoDB Database Server...
Jun 13 21:40:38 ddd systemd[1]: mongod.service: control process exited, code=killed status=6
Jun 13 21:40:38 ddd systemd[1]: Failed to start MongoDB Database Server.
Jun 13 21:40:38 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 13 21:40:38 ddd systemd[1]: mongod.service failed.
[aaa@ddd mongo]$ sudo journalctl -xe
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PATH msg=audit(1686685238.661:492779): item=0 name="/sys/fs/cgroup/memory/memory.limit_in_b
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685238.661:492779): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=AVC msg=audit(1686685238.697:492780): avc:  denied  { write } for  pid=101600 comm="mongod"
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685238.697:492780): arch=c000003e syscall=2 success=no exit=-13 a0=55
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=CWD msg=audit(1686685238.697:492780):  cwd="/"
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PATH msg=audit(1686685238.697:492780): item=0 name="/mongo/" inode=64 dev=fd:02 mode=040755
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PATH msg=audit(1686685238.697:492780): item=1 name="/mongo/mongod.lock" objtype=CREATE cap_
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685238.697:492780): proctitle=2F7573722F62696E2F6D6F6E676F64002D660
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=ANOM_ABEND msg=audit(1686685238.711:492781): auid=4294967295 uid=776 gid=598 ses=4294967295
Jun 13 21:40:38 ddd systemd[1]: mongod.service: control process exited, code=killed status=6
Jun 13 21:40:38 ddd systemd[1]: Failed to start MongoDB Database Server.
-- Subject: Unit mongod.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit mongod.service has failed.
--
-- The result is failed.
Jun 13 21:40:38 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 13 21:40:38 ddd systemd[1]: mongod.service failed.
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=SERVICE_START msg=audit(1686685238.720:492782): pid=1 uid=0 auid=4294967295 ses=4294967295
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685238.724:492783): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686685238.724:492783): opid=101587 oauid=303693 ouid=0 oses=25537 obj=un
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685238.724:492783): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:40:38 ddd polkitd[1028]: Unregistered Authentication Agent for unix-process:101586:453401998 (system bus name :1.51432, object path /org/fr
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685238.725:492784): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686685238.725:492784): opid=101588 oauid=303693 ouid=0 oses=25537 obj=un
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685238.725:492784): proctitle=73797374656D63746C0072657374617274006
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685238.725:492785): arch=c000003e syscall=62 success=yes exit=0 a0=18
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=OBJ_PID msg=audit(1686685238.725:492785): opid=101588 oauid=303693 ouid=0 oses=25537 obj=un
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685238.725:492785): proctitle="(null)"
Jun 13 21:40:38 ddd sudo[101584]: pam_unix(sudo:session): session closed for user root
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=USER_END msg=audit(1686685238.728:492786): pid=101584 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:40:38 ddd audispd[988]: node=ddd type=CRED_DISP msg=audit(1686685238.728:492787): pid=101584 uid=0 auid=303693 ses=25537 subj=unc
Jun 13 21:40:44 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685244.085:492788): arch=c000003e syscall=159 success=yes exit=0 a0=7
Jun 13 21:40:44 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685244.085:492788): proctitle="/usr/bin/vmtoolsd"
Jun 13 21:40:44 ddd audispd[988]: node=ddd type=SYSCALL msg=audit(1686685244.085:492789): arch=c000003e syscall=159 success=yes exit=0 a0=7
Jun 13 21:40:44 ddd audispd[988]: node=ddd type=PROCTITLE msg=audit(1686685244.085:492789): proctitle="/usr/bin/vmtoolsd"
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=USER_ACCT msg=audit(1686685248.074:492790): pid=101614 uid=303693 auid=303693 ses=25537 sub
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=USER_CMD msg=audit(1686685248.075:492791): pid=101614 uid=303693 auid=303693 ses=25537 subj
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=CRED_REFR msg=audit(1686685248.075:492792): pid=101614 uid=0 auid=303693 ses=25537 subj=unc
Jun 13 21:40:48 ddd sudo[101614]: pam_unix(sudo:session): session opened for user root by aaa(uid=0)
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=USER_START msg=audit(1686685248.083:492793): pid=101614 uid=0 auid=303693 ses=25537 subj=un
Jun 13 21:40:48 ddd sudo[101614]: pam_unix(sudo:session): session closed for user root
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=USER_END msg=audit(1686685248.097:492794): pid=101614 uid=0 auid=303693 ses=25537 subj=unco
Jun 13 21:40:48 ddd audispd[988]: node=ddd type=CRED_DISP msg=audit(1686685248.097:492795): pid=101614 uid=0 auid=303693 ses=25537 subj=unc
[aaa@ddd mongo]$ sudo lsof -i:27017
[aaa@ddd mongo]$

Hi @Brian_Bell ,
As suggested from @Ramachandra_Tummala ,
Verify the permission of dbpath, so:
ls -latrh /
Paste the result of directory mongo here.

BR

1 Like

I shorted the long list of similar files with […]
These permission change when I run some typical commands. He is the output:

[ddd@ddd mongo]$ ls -latrh /mongo
total 4.2G
-rwxr-xr-x.  1 mongod mongod   21 Apr  8  2020 WiredTiger.lock
-rwxr-xr-x.  1 mongod mongod   45 Apr  8  2020 WiredTiger
-rwxr-xr-x.  1 mongod mongod  114 Apr  8  2020 storage.bson
drwxr-xr-x.  5 mongod mongod  16K Jun 19  2020 mongo19.bck
dr-xr-xr-x. 18 root   root    276 Apr 18 10:08 ..
drwxr-xr-x.  2 mongod mongod 4.0K Jun 12 16:54 diagnostic.data
drwxr-xr-x.  4 mongod mongod 4.0K Jun 13 00:00 backups
-rw-------.  1 mongod mongod  16K Jun 13 20:49 index-0-6233074057024477686.wt
-rw-------.  1 mongod mongod  16K Jun 13 20:49 index-1-6233074057024477686.wt
[...]
-rw-------.  1 mongod mongod 292K Jun 13 20:50 index-196-6233074057024477686.wt
drwx------.  2 mongod mongod    6 Jun 13 20:50 _tmp
-rw-------.  1 mongod mongod  11M Jun 13 20:50 index-197-6233074057024477686.wt
[...]
-rw-------.  1 mongod mongod  16K Jun 13 20:50 index-236-6233074057024477686.wt
-rwxr-xr-x.  1 mongod mongod  32K Jun 13 20:50 collection-7-8546731449771946358.wt
[...]
-rwxr-xr-x.  1 mongod mongod  36K Jun 13 20:50 collection-133-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod 4.0K Jun 13 20:50 collection-121-8546731449771946358.wt
-rw-------.  1 mongod mongod 4.0K Jun 13 20:50 WiredTigerLAS.wt
-rwxr-xr-x.  1 mongod mongod  64K Jun 13 20:50 sizeStorer.wt
-rwxr-xr-x.  1 mongod mongod  92K Jun 13 20:50 _mdb_catalog.wt
-rwxr-xr-x.  1 mongod mongod 6.3M Jun 13 20:50 collection-96-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  16K Jun 13 20:50 collection-80-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod 124K Jun 13 20:50 collection-73-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  44K Jun 13 20:50 collection-52-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod 4.0K Jun 13 20:50 collection-37-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  16K Jun 13 20:50 collection-144-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  32K Jun 13 20:50 collection-141-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  76K Jun 13 20:50 collection-137-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod  52K Jun 13 20:50 collection-114-8546731449771946358.wt
-rwxr-xr-x.  1 mongod mongod 4.0K Jun 13 20:50 collection-104-8546731449771946358.wt
-rw-------.  1 mongod mongod 1.2K Jun 13 20:50 WiredTiger.turtle
-rwxr-xr-x.  1 mongod mongod 768K Jun 13 20:50 WiredTiger.wt
drwxr-xr-x.  6 mongod mongod  20K Jun 13 21:23 .

Hi @Brian_Bell,
The root directory of dbpath have the same permission?

What typical commands you ran?
Permissions will not change unless you tried to start mongod directly as root
You should not start mongod as root.Use systemctl.Internally it will call mongod
Definitely there is some issue with your dbpath permissions
From the logs you pasted it clearly says unable to read lock file at one time and /mongo is a read only file system at another time?

Which root directory?
What i pasted is main directory of mongodb, its /mongo volume in my case.

I run mongo service with sudo, because when I try to run it without sudo, that it ask me for root password.
I think this might be the problem here.

After server rebooting there is no file permission problem abymore, but I still cannot run service.
Looks like connection issue?

here is my output, please take a look:


[aaa@ddd ~]$ sudo tail -n 60 /var/log/mongodb/mongod.log
2023-06-14T08:24:57.356+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2023-06-14T08:24:57.356+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2023-06-14T08:24:57.356+0200 I CONTROL  [initandlisten]
2023-06-14T08:24:57.376+0200 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/diagnostic.data'
2023-06-14T08:24:57.377+0200 I NETWORK  [initandlisten] listening via socket bound to 0.0.0.0
2023-06-14T08:24:57.377+0200 I NETWORK  [initandlisten] listening via socket bound to /tmp/mongodb-27017.sock
2023-06-14T08:24:57.377+0200 I NETWORK  [initandlisten] waiting for connections on port 27017
2023-06-14T08:24:58.592+0200 I NETWORK  [listener] connection accepted from 10.150.18.158:49367 #1 (1 connection now open)
2023-06-14T08:24:58.676+0200 I NETWORK  [conn1] received client metadata from 10.150.18.158:49367 conn1: { driver: { name: "mongo-java-driver|legacy", version: "3.10.2" }, os: { type: "Windows", name: "Windows Server 2012 R2", architecture: "amd64", version: "6.3" }, platform: "Java/Oracle Corporation/1.8.0_211-b12" }
2023-06-14T08:26:26.093+0200 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2023-06-14T08:26:26.094+0200 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2023-06-14T08:26:26.094+0200 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2023-06-14T08:26:26.096+0200 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2023-06-14T08:26:26.098+0200 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2023-06-14T08:26:26.248+0200 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2023-06-14T08:26:26.248+0200 I CONTROL  [signalProcessingThread] now exiting
2023-06-14T08:26:26.248+0200 I CONTROL  [signalProcessingThread] shutting down with code:0
2023-06-14T08:37:38.057+0200 I CONTROL  [main] ***** SERVER RESTARTED *****
2023-06-14T08:37:38.087+0200 I CONTROL  [initandlisten] MongoDB starting : pid=13496 port=27017 dbpath=/mongo 64-bit host=ddd
2023-06-14T08:37:38.087+0200 I CONTROL  [initandlisten] db version v3.6.17
2023-06-14T08:37:38.087+0200 I CONTROL  [initandlisten] git version: 3d6953c361213c5bfab23e51ab274ce592edafe6
2023-06-14T08:37:38.087+0200 I CONTROL  [initandlisten] OpenSSL version: OpenSSL 1.0.1e-fips 11 Feb 2013
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten] allocator: tcmalloc
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten] modules: none
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten] build environment:
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten]     distmod: rhel70
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten]     distarch: x86_64
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten]     target_arch: x86_64
2023-06-14T08:37:38.088+0200 I CONTROL  [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "0.0.0.0", port: 27017 }, processManagement: { pidFilePath: "/var/run/mongodb/mongod.pid", timeZoneInfo: "/usr/share/zoneinfo" }, storage: { dbPath: "/mongo", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } }
2023-06-14T08:37:38.089+0200 I -        [initandlisten] Detected data files in /mongo created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2023-06-14T08:37:38.089+0200 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=1373M,cache_overflow=(file_max=0M),session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),compatibility=(release="3.0",require_max="3.0"),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),
2023-06-14T08:37:39.057+0200 I STORAGE  [initandlisten] WiredTiger message [1686724659:57091][13496:0x7fe38e1abb80], txn-recover: Main recovery loop: starting at 115/6784
2023-06-14T08:37:39.199+0200 I STORAGE  [initandlisten] WiredTiger message [1686724659:199445][13496:0x7fe38e1abb80], txn-recover: Recovering log 115 through 116
2023-06-14T08:37:39.299+0200 I STORAGE  [initandlisten] WiredTiger message [1686724659:299223][13496:0x7fe38e1abb80], txn-recover: Recovering log 116 through 116
2023-06-14T08:37:39.371+0200 I STORAGE  [initandlisten] WiredTiger message [1686724659:371203][13496:0x7fe38e1abb80], txn-recover: Set global recovery timestamp: 0
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten]
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten]
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten]
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten]
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2023-06-14T08:37:39.402+0200 I CONTROL  [initandlisten]
2023-06-14T08:37:39.422+0200 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/mongo/diagnostic.data'
2023-06-14T08:37:39.423+0200 I NETWORK  [initandlisten] listening via socket bound to 0.0.0.0
2023-06-14T08:37:39.423+0200 I NETWORK  [initandlisten] listening via socket bound to /tmp/mongodb-27017.sock
2023-06-14T08:37:39.423+0200 I NETWORK  [initandlisten] waiting for connections on port 27017
2023-06-14T08:37:41.435+0200 I NETWORK  [listener] connection accepted from 10.150.18.158:49452 #1 (1 connection now open)
2023-06-14T08:37:41.435+0200 I NETWORK  [conn1] received client metadata from 10.150.18.158:49452 conn1: { driver: { name: "mongo-java-driver|legacy", version: "3.10.2" }, os: { type: "Windows", name: "Windows Server 2012 R2", architecture: "amd64", version: "6.3" }, platform: "Java/Oracle Corporation/1.8.0_211-b12" }
2023-06-14T08:39:08.094+0200 I CONTROL  [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2023-06-14T08:39:08.094+0200 I NETWORK  [signalProcessingThread] shutdown: going to close listening sockets...
2023-06-14T08:39:08.094+0200 I NETWORK  [signalProcessingThread] removing socket file: /tmp/mongodb-27017.sock
2023-06-14T08:39:08.096+0200 I FTDC     [signalProcessingThread] Shutting down full-time diagnostic data capture
2023-06-14T08:39:08.097+0200 I STORAGE  [signalProcessingThread] WiredTigerKVEngine shutting down
2023-06-14T08:39:08.126+0200 I STORAGE  [signalProcessingThread] shutdown: removing fs lock...
2023-06-14T08:39:08.126+0200 I CONTROL  [signalProcessingThread] now exiting
2023-06-14T08:39:08.126+0200 I CONTROL  [signalProcessingThread] shutting down with code:0
[aaa@ddd ~]$ sudo systemctl status mongod
● mongod.service - MongoDB Database Server
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: timeout) since Wed 2023-06-14 08:39:08 CEST; 5min ago
     Docs: https://docs.mongodb.org/manual
  Process: 13496 ExecStart=/usr/bin/mongod $OPTIONS (code=exited, status=0/SUCCESS)
  Process: 13493 ExecStartPre=/usr/bin/chmod 0755 /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 13490 ExecStartPre=/usr/bin/chown mongod:mongod /var/run/mongodb (code=exited, status=0/SUCCESS)
  Process: 13488 ExecStartPre=/usr/bin/mkdir -p /var/run/mongodb (code=exited, status=0/SUCCESS)

Jun 14 08:37:37 ddd systemd[1]: Starting MongoDB Database Server...
Jun 14 08:39:08 ddd systemd[1]: mongod.service start operation timed out. Terminating.
Jun 14 08:39:08 ddd systemd[1]: Failed to start MongoDB Database Server.
Jun 14 08:39:08 ddd systemd[1]: Unit mongod.service entered failed state.
Jun 14 08:39:08 ddd systemd[1]: mongod.service failed.
[aaa@ddd ~]$

Yes it came up but getting terminated by signal 15 graceful shutdown
You have to identify if anyone or process issuing this
Check your system logs,/var/adm/messages etc

Ok thank you.
Should my fork be still commented in config file?

edit:
I deleted fork comment, restarted server on application side and it worked :slight_smile:

1 Like