Mongod crashes after update to Katello 3.7

Support
1

Problem:
Mongod crashes after update to TFM 1.18 & Katello 3.7

Expected outcome:
Running Mongod.
Foreman and Proxy versions:
foreman-1.18.0-2.el7.noarch
foreman-proxy-1.18.0-1.el7.noarch
katello-3.7.0-4.el7.noarch

Foreman and Proxy plugin versions:

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
Mongod crashes when starting a repo sync with the following error:
Aug 03 08:15:27 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [thread1] waiting for connections on port 27017
Aug 03 08:15:27 spfy-tfm.prd.sapify.ch mongodb-scl-helper[5099]: child process started successfully, parent exiting
Aug 03 08:15:27 spfy-tfm.prd.sapify.ch systemd[1]: Started High-performance, schema-free document-oriented database.
Aug 03 08:15:28 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
Aug 03 08:16:18 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn9] received client metadata from 127.0.0.1:43912 conn9: { application: { name: “MongoDB Shell” }, driver: { name: “MongoDB Internal Client”, version: “3.4.9” }, os: { type: “Linux”, name: "CentOS Linux release 7.5.1804 (Core) ", architecture: “x86_64”, version: “Kernel 3.10.0-862.9.1.el7.x86_64” } }
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn67] Fatal Assertion 17441 at src/mongo/db/storage/mmap_v1/record_store_v1_base.cpp 282
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn67]

                                                       ***aborting after fassert() failure

Aug 03 08:24:08 spfy-tfm.prd.sapify.ch systemd[1]: rh-mongodb34-mongod.service: main process exited, code=killed, status=6/ABRT
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch systemd[1]: Unit rh-mongodb34-mongod.service entered failed state.
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch systemd[1]: rh-mongodb34-mongod.service failed.
[root@spfy-tfm foreman]# journalctl -e
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn67] Fatal Assertion 17441 at src/mongo/db/storage/mmap_v1/record_store_v1_base.cpp 282
Aug 03 08:24:08 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn67]

                                                       ***aborting after fassert() failure

Aug 03 08:24:08 spfy-tfm.prd.sapify.ch mongod.27017[5104]: [conn67] Got signal: 6 (Aborted).

                                                        0x55ae8e28f35a 0x55ae8e28e699 0x55ae8e28eb66 0x7f5a6c2e86d0 0x7f5a6bf42277 0x7f5a6bf43968 0x55ae8d4ef45
                                                       ----- BEGIN BACKTRACE -----
                                                       {"backtrace":[{"b":"55AE8D2EC000","o":"FA335A"},{"b":"55AE8D2EC000","o":"FA2699"},{"b":"55AE8D2EC000","o
                                                        mongod(+0xFA335A) [0x55ae8e28f35a]
                                                        mongod(+0xFA2699) [0x55ae8e28e699]
                                                        mongod(+0xFA2B66) [0x55ae8e28eb66]
                                                        libpthread.so.0(+0xF6D0) [0x7f5a6c2e86d0]
                                                        libc.so.6(gsignal+0x37) [0x7f5a6bf42277]
                                                        libc.so.6(abort+0x148) [0x7f5a6bf43968]
                                                        mongod(+0x20345B) [0x55ae8d4ef45b]
                                                        mongod(+0xC51706) [0x55ae8df3d706]
                                                        mongod(+0xC51735) [0x55ae8df3d735]
                                                        mongod(+0xC6612D) [0x55ae8df5212d]
                                                        mongod(+0xC66407) [0x55ae8df52407]
                                                        mongod(+0x5DF688) [0x55ae8d8cb688]
                                                        mongod(+0x582F5E) [0x55ae8d86ef5e]
                                                        mongod(+0x5A5453) [0x55ae8d891453]
                                                        mongod(+0x59CDD1) [0x55ae8d888dd1]
                                                        mongod(+0x5A5453) [0x55ae8d891453]
                                                        mongod(+0x57B5B2) [0x55ae8d8675b2]
                                                        mongod(+0x5A5453) [0x55ae8d891453]
                                                        mongod(+0x8B809B) [0x55ae8dba409b]
                                                        mongod(+0x8B899B) [0x55ae8dba499b]
                                                        mongod(+0x8B8ABD) [0x55ae8dba4abd]
                                                        mongod(+0x456357) [0x55ae8d742357]
                                                        mongod(+0x45FCE6) [0x55ae8d74bce6]
                                                        mongod(+0x4613CA) [0x55ae8d74d3ca]
                                                        mongod(+0xA92D6D) [0x55ae8dd7ed6d]
                                                        mongod(+0x68C67E) [0x55ae8d97867e]
                                                        mongod(+0x68E554) [0x55ae8d97a554]
                                                        mongod(+0x270BD5) [0x55ae8d55cbd5]
                                                        mongod(+0x27150D) [0x55ae8d55d50d]
                                                        mongod(+0xF051F4) [0x55ae8e1f11f4]
                                                        libpthread.so.0(+0x7E25) [0x7f5a6c2e0e25]
                                                        libc.so.6(clone+0x6D) [0x7f5a6c00abad]
                                                       -----  END BACKTRACE  -----

Any Idea why this happen?
Thanks for any help.
Arsene

2

Does this happen every time you restart the service?

3

When I restart the service it runs and as soon a repo sync start it crash.

4

Googling around, the Mongo Jira reports such as [1] seem to indicate a possible disk corruption. Some users reported trying a mongod --repair to fix the issue. The ticket contains some information around how to try to restore from a backup if you have one.

[1] https://jira.mongodb.org/browse/SERVER-31340

5

I did try the --repair several time but it didn’t help since the repair itself crashed.
Finally i did revert the system to a previous running snapshot with TFM 1.17 and Katello 3.6 and set selinux to permissive and then replayed the upgrade and since then it works.

Could this problem be related to selinux?

6

After several day running smoothly i’ve got this problem again.
I unable to repair the mongodb using either mongodump --repair or mongod --repair.
Mongodb will start but when sync a repo it will fail after a while with the error above.

Is there a method to completely recreate the mongodb?

Thanks,
arsene

7

Any idea about this problem?
After another restore it did run for several day and today it crashed again with the same failure.
Any idea how to debug that issue ?

Thanks,
Arsene