We have Katello 2.4 running on CentOS 7.2.
The VMWare host that this is running on had some issue that stopped the
host without giving it the chance to properly shudown and on reboot,
mongodb fails to start, which causes pulp to fail.
The rest of the system functions and you can login to the gui and browse
hosts, but clicking on anything that uses pulp like the repository views
will timeout.
I'm not sure how to debug mongodb. Are there any tools to fix it?
here is the error log when you try to restart the service :
2016-05-10T20:53:28.410-0700 ***** SERVER RESTARTED *****
2016-05-10T20:53:28.414-0700 [initandlisten] MongoDB starting : pid=5979
port=27017 dbpath=/var/lib/mongodb 64-bit host=katello1.mydomain.net
2016-05-10T20:53:28.414-0700 [initandlisten] db version v2.6.11
2016-05-10T20:53:28.414-0700 [initandlisten] git version: nogitversion
2016-05-10T20:53:28.414-0700 [initandlisten] OpenSSL version: OpenSSL
1.0.1e-fips 11 Feb 2013
2016-05-10T20:53:28.414-0700 [initandlisten] build info: Linux
buildvm-08.phx2.fedoraproject.org 4.1.4-100.fc21.x86_64 #1 SMP Tue Aug 4
03:25:05 UTC 2015 x86_64 BOOST_LIB_VERSION=1_53
2016-05-10T20:53:28.414-0700 [initandlisten] allocator: tcmalloc
2016-05-10T20:53:28.414-0700 [initandlisten] options: { command: [ "run" ],
config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" },
processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid"
}, security: { authorization: "disabled" }, storage: { dbPath:
"/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination:
"file", logAppend: true, path: "/var/log/mongodb/mongodb.log", quiet: true
} }
2016-05-10T20:53:28.414-0700 [initandlisten]
2016-05-10T20:53:28.414-0700 [initandlisten] ** WARNING: Readahead for
/var/lib/mongodb is set to 4096KB
2016-05-10T20:53:28.414-0700 [initandlisten] ** We suggest setting
it to 256KB (512 sectors) or less
2016-05-10T20:53:28.414-0700 [initandlisten] **
http://dochub.mongodb.org/core/readahead
2016-05-10T20:53:28.484-0700 [initandlisten] journal
dir=/var/lib/mongodb/journal
2016-05-10T20:53:28.484-0700 [initandlisten] recover : no journal files
present, no recovery needed
2016-05-10T20:53:36.576-0700 [initandlisten] preallocateIsFaster=true 19.96
2016-05-10T20:53:43.089-0700 [initandlisten] preallocateIsFaster=true 6.98
2016-05-10T20:53:50.913-0700 [initandlisten] preallocateIsFaster=true 5.04
2016-05-10T20:53:50.913-0700 [initandlisten] preallocateIsFaster check took
22.429 secs
2016-05-10T20:53:50.913-0700 [initandlisten] preallocating a journal file
/var/lib/mongodb/journal/prealloc.0
2016-05-10T20:54:31.268-0700 [initandlisten] preallocating a journal file
/var/lib/mongodb/journal/prealloc.1
2016-05-10T20:55:12.110-0700 [initandlisten] preallocating a journal file
/var/lib/mongodb/journal/prealloc.2
2016-05-10T20:56:28.545-0700 [signalProcessingThread] got signal 15
(Terminated), will terminate after current cmd ends
2016-05-10T20:56:28.545-0700 [signalProcessingThread] now exiting
2016-05-10T20:56:28.545-0700 [signalProcessingThread] dbexit:
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: going to
close listening sockets…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: going to
flush diaglog…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: going to
close sockets…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: waiting for
fs preallocator…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: lock for
final commit…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: final
commit…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] shutdown: closing all
files…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] closeAllFiles()
finished
2016-05-10T20:56:28.545-0700 [signalProcessingThread] journalCleanup…
2016-05-10T20:56:28.545-0700 [signalProcessingThread] removeJournalFiles
2016-05-10T20:56:37.120-0700 [signalProcessingThread] shutdown: removing fs
lock…
2016-05-10T20:56:37.121-0700 [signalProcessingThread] dbexit: really
exiting now