Dnf db errors since upgrading

Support
1

Problem:
dnf failures since upgrading Foreman software

Expected outcome:

Foreman and Proxy versions:
3.15

Hi all
We are getting dnf failures due to corrupt databases ever since upgrading Foreman and our 4 proxies last month
Noticed this when patching but also see in the logs when the default dnf makecache timer units runs hourly

To fix it, we’ve had to clear the lockfiles and rebuild with

rm -f /var/lib/rpm__db* ; rpm --rebuilddb ; dnf clean all

then it works. Trying to find root cause isnt so easy, but would like to know if possibly linked to the foreman upgrades last month, because that’s when it started

2

sorry the errors are:

dnf[2220657]: error: cannot open Packages index using db5 - (-30973)
dnf[2220657]: error: cannot open Packages database in /var/lib/rpm
dnf[2221681]: error: rpmdb: BDB0113 Thread/process 76840/140317474207616 failed: BDB1507 Thread died in Berkeley DB library
dnf[2221681]: error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

3

also worth mentioning that this only occurs on a handful of servers, not all

4

no replies so presume it’s not foreman upgrade (and katello db fix) related and down to a coincidence in timing for that. I’ve since disabled all systemd makecache timer units - as maybe there was a process clash. Very weird how this only started after the Foreman upgrade though