Package/errata info disappears after a while on Content hosts page/Foreman 2.1.2

matemikulic · September 27, 2020, 10:23am

Problem:
I have a fresh installation of Foreman 2.1.2/Katello 3.16. Which was installed with options:

foreman-installer --scenario katello   --enable-foreman-plugin-chef   --enable-foreman-plugin-tasks   --enable-foreman-proxy-plugin-chef --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh --tuning medium

On Content host page where installable errata should be seen and upgradeable packages, all show 0.

If I run “katello-package-upload -f” I can refresh the Content hosts page and have package and errata info shown, but in a couple of hours(I’ve checked ~16 hours after, could be sooner), Content hosts page will again have all the 0’s on upgradable packages and installable errata.

But when I do a search for example for installed_package_name the search works, I can get package info although Content hosts page shows 0. I’ve tried a package that all servers will have and truly I got the number of total number of servers.

I don’t use puppet but subscription-manager. I have around 9300 servers.

To ease the load on DB i have additional filters for facts I don’t have use for.
Exclude pattern for facts:

[ lo, en*v*, usb*, vnet*, macvtap*, _vdsmdummy_, veth*, docker*, tap*, qbr*, qvb*, qvo*, qr-*, qg-*, vlinuxbr*, vovsbr*, proc_cpuinfo*, lscpu*, dmi.baseboard*, dmi.bios*, dmi.chasis*, dmi.memory*, dmi.processor*, cpu* ]

Ignore interfaces:

[ lo, en*v*, usb*, vnet*, macvtap*, _vdsmdummy_, veth*, docker*, tap*, qbr*, qvb*, qvo*, qr-*, qg-*, vlinuxbr*, vovsbr*, cali*, kube*, nodelocal*, dummy* ]

Expected outcome:
For content host page to constantly show applicable errata and installable packages.

Foreman and Proxy versions:
Foreman 2.12

Foreman and Proxy plugin versions:
Katello 3.16

Distribution and version:
CentOS 7

Other relevant data:
I’m not seeing any errors in production.log.
From warnings this is all I see:

2020-09-27T10:00:07 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-09-27T10:00:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2020-09-27T10:00:08 [W|app|] Scoped order is ignored, it's forced to be batch order.
2020-09-27T10:00:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-09-27T10:00:08 [W|app|] Scoped order is ignored, it's forced to be batch order.
2020-09-27T10:00:08 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2020-09-27T10:06:00 [W|app|] Scoped order is ignored, it's forced to be batch order.

matemikulic · September 28, 2020, 11:12am

It seems that errata is not calculated well. Foreman says errata is installable on host, but when yum update-minimal is ran for that advisory host itself, it shows no packages need to be upgraded. So it seems that even package info is not there to be searched properly. This reminds me of this bug from 2.0 Could not calculate errata status /no packake info/ Foreman 2.0/Katello 3.15

matemikulic · September 28, 2020, 1:11pm

Found a bunch of tasks for “Bulk generate applicability for hosts” with error:

ActiveRecord::ConnectionTimeoutError: could not obtain a connection from the pool within 5.000 seconds (waited 5.000 seconds); all pooled connections were in use

Since then I’ve changed pool to 16 in database.yml and restarted foreman to apply this. This task are 10 days old, when pool connection was still at 5. Could it be that these are causing this. Can I force cancel them as regular cancel does not seem to work?

matemikulic · September 30, 2020, 9:30am

I’ve force unlocked those tasks, no issue with DB. This was unrelated to my original problem. That still persits. Upgradeable packages and applicable errata still reverts to all zeros on content hosts page.

Jonathon_Turel · September 30, 2020, 8:48pm

@jjeffers this seems reminiscent of Bug #30636: Errata status is not update after applicability regeneration (pulp3) - Katello - Foreman can you say one way or another?

jjeffers · October 1, 2020, 11:15am

It appears to be similar. If we look at the content host in the console host, and invoke .update_errata_status, does the displayed errata status display?

matemikulic · October 1, 2020, 12:13pm

Thank you both for looking into this.
@jjeffers When content hosts page shows all 0’s and I click on hostname -> errata -> recalculate, upgradeable packages and errata will appear for that host.
If I go and then apply that errata to that host, it will not fully recalculate errata, it will show some still applicable,although all has been applied. If I then click on recalculate again, nothing will happen, it will still show applicable some errata applicable although all has been applied.

I assume this is what you meant. If you need me to do some other tests, please give me detailed instructions.

I also found some similarities with this bug as well Bug #30427: Host errata status not refresh after CV publish/promote - Katello - Foreman

Thanks again for looking into this. Much appreciated.

Jonathon_Turel · October 2, 2020, 8:13pm

I’m stumped! My only recommendation would be to upgrade to 3.16.1 when it becomes available and see if gets resolved. @iballou does this look like anything you’re aware of landing in the 3.16.1 release?

iballou · October 5, 2020, 2:01pm

@matemikulic can you find out if you’re on Pulp 2 or Pulp 3 (Pulpcore) for yum content? You can find this info under your primary smart proxy’s “Services” tab. I’m looking for “yum” to be under Pulp or Pulpcore.

I’ve seen people having applicability bugs recently but they’ve all been Pulp 2 cases. Our Pulp 3 yum applicability implementation is pretty new however so you might’ve stumbled on a bug.

Katello 3.16.1.2 will be released shortly (aiming for within the next couple of days) so I would be curious to see if the issue was resolved.

matemikulic · October 5, 2020, 2:09pm

Hey, tnx for checking this out. This is info for my main katello server. This was a fresh 2.1 installation. I didn’t upgrade from 2.0. All my servers are CentOS 7, so yum content is the only one in use.

### Chef

**Version**

0.2.0

### Dynflow

**Version**

0.2.4

### HTTPBoot

**Version**

2.1.2

### Pulp

**Version**

2.1.0

**Pulp server version**

2.21.3

**Database connection**

**Messaging connection**

**Workers**

10

**Supported Content Types**

* deb
* puppet

### Pulpcore

**Version**

2.1.0

**Supported Content Types**

* docker
* file
* yum

### SSH

**Version**

0.3.0

### TFTP

**Version**

2.1.2

**TFTP server**

false

iballou · October 5, 2020, 3:20pm

Okay great, so you’re on Pulp 3. That’s what I was expected, but just wanted to double check.

If 3.16.1.2 doesn’t fix things when it’s out, I’ll ask you to file a redmine bug under Katello at Overview - Katello - Foreman.

In the meantime, there are some things we can try. Expanding on what @jjeffers mentioned before, try this in the foreman Rails console (sudo foreman-rake console):

Choose a content host who has improper errata applicability and note its numerical ID from the URL
In the console:

::Katello::Host::ContentFacet.find(PUT ID HERE).calculate_and_import_applicability

Check if the applicable errata updated to what you want to see.
If the errata is not updated in the UI, check the errata from the console:

::Katello::Host::ContentFacet.find(PUT THE ID HERE).applicable_errata.pluck(:id, :errata_id, :title)

Check that the errata noted there matches what you see in the UI. Let me know if it doesn’t.
Show me what the following command outputs:

::Katello::ApplicableHostQueue.queue_depth

The above is a queue that Katello consumes from to calculate host applicability in an orderly fashion. I want to make sure everything is getting cleared from the queue.

Lastly, are you seeing a “Bulk generate applicability for hosts” task get created in the tasks monitor when you click the recalculate errata button?

matemikulic · October 5, 2020, 3:52pm

I hope also new version fixes it.
I’ve chosen a random host and ran the:

irb(main):004:0> ::Katello::Host::ContentFacet.find(6540).calculate_and_import_applicability
=> true

Applicable errata which was calculated wrong still stayed the same, same as if I would click recalculate button on UI and matches the one on UI.

irb(main):005:0> ::Katello::Host::ContentFacet.find(6540).applicable_errata.pluck(:id, :errata_id, :title)
=> [[4593, "CESA-2020:3220", "Important CentOS kernel Security Update"], [4600, "CESA-2020:2664", "Important CentOS kernel Security Update"], [4612, "CESA-2020:2082", "Important CentOS kernel Update"], [4626, "CESA-2020:1190", "Moderate CentOS libxml2 Security Update"], [4629, "CESA-2020:1181", "Low CentOS unzip Security Update"], [4630, "CESA-2020:1180", "Moderate CentOS autotrace Security Update"], [4632, "CESA-2020:1176", "Low CentOS avahi Security Update"], [4638, "CESA-2020:1138", "Low CentOS gettext Security Update"], [4639, "CESA-2020:1135", "Low CentOS polkit Security Update"], [4641, "CESA-2020:1131", "Moderate CentOS python Security Update"], [4645, "CESA-2020:1113", "Moderate CentOS bash Security Update"], [4648, "CESA-2020:1100", "Moderate CentOS mariadb Security Update"], [4652, "CESA-2020:1080", "Moderate CentOS atk Security Update"], [4659, "CESA-2020:1050", "Moderate CentOS cups Security Update"], [4665, "CESA-2020:1022", "Low CentOS file Security Update"], [4666, "CESA-2020:1021", "Moderate CentOS accountsservice Security Update"], [4668, "CESA-2020:1016", "Moderate CentOS kernel Security Update"], [4669, "CESA-2020:1011", "Moderate CentOS expat Security Update"], [4696, "CESA-2020:0227", "Important CentOS sqlite Security Update"], [4697, "CESA-2020:0203", "Important CentOS libarchive Security Update"], [4705, "CESA-2019:4326", "Important CentOS fribidi Security Update"], [2384, "FEDORA-EPEL-2020-95e11af536", "nagios-plugins-2.3.3-2.el7"], [5533, "CEEA-2020:2743", "CentOS microcode_ctl Enhancement Update"], [5536, "CEEA-2020:1165", "CentOS libselinux Enhancement Update"], [5540, "CEEA-2020:0988", "CentOS strace Enhancement Update"], [3720, "FEDORA-EPEL-2020-3e2935b6e2", "nagios-4.4.5-7.el7"], [6156, "CEBA-2020:3528", "CentOS kernel BugFix Update"], [6177, "CEBA-2020:2355", "CentOS kernel BugFix Update"], [6211, "CEBA-2020:1200", "CentOS sysstat BugFix Update"], [6213, "CEBA-2020:1198", "CentOS e2fsprogs BugFix Update"], [6217, "CEBA-2020:1194", "CentOS libseccomp BugFix Update"], [6221, "CEBA-2020:1188", "CentOS device-mapper-persistent-data BugFix Update"], [6222, "CEBA-2020:1186", "CentOS parted BugFix Update"], [6228, "CEBA-2020:1174", "CentOS iptables BugFix Update"], [6229, "CEBA-2020:1171", "CentOS libcap BugFix Update"], [6232, "CEBA-2020:1168", "CentOS rpcbind BugFix Update"], [6234, "CEBA-2020:1163", "CentOS numactl BugFix Update"], [6235, "CEBA-2020:1162", "CentOS NetworkManager BugFix Update"], [6237, "CEBA-2020:1157", "CentOS policycoreutils BugFix Update"], [6250, "CEBA-2020:1142", "CentOS kmod BugFix Update"], [6251, "CEBA-2020:1140", "CentOS plymouth BugFix Update"], [6252, "CEBA-2020:1139", "CentOS dracut BugFix Update"], [6256, "CEBA-2020:1133", "CentOS libteam BugFix Update"], [6257, "CEBA-2020:1130", "CentOS cryptsetup BugFix Update"], [6262, "CEBA-2020:1123", "CentOS python-urlgrabber BugFix Update"], [6263, "CEBA-2020:1122", "CentOS yum BugFix Update"], [6264, "CEBA-2020:1120", "CentOS setup BugFix Update"], [6267, "CEBA-2020:1114", "CentOS rpm BugFix Update"], [6274, "CEBA-2020:1104", "CentOS linux-firmware BugFix Update"], [6276, "CEBA-2020:1102", "CentOS util-linux BugFix Update"], [6285, "CEBA-2020:1090", "CentOS libffi BugFix Update"], [6286, "CEBA-2020:1089", "CentOS ltrace BugFix Update"], [6288, "CEBA-2020:1087", "CentOS dhcp BugFix Update"], [6295, "CEBA-2020:1077", "CentOS kexec-tools BugFix Update"], [6303, "CEBA-2020:1066", "CentOS device-mapper-multipath BugFix Update"], [6309, "CEBA-2020:1058", "CentOS passwd BugFix Update"], [6312, "CEBA-2020:1055", "CentOS adcli BugFix Update"], [6316, "CEBA-2020:1048", "CentOS sudo BugFix Update"], [6317, "CEBA-2020:1046", "CentOS rsync BugFix Update"], [6319, "CEBA-2020:1043", "CentOS libpcap BugFix Update"], [6320, "CEBA-2020:1042", "CentOS initscripts BugFix Update"], [6321, "CEBA-2020:1041", "CentOS sed BugFix Update"], [6329, "CEBA-2020:1029", "CentOS krb5 BugFix Update"], [6334, "CEBA-2020:1024", "CentOS logrotate BugFix Update"], [6335, "CEBA-2020:1023", "CentOS acl BugFix Update"], [6337, "CEBA-2020:1018", "CentOS procps-ng BugFix Update"], [6344, "CEBA-2020:1009", "CentOS hwdata BugFix Update"], [6345, "CEBA-2020:1008", "CentOS tuned BugFix Update"], [6348, "CEBA-2020:1005", "CentOS pam BugFix Update"], [6349, "CEBA-2020:1004", "CentOS postfix BugFix Update"], [6352, "CEBA-2020:0999", "CentOS perl BugFix Update"], [6354, "CEBA-2020:0997", "CentOS perl-Socket BugFix Update"], [6361, "CEBA-2020:0989", "CentOS glibc BugFix Update"], [6362, "CEBA-2020:0987", "CentOS elfutils BugFix Update"]]

Queue remain empty ever since I cleaned up those stuck “Bulk generate applicability” tasks because my db connection pool was not large enough. Ever since I increased db pool, I see them running a lot of times per day and all end in success.

irb(main):006:0> ::Katello::ApplicableHostQueue.queue_depth
=> 0

But if I go to that host and try to apply that errata, it says that it is already applied, which it is, inspecting errata packages they are already on versions they should be. Host is correct.

# yum update-minimal --security --bugfix -y
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-
              : manager, tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base                                                                                | 2.1 kB  00:00:00     
organization_centos7_epel                                                                                | 2.1 kB  00:00:00     
organization_centos7_extras                                                                              | 2.1 kB  00:00:00     
organization_centos7_internalrepo                                                                     | 2.1 kB  00:00:00     
organization_centos7_updates                                                                             | 2.1 kB  00:00:00     
No Packages marked for minimal Update
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager

iballou · October 5, 2020, 4:02pm

Okay, good info there. Seems like the applicability calculation itself is going wrong somewhere.

I’d like to try to reproduce the issue. Are you able to give me an example of an erratum that is incorrectly sticking around and the repository where it came from?

iballou · October 5, 2020, 4:08pm

Also, are there incorrect applicable RPM packages showing up too? If you can give me a bad applicable package that relates to one of the bad applicable errata, that would be helpful too. I’ll have to try installing an older version of the same package.

matemikulic · October 5, 2020, 6:09pm

Sure. Tnx again. I’ll take the example of the same host.
Errata CESA-2020:3220 has the following packages listed:

CESA-2020:3220
Type
security
Title
Important CentOS kernel Security Update
Issued
2020-07-30
Updated
Reboot Suggested
false
Description
Important CentOS kernel Security Update

Solution
Packages
bpftool-3.10.0-1127.18.2.el7.x86_64
kernel-3.10.0-1127.18.2.el7.src
kernel-3.10.0-1127.18.2.el7.x86_64
kernel-abi-whitelists-3.10.0-1127.18.2.el7.noarch
kernel-debug-3.10.0-1127.18.2.el7.x86_64
kernel-debug-devel-3.10.0-1127.18.2.el7.x86_64
kernel-devel-3.10.0-1127.18.2.el7.x86_64
kernel-doc-3.10.0-1127.18.2.el7.noarch
kernel-headers-3.10.0-1127.18.2.el7.x86_64
kernel-tools-3.10.0-1127.18.2.el7.x86_64
kernel-tools-libs-3.10.0-1127.18.2.el7.x86_64
kernel-tools-libs-devel-3.10.0-1127.18.2.el7.x86_64
perf-3.10.0-1127.18.2.el7.x86_64
python-perf-3.10.0-1127.18.2.el7.x86_64

If I run it to apply it on host:

# yum update-minimal --advisory=CESA-2020:3220
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-
              : manager, tracer_upload
Loading mirror speeds from cached hostfile
organization_centos7_base                                                                                | 2.1 kB  00:00:00     
organization_centos7_epel                                                                                | 2.1 kB  00:00:00     
organization_centos7_extras                                                                              | 2.1 kB  00:00:00     
organization_centos7_internalrepo                                                                              | 2.1 kB  00:00:00     
organization_centos7_updates                                                                             | 2.1 kB  00:00:00     
No Packages marked for minimal Update
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager

Checking packages on server:

# rpm -qa | grep -E 'kernel-*|python-perf|bpftool*'
kernel-3.10.0-1127.19.1.el7.x86_64
kernel-tools-3.10.0-1127.19.1.el7.x86_64
python-perf-3.10.0-1127.19.1.el7.x86_64
kernel-3.10.0-514.10.2.el7.x86_64
kernel-3.10.0-1127.18.2.el7.x86_64
kernel-tools-libs-3.10.0-1127.19.1.el7.x86_64

For that host these are under installed on UI.
Selection_031
Checking applicable packages for that same host:
Selection_030

I get my errata from here Index of /7-sync/updates/x86_64/repodata
Repo is open except updateinfo file containing errata. The owner has it as hidden and password protected so only ones who support his efforts on Patreon with $1/month can download it. Let me know if you would still like to reproduce the issue on your side, I’ll provide that missing file to you.

iballou · October 5, 2020, 6:56pm

Interesting, I haven’t worked with CentOS errata before so maybe something new is going on. You can just PM me the file if you’d like, or email me at iballou@redhat.com.

matemikulic · October 6, 2020, 9:35am

File sent via DM.
If anybody else would like to test feel free to ping me and I’ll provide the same.

matemikulic · October 6, 2020, 1:41pm

fyi, I also found a bug in pulp3 earlier. this worked in all previous versions that used pulp2 Pulp 3 can't work with yum repo requiring auth/Foreman 2.1.2

iballou · October 6, 2020, 2:58pm

@matemikulic I’m not super confident it’ll fix your issue since I think it’s related to our EVR parsing, but 3.16.1.2 is out Katello 3.16.1.2 has been released!

matemikulic · October 6, 2020, 3:24pm

Excellent. Tnx. Will give it a go anyway Fingers crossed.
I’m in the middle of patching servers right now. I’ll test it tomorrow and update the thread.