Katello 4.0/pulp3: no erratas anymore

Support
1

Problem:
Since the switchover to pulp3 and upgrade to katello 4.0 I don’t see any installable errata anymore. Hosts show installable packages but don’t show the matching errata. I can see the errata in the sync summary emails, e.g. yesterday for EPEL7 clamav-0.103.2.1.el7 errata FEDORA-EPEL-2021-3f9b6786f4

However, when I publish and promote the content view which contains those packages, the promotion summary says “0 needed errata”, even though the packages are installed and can be updated on several hosts using that content view.

The updates are available on the host if I check if “yum check-update”.

On the “Content Host” page in the web gui it can see the host has 5 installable packages, but it shows 0 for all errata, although it should show 1 security errata for clamav.

Errata counts are also missing on the content view versions page for this content view. Before the pulp3 migration it says:

59060 Packages
170 Source RPMs
4647 Errata ( 262 (!) 1360 (bug) 1180 (+))

Now it only says:

60781 Packages
171 Source RPMs

So it seems as if the content views lost the errata information completely.

Expected outcome:
Show counts of installable errata.

Foreman and Proxy versions:
Katello 4.0

Distribution and version:
CentOS 7.9

2

On a sidenote: in the daily “Host Errata Advisory” mail I can see the it recognizes the errata as applicable but not installable, i.e. it lists for Security for on of the hosts “0 (1)”.

3

Hey @gvde ,

I’ll take a look. Is it one specific repo that’s causing troubles or is it all repos with errata? Also, can we see the URL for one of the repos you synced? It makes for a better reproducer environment if we can use the same ones.

4

I can’t tell at the moment. I only have EPEL 7 and EPEL 8 which provide errata to my CentOS servers. There is one more server with a RedHat license, but is connected to the Library environment, not in a content view.

And since Friday, when I made the pulp3 switchover and then the upgrade to katello 4.0 the only applicable errata that came in, was clamav.

I have checked my four content views I have and found that actually one of them does show an errata count in the content view version list.

I have two CVs for centos7, one with full epel7 (centos7-epel7) and a much smaller one with a filtered epel7 containing only those epel packages and dependencies which we need on all our servers for monitoring (centos7). I have the same for centos8, i.e. centos8-epel8 with full epel8 and centos8 with a filtered epel8.

For the centos7 CV the content counters are those before the switchover:

43679 Packages
128 Source RPMs
7 Errata ( 0 2 2 )

and that after

43708 Packages
128 Source RPMs

centos7-epel7 has

59060 Packages
170 Source RPMs
4647 Errata ( 262 1360 1180 )

vs.

60773 Packages
170 Source RPMs

centos8 has

17998 Packages
42 Source RPMs
9 Errata ( 0 2 2 )
111 Module Streams

vs.

18015 Packages
42 Source RPMs
111 Module Streams

centos8-epel8, however, shows

23656 Packages
42 Source RPMs
2897 Errata ( 59 364 387 )
127 Module Streams

vs.

23673 Packages
42 Source RPMs
2908 Errata ( 59 368 391 )
127 Module Streams

So as you can see, the errata counts are completely missing from three of the CVs except the last. The last one shows the usual numbers.

If I check the product repositories I can see the full errata count for both repositories, EPEL7 with 4666 errata and EPEL8 with 2907 errata.

EPEL 7 is synced from Index of /epel/7/x86_64/
EPEL 8 is synced from Index of /epel/8/Everything/x86_64/

5

So publishing a new content view version in one of your content views that is missing the errata results in a new content view version that is still missing the errata?

Sounds like it might be more of a migration switchover-related issue then. Are you able to test creating a new content view and publishing+promoting a new content view version? The errata should show up in the new content view version.

If it doesn’t we can dig into the content view versions’ backend pulp repositories to try to see what went wrong.

6

Correct. I have published new versions of the two centos7 CVs three times since the content migration and each time it’s missing the errata. As a test, I have just published a new version of the centos8-epel8 CV and this one still shows the errata count…

As I also see extremely longer publishing times since the migration I have checked the tasks and noticed that those three CVs with the missing errata require much longer:

centos7: 2m to 16m
centos7-epel7: 4m to 25m
centos8: 2m to 10m

However, the centos8-epel7 is much faster and remains at about 2-3 minutes.

So maybe, this issue and the slow publishing times (topic Pulp3 content view publish much slower) are related?

I have just created another test CV using the identical repositories and filters like my centos7 CV. Took 17 minutes to publish and has the same problem. Counts don’t show errata:

43698 Packages
129 Source RPMs

just like the ‘centos7’ CV.

I guess that’s necessary.

7

@gvde I could see it being tied into the extra long publishing times.

I’m curious if the errata shows up in the repodata for your content view versions’ repositories. Check with the following steps if you’re unsure how:

  1. Choose a content view version that is missing errata.
  2. Choose a repo in that content view version that should have errata.
  3. Find that repo’s content view URL using subscription-manager repos (or however you prefer) on a host that is assigned to the content view & lifecycle environment that the chosen content view version belongs to.
  4. Browse to the repository’s hosted content using that URL and look for errata in /repodata/<hash>-updateinfo.xml.gz
1 Like
8

If I browser to the content view, click on the latest version, then on “Yum Repositories” and the search for the EPEL 7 repository, it shows with “0 Errata”.

I have looked for the URL in /etc/yum.repos.d/redhat.repo on a server subscribed to the content view and using the katello 4.0 server as content proxy. Browser to the URL in redhat.repo, added /repodata/ to the URL and then downloaded the updateinfo.xml.gz. Content:

<?xml version="1.0" encoding="UTF-8"?>
<updates>
</updates>

Only the centos8-epel8 has the errata listed.

9

Thanks for that, so it’s a content copying issue then and not an indexing issue.

I think I see where in the code things are going wrong for you. In your thread about the slow publishing times, I noticed in the Dynflow that the MultiCopyContent action was run during publishing. That should only run if you’re using a composite content view, or if you’re using filters. It sounds like you’re using neither in that case, however.

In the Dynflow console for that content view publish task, under the “Run” tab, can you expand the Actions::Pulp3::Repository::MultiCopyContent action and paste what is in there? There should be a repo_id_map.

10

For my memory and other onlookers, the code that decides if MultiCopyContent runs is the if statement here: katello/multi_copy_all_units.rb at master · Katello/katello · GitHub

11

Well, the CVs centos7 and centos8 have filters on the epel 7/8 repository (a couple of include filters leaving only ~80 epel rpms in the content view).

I totally forgot, that the centos7-epel7 does have a filter, too, because we have to filter out the slurm rpm there, i.e. centos7-epel7 has full epel7 but filters out slurm.

centos8-epel8 does not have any filter.

So, as I understand MultiCopyContent should run for those three, slower content views. That’s correct. But those three content views with filters on the epel repository don’t have any errata. That looks like the filter “breaks” the errata generation for the content view?

That output is huge. It has a repo_id_map and among that it also shows the filters for the epel repository:

"[7]":
    dest_repo: 42773
    filter_ids:
    - 15
    - 16
    - 17
    - 18

7 is the id of the epel7 repository and 15-18 are the ids of the filters I have set up for my test content view, mirroring the settings of the centos7 content view.

Most of the time is spent in those 56 pulp_rpm.app.tasks.copy.copy_content tasks (one for each repository in the content view) each taking ~12s, but as it seems not running in parallel.

12

@gvde okay, so it looks like we might be dealing with a filtering bug then. It sounds like the errata is being filtered out when it shouldn’t.

I’d like to try re-creating one of your content views on my Katello 4.0 systems. Are you able to show me the full details of your “centos8” content view? Looks like a small one. If possible, I’d like the list of repositories with URLs and the list of filters. If the list of repositories is too long or sensitive, we can just start with EPEL8 and EPEL8’s filters.

Also, can you show me your Pulp 3 versions?

pip3 list | grep pulp
13

I just set up a test content view with only epel7 and a simple exclude filter for slurm and it’s also missing the errata.

epel 7 repo load from http://download.fedoraproject.org/pub/epel/7/x86_64/

Test CV only contains epel7 repository.

Test CV has one rpm exclude filter named “slurm-epel7”, which filters out all version of the RPM “slurm” (currently that should be slurm-20.11.5-1.el7.x86_64). Affected repositories for the filter is set to the epel7 repository.

After I publish this test CV is shows no errata.

# pip3 list | grep pulp
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pulp-2to3-migration (0.11.0)
pulp-certguard (1.1.0)
pulp-container (2.2.1)
pulp-deb (2.8.0)
pulp-file (1.5.0)
pulp-rpm (3.10.0)
pulpcore (3.9.1)
14

@gvde great, good info. I was able to reproduce the problem. I’m going to figure out if it’s a Pulp or a Katello bug and keep you updated.

2 Likes
15

I’ve created an upstream issue for it: Bug #32435: Excluding packages from EPEL 7 (at least) causes all errata to disappear - Katello - Foreman

It looks like this issue has existed on Katello with Pulp 3 since 3.16, but only for EPEL repositories as far as I can tell. It doesn’t seem to be a size issue since RHEL 7 RPMs filters okay, which is larger than EPEL 7.

1 Like
16

I’ve hit the same problem with a ContentView only containing SLES repositories: New CV versions with SLES repos do not contain (all) errata after Pulp-Migration

It is also a regular ContentView (no composite ContentView), but had one filter applied. As a result the new versions had 18 errata (instead of ~4600). After removing the filter, the published version contains all errata again.

So this problem seems to not only affect EPEL but also SLES repos (at minimum). We also do have EPEL and RHEL content, but those CVs do not use any filters; so I don’t know if we would have the same problem there (probably yes).

17

I’ve found the sources of the issue. Firstly, EPEL has errata in its repositories that have packages that exist in other EPEL repos. There was a bug in Katello since 3.16 that caused these cross-repo errata to be excluded from content view versions with filters: Bug #33254: CV Package count ok, but Errata count (huge) mismatches with pulp2 - Katello - Foreman. This is fixed for Katello 4.2

Secondly, I just discovered another bug. EPEL includes source RPMs in its errata in a way that we weren’t expecting. Right now, errata with SRPMs in the package list (more specifically among the filenames) will be excluded from filtered content view versions. I’ve made a note to fix that in the original bug from this thread.

1 Like