Most repos from content view no longer appear on hosts

Problem:

Our /var/lib/pulp is taking too much space, so in order to try to recover some back we tried to activate mirror-on-sync so on the next sync the rpms that had multiple versions would be eliminated.

We ran this script to try to enable the option on all repos:

#!/bin/bash
declare -a repo_list=$(hammer repository list | awk '/^[0-9]/ {print $1}')
for i in ${repo_list[@]}; do
    hammer repository update --id "$i" --mirror-on-sync true
done

Apparently all repos were updated - we got a message saying the repo was updated for each one.
Since then the repos were synced and Content Views were published.

Today, when registering new hosts, we found that only one repo appears on the new hosts.
We made no changes to the products neither to the Content Views.
We doubled checked and altough the hammer repository update … apparently worked all the repos we checked have the mirror-on-sync set to false.

Some extra weird behavior:

  • some older hosts subscribed exactly the same can still “see” all the content view repos and install rpms from them
  • if we had a new repo to the product and content view it appears right away on the hosts (after publishing the CV)

Foreman and Proxy versions:
Foreman version: 3.2.1

Foreman and Proxy plugin versions:
Plugins:

  • foreman-tasks 6.0.1
  • foreman_host_reports 1.0.2
  • foreman_puppet 3.0.7
  • foreman_remote_execution 6.0.0
  • katello 4.4.1

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Other relevant data:
We tried several sync methods on the repos that “disappeared” but none made the repos appear on new machines.
We also ran the hammer repository update command for the repo that was added and that appeared right away but it did not make it “disappear” (even after syncing and publishing a new CV)

mirror-on-sync has been deprecated:

 --mirror-on-sync BOOLEAN                     True if this repository when synced has to be mirrored from the source and stale
                                              rpms removed (Deprecated)

I don’t know what it does exactly if you try to set it.

I would recommend to set it properly using the options :slight_smile:

 --download-policy ENUM                       Download policy for yum repos (either 'immediate' or 'on_demand')
                                              Possible value(s): 'immediate', 'on_demand'
...
 --mirroring-policy ENUM                      Policy to set for mirroring content.  Must be one of additive.
                                              Possible value(s): 'additive', 'mirror_complete', 'mirror_content_only'

The default would be immediate and mirror_content_only.

Hello gvde,

Thank you for your reply.
Unfortunately after running:

hammer repository update --id 17082 --download-policy 'on_demand' --mirroring-policy 'mirror_content_only'
Repository updated.

Followed by syncing the repo and publishing a new Content View the repo still does not appear on the hosts.

I tried to find all the repo settings - to compare with one that it is still working, but hammer repository info does not show everything and the info appears to be spread across several tables on the database (besides core_repository, core_remote and rpm_rpmrepository which tables have more info?).
Thank you.

Best regards,
Armindo Silva.

What exactly do you mean by that? How does it not “appear”?

What do you see with yum repolist -v?

What do you see with subscription-manager repos?

Our Rocky 8 Content View

all the repos were available on the hosts but now:

$ dnf repolist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, etckeeper, generate_completion_cache, groups-manager, kpatch, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, tracer-upload, uploadprofile
Updating Subscription Management repositories.
DNF version: 4.7.0
cachedir: /var/cache/dnf
Rocky Linux 8 Foreman Client RPMS                                                                                                                                                                           830  B/s | 2.3 kB     00:02
Rocky 8 AppStream                                                                                                                                                                                            16 kB/s | 2.6 kB     00:00
Repo-id            : Example_Rocky_Linux_8_x86_64_Rocky_8_AppStream
Repo-name          : Rocky 8 AppStream
Repo-revision      : 1655183256
Repo-updated       : Tue 14 Jun 2022 06:07:35 AM WEST
Repo-pkgs          : 6,862
Repo-available-pkgs: 5,489
Repo-size          : 11 G
Repo-baseurl       : https://updates.example.com/pulp/content/Example/Library/RockyLinux8/custom/Rocky_Linux_8_x86_64/Rocky_8_AppStream
Repo-expire        : 1 second(s) (last: Tue 14 Jun 2022 12:40:31 PM WEST)
Repo-filename      : /etc/yum.repos.d/redhat.repo

Repo-id            : Example_Rocky_Linux_8_x86_64_Rocky_Linux_8_Foreman_Client_RPMS
Repo-name          : Rocky Linux 8 Foreman Client RPMS
Repo-revision      : 1655184416
Repo-updated       : Tue 14 Jun 2022 06:26:56 AM WEST
Repo-pkgs          : 15
Repo-available-pkgs: 15
Repo-size          : 8.1 M
Repo-baseurl       : https://updates.example.com/pulp/content/Example/Library/RockyLinux8/custom/Rocky_Linux_8_x86_64/Rocky_Linux_8_Foreman_Client_RPMS
Repo-expire        : 1 second(s) (last: Tue 14 Jun 2022 12:40:27 PM WEST)
Repo-filename      : /etc/yum.repos.d/redhat.repo
Total packages: 6,877
$ subscription-manager repos
+----------------------------------------------------------+
    Available Repositories in /etc/yum.repos.d/redhat.repo
+----------------------------------------------------------+
Repo ID:   Example_Rocky_Linux_8_x86_64_Rocky_Linux_8_Foreman_Client_RPMS
Repo Name: Rocky Linux 8 Foreman Client RPMS
Repo URL:  https://updates.example.com/pulp/content/Example/Library/RockyLinux8/custom/Rocky_Linux_8_x86_64/Rocky_Linux_8_Foreman_Client_RPMS
Enabled:   1

Repo ID:   Example_Rocky_Linux_8_x86_64_Rocky_8_AppStream
Repo Name: Rocky 8 AppStream
Repo URL:  https://updates.example.com/pulp/content/Example/Library/RockyLinux8/custom/Rocky_Linux_8_x86_64/Rocky_8_AppStream
Enabled:   1

The repo Rocky 8 AppStream was the one we added after finding the issue.
It was also on this repo that ran the same hammer repository update … command.

Could be the same issues as in Subscription-manager is missing repository

1 Like

Hi @mindo, thanks for the issue report (and thanks @gvde for jumping in). I have a few questions to better understand

  1. What is the exact candlepin version installed?

  2. Are the missing repositories all associated with the same custom product? Multiple custom products?

  3. For the custom product(s) affected, are there any foreman tasks specifically related to these products, from the time period where the issue may have occurred?

  4. Can you find any requests referencing those affected productions in the candlepin logs, within the time/date range after the repos were known working, and before they were known broken?

If it is indeed https://bugzilla.redhat.com/show_bug.cgi?id=1931027 then you can remediate by getting the ID of the affected product in Katello, in this example it’s 37, then running this short script:

cat << EOF |foreman-rake console
root = Katello::RootRepository.find(37);
ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
EOF

Wait for the task the complete, then try subscription-manager refresh on the affected content hosts.

Please let us know if that resolves it, and additionally, let us know if you can find anything referencing that candlepin product or content in Candlepin’s logs, or in the history for foreman-tasks, in the date range when the issue could have occurred.

Looking forward to hearing from you,

3 Likes

Hello @gvde @wbclark

Thank you very much for your help, I was able to solve the issue with the foreman-rake console commands,

@wbclark answering your questions:
1.

um info candlepin
Loaded plugins: changelog, etckeeper, fastestmirror, versionlock
Loading mirror speeds from cached hostfile
(...)
Installed Packages
Name        : candlepin
Arch        : noarch
Version     : 4.1.10
Release     : 1.el7
Size        : 88 M
Repo        : installed
From repo   : katello-candlepin
Summary     : Candlepin is an open source entitlement management system
URL         : http://www.candlepinproject.org
License     : GPLv2
Description : Candlepin is an open source entitlement management system.

It happen in (almost?) all our custom products

Some tasks were ran - we have some scheduled tasks that sync repos and update errata (CentOS7)

I found some errors, but I am not sure if they are related or in the correct timeframe

$ grep ERROR /var/log/candlepin/candlepin.log
2022-06-14 16:30:34,960 [thread=http-bio-127.0.0.1-23443-exec-10] [req=0edb1b86-304a-4d8f-88ea-2c72a8b6e4af, org=, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error Row was updated or deleted by another transaction (or unsaved-value mapping was incorrect) : [org.candlepin.model.OwnerProduct#org.candlepin.model.OwnerProductKey@a0e2f120] at org.hibernate.dialect.lock.PessimisticWriteSelectLockingStrategy.lock:81
2022-06-14 16:37:30,123 [thread=http-bio-127.0.0.1-23443-exec-3] [req=fd0a532c-556d-477d-90b8-9ff55cf05932, org=, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
2022-06-14 16:37:30,137 [thread=http-bio-127.0.0.1-23443-exec-3] [req=fd0a532c-556d-477d-90b8-9ff55cf05932, org=, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
2022-06-14 16:39:00,442 [thread=http-bio-127.0.0.1-23443-exec-7] [req=131cdd50-04a1-4cbe-87bd-5cb366140307, org=, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
2022-06-14 16:39:00,445 [thread=http-bio-127.0.0.1-23443-exec-7] [req=131cdd50-04a1-4cbe-87bd-5cb366140307, org=, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
2022-06-14 16:39:13,471 [thread=http-bio-127.0.0.1-23443-exec-6] [req=a717ac5a-51f6-475f-a6e7-6d7d4d3e3b83, org=, csid=] ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper - ERROR: duplicate key value violates unique constraint "cp2_product_entity_version"
2022-06-14 16:39:13,473 [thread=http-bio-127.0.0.1-23443-exec-6] [req=a717ac5a-51f6-475f-a6e7-6d7d4d3e3b83, org=, csid=] ERROR org.candlepin.exceptions.mappers.CandlepinExceptionMapper - Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433
Caused by: org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "cp2_product_entity_version

Thank you again for your help.

Best regards,
Armindo Silva.