Missing repos in redhat.repo on host after (forced) subscription manager register

[langesmalle]

Problem:
I have issue after creating new content view, lifecycle environment, promoting the contentview to the lifecycle env and configuring the activation key.
When I execute subscription-manager register with --force on the host(s), not all the repos for the CentOS 7 product are configured in the redhat.repo file.

Configuration details:

Product CentOS_7:
[root@dc1-bootstrap ~]# hammer product list --organization DIDM | grep CentOS_7
12 | CentOS_7 | | DIDM | 5 | Syncing Complete.
[root@dc1-bootstrap ~]# hammer product info --organization DIDM --id 12
Id: 12
Name: CentOS_7
Label: CentOS_7
Description:
Sync State (all): Syncing Complete.
Sync State (last): Syncing Complete.
Sync Plan ID:
GPG:
GPG Key ID:
GPG Key:
Organization: DIDM
Content:

1. Repo Name: extras x86_64
   Url: /custom/CentOS_7/extras_x86_64
   Content Type: yum
2. Repo Name: os x86_64
   Url: /custom/CentOS_7/os_x86_64
   Content Type: yum
3. Repo Name: updates x86_64
   Url: /custom/CentOS_7/updates_x86_64
   Content Type: yum
4. Repo Name: sclo x86_64 rh
   Url: /custom/CentOS_7/sclo_x86_64_rh
   Content Type: yum
5. Repo Name: sclo x86_64 sclo
   Url: /custom/CentOS_7/sclo_x86_64_sclo
   Content Type: yum

CentOS_7 repositories:
[root@dc1-bootstrap ~]# hammer repository list | grep CentOS_7
22 | extras x86_64 | CentOS_7 | yum | http://ftp.belnet.be/mirror/ftp.centos.org/7/extras/x86_64/
23 | os x86_64 | CentOS_7 | yum | http://ftp.belnet.be/mirror/ftp.centos.org/7/os/x86_64/
25 | sclo x86_64 rh | CentOS_7 | yum | http://ftp.belnet.be/mirror/ftp.centos.org/7/sclo/x86_64/rh/
26 | sclo x86_64 sclo | CentOS_7 | yum | http://ftp.belnet.be/mirror/ftp.centos.org/7/sclo/x86_64/sclo/
24 | updates x86_64 | CentOS_7 | yum | http://ftp.belnet.be/mirror/ftp.centos.org/7/updates/x86_64/

ContentView:
[root@dc1-bootstrap ~]# hammer content-view version info --id 87
Id: 87
Name: dc1_testlab_Ceph 2.0
Version: 2.0
Description:
Content View ID: 23
Content View Name: dc1_testlab_Ceph
Content View Label: dc1_testlab_Ceph
Lifecycle Environments:

1. Id: 1
   Name: Library
   Label: Library
2. Id: 37
   Name: DC1_Testlab_Ceph
   Label: DC1_Testlab_Ceph
   Repositories:
3. Id: 1712
   Name: os x86_64
   Label: os_x86_64
4. Id: 1714
   Name: sclo x86_64 rh
   Label: sclo_x86_64_rh
5. Id: 1715
   Name: sclo x86_64 sclo
   Label: sclo_x86_64_sclo
6. Id: 1719
   Name: x86_64 stable
   Label: x86_64_stable
7. Id: 1716
   Name: Puppet_6_el7_x86_64
   Label: Puppet_6_el7_x86_64
8. Id: 1718
   Name: Pacific
   Label: Pacific
9. Id: 1713
   Name: updates x86_64
   Label: updates_x86_64
10. Id: 1722
    Name: x86_64
    Label: x86_64
11. Id: 1720
    Name: x86_64
    Label: x86_64
    10)Id: 1711
    Name: extras x86_64
    Label: extras_x86_64
    11)Id: 1721
    Name: noarch
    Label: noarch
    12)Id: 1717
    Name: Foreman_Client_el7
    Label: Foreman_Client_el7

[root@dc1-bootstrap ~]# hammer content-view info --id 23
Id: 23
Name: dc1_testlab_Ceph
Label: dc1_testlab_Ceph
Composite: false
Description: dc1_testlab_Ceph
Content Host Count: 6
Solve Dependencies: false
Organization: DIDM
Yum Repositories:

1. Id: 22
   Name: extras x86_64
   Label: extras_x86_64
2. Id: 23
   Name: os x86_64
   Label: os_x86_64
3. Id: 24
   Name: updates x86_64
   Label: updates_x86_64
4. Id: 25
   Name: sclo x86_64 rh
   Label: sclo_x86_64_rh
5. Id: 26
   Name: sclo x86_64 sclo
   Label: sclo_x86_64_sclo
6. Id: 19
   Name: Puppet_6_el7_x86_64
   Label: Puppet_6_el7_x86_64
7. Id: 17
   Name: Foreman_Client_el7
   Label: Foreman_Client_el7
8. Id: 1607
   Name: x86_64 stable
   Label: x86_64_stable
9. Id: 39
   Name: x86_64
   Label: x86_64
   10)Id: 27
   Name: noarch
   Label: noarch
   11)Id: 31
   Name: x86_64
   Label: x86_64
   Container Image Repositories:
10. Id: 1464
    Name: Pacific
    Label: Pacific
    OSTree Repositories:

Lifecycle Environments:

1. Id: 1
   Name: Library
2. Id: 37
   Name: DC1_Testlab_Ceph
   Versions:
3. Id: 86
   Version: 1.0
   Published: 2023/05/30 19:37:20
4. Id: 87
   Version: 2.0
   Published: 2023/05/30 19:44:47
   Components:

Activation Keys:

1. Key_DC1_Testlab_Ceph_Servers

Activation Key:
[root@dc1-bootstrap ~]# hammer activation-key info --organization DIDM --id 9
Name: Key_DC1_Testlab_Ceph_Servers
Id: 9
Description: Key_DC1_Testlab_Ceph_Servers
Host Limit: Unlimited
Auto Attach: true
Lifecycle Environment: DC1_Testlab_Ceph
Content View: dc1_testlab_Ceph
Host Collections:

Content Overrides:

1. Content Label: DIDM_CentOS_7_sclo_x86_64_rh
   Name: enabled
   Value: 0
2. Content Label: DIDM_CentOS_7_sclo_x86_64_sclo
   Name: enabled
   Value: 0
3. Content Label: DIDM_Epel_EL7_x86_64
   Name: enabled
   Value: 0
   System Purpose:
   Service Level:
   Purpose Usage:
   Purpose Role:
   Purpose Addons:

[root@dc1-bootstrap ~]# hammer activation-key product-content --organization DIDM --id 9
----|---------------------|--------------------------------------------|------------------|----------

ID	NAME	LABEL	DEFAULT ENABLED?	OVERRIDE
22	extras x86_64	DIDM_CentOS_7_extras_x86_64	yes
17	Foreman_Client_el7	DIDM_Foreman_Client_EL7_Foreman_Client_el7	yes
27	noarch	DIDM_Ceph_Nautilus_EL7_noarch	yes
23	os x86_64	DIDM_CentOS_7_os_x86_64	yes
19	Puppet_6_el7_x86_64	DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64	yes
25	sclo x86_64 rh	DIDM_CentOS_7_sclo_x86_64_rh	yes	enabled:0
26	sclo x86_64 sclo	DIDM_CentOS_7_sclo_x86_64_sclo	yes	enabled:0
24	updates x86_64	DIDM_CentOS_7_updates_x86_64	yes
39	x86_64	DIDM_Epel_EL7_x86_64	yes	enabled:0
31	x86_64	DIDM_Ceph_Nautilus_EL7_x86_64	yes
145	x86_64 stable	DIDM_Docker_x86_64_stable	yes
----	---------------------	--------------------------------------------	------------------	----------

Host - subscription-manager repos:
[root@dc1-testlab-mon1 ~]# subscription-manager repos
±---------------------------------------------------------+
Available Repositories in /etc/yum.repos.d/redhat.repo
±---------------------------------------------------------+
Repo ID: DIDM_Epel_EL7_x86_64
Repo Name: x86_64
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Epel_EL7/x86_64
Enabled: 0

Repo ID: DIDM_Ceph_Nautilus_EL7_noarch
Repo Name: noarch
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/noarch
Enabled: 1

Repo ID: DIDM_Ceph_Nautilus_EL7_x86_64
Repo Name: x86_64
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/x86_64
Enabled: 1

Repo ID: DIDM_Foreman_Client_EL7_Foreman_Client_el7
Repo Name: Foreman_Client_el7
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Foreman_Client_EL7/Foreman_Client_el7
Enabled: 1

Repo ID: DIDM_CentOS_7_extras_x86_64
Repo Name: extras x86_64
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/CentOS_7/extras_x86_64
Enabled: 1

Repo ID: DIDM_Docker_x86_64_stable
Repo Name: x86_64 stable
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Docker/x86_64_stable
Enabled: 1

Repo ID: DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64
Repo Name: Puppet_6_el7_x86_64
Repo URL: https://dc1-bootstrap/pulp/content/DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Puppet_6_EL7/Puppet_6_el7_x86_64
Enabled: 1

It only contains the CentOS_7_extra repo, all others are missing!

Expected outcome:

All CentOS_7 repos configured in the activation key be present in the file redhat.repo.

Foreman and Proxy versions:
Foreman 3.5.1
Katello 4.7.0

Foreman and Proxy plugin versions:

Distribution and version:
Rocky 8.7

Other relevant data:

[iballou]

Hi @langesmalle,

Just some basic debugging questions to start:

1. Does enabling or disabling content overrides on repository sets do anything?

2. What happens if you register a host without an activation key?

3. Does the following return any of your missing CentOS 7 repos?

::Host.find_by(name: "your host name").content_facet.bound_repositories

@jeremylenz @cintrix84 do you think this could be an issue with either Candlepin itself or with Katello not configuring Candlepin properly? This isn’t the first time that I’ve seen users hit issues with repos not making it to hosts on Katello 4.7.

[jeremylenz]

More basics:

Has the host’s content view been published since adding the repos?
Are any of the repos arch- or OS-version restricted? That would cause them not to show up in redhat.repo.

Failing that,

On the host, in /etc/pki/entitlement , if you run rct cat-cert on the .pem file for that product, is the content section missing? (see https://bugzilla.redhat.com/show_bug.cgi?id=1931027)

If this is Katello 4.7+, you can also try running

foreman-rake katello:check_candlepin_content

to confirm if this is the issue.

If so, you’ll have to delete and recreate the offending custom repositories.

[jeremylenz]

@iballou

Does enabling or disabling content overrides on repository sets do anything

I think that would affect the Enabled: 0 or Enabled: 1 in subscription-manager repos, but content overrides wouldn’t cause a repo to disappear completely from that list.

[langesmalle]

None of the CentOS_7 repos have restricted arch or OS version configured.
For all the repos that are configured in the /etc/yum.repos.d/redhat.repo, the command rct cat-cert on the .pem files return the content:

[root@dc1-testlab4-mon1 entitlement]# for pem in ls -1 | grep -v key

do
rct cat-cert $pem | grep ^Content -A 9
done
Content:
Type: yum
Name: Puppet_6_el7_x86_64
Label: DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Puppet_6_EL7/Puppet_6_el7_x86_64
GPG:
Enabled: True
Expires: 1
Required Tags:
Content:
Type: yum
Name: Foreman_Client_el7
Label: DIDM_Foreman_Client_EL7_Foreman_Client_el7
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Foreman_Client_EL7/Foreman_Client_el7
GPG:
Enabled: True
Expires: 1
Required Tags:
Content:
Type: yum
Name: extras x86_64
Label: DIDM_CentOS_7_extras_x86_64
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/CentOS_7/extras_x86_64
GPG:
Enabled: True
Expires: 1
Required Tags:
Content:
Type: yum
Name: x86_64 stable
Label: DIDM_Docker_x86_64_stable
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Docker/x86_64_stable
GPG:
Enabled: True
Expires: 1
Required Tags:
Content:
Type: yum
Name: x86_64
Label: DIDM_Epel_EL7_x86_64
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Epel_EL7/x86_64
GPG:
Enabled: True
Expires: 1
Required Tags:
Content:
Type: yum
Name: noarch
Label: DIDM_Ceph_Nautilus_EL7_noarch
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/noarch
GPG:
Enabled: True
Expires: 1
Required Tags:
–
Content:
Type: yum
Name: x86_64
Label: DIDM_Ceph_Nautilus_EL7_x86_64
Vendor: Custom
URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/x86_64
GPG:
Enabled: True
Expires: 1
Required Tags:

But as said, the CentOS_7 product contains 5 repos (os, extras, updates, sclo and sclo rh) and only the extras repo is configured in the redhat.repo and as well the pem file is available.
For the other repos, they are is missing.
Only the sclo and sclo rh repos are disabled in the activation key.
And the other products are well processed/configured in th redhat.repo file, such as epel9 (disabled in the act-key), Ceph, puppet, foreman-client,…

Katello version is 4.7.0-1.el8 and the check returns me:
[root@dc1-bootstrap ~]# foreman-rake katello:check_candlepin_content
I, [2023-06-06T08:54:01.559066 #3767158] INFO – : Checked 65 repositories.

As the CentOS_7 repo is also used for other hosts (in production) I prefer not to remove the repo’s unless it should be the final solution.
But that means much more clean-up like in the content view, activation keys, …, right?

Please let me know if there are other actions/checks I should try.

Thnx in advance for you help.

[jeremylenz]

If rake katello:check_candlepin_content didn’t find any bad repositories, we shouldn’t need to recreate anything.

Has the host’s content view been published since adding the repos?

In Hosts > All Hosts > (your host) > Overview tab > Content view details card, can you please share a screenshot of the content view version assigned to your host?

In Content > Content Views > (your content view) > Versions tab > (your host’s content view version) > do the contents of that CV version show the repositories in question? (keep in mind this is the Versions tab, not Repositories tab.) Can you please share a screenshot?

[langesmalle]

Hi,

Here the screenshots:

Host-overview:

Contentview-versions:

[jeremylenz]

Thanks. Everything looks fine

On the rct cat-cert, right after Required Tags there is a field Arches:. Can you share that output again but with -A 10 so I can see the Arches?

A mismatch of Required Tags or Arches, or the actual absence of the repo in the content view version, are the only things I know that would cause the repo not to show up at all.

[langesmalle]

Here the output:

Content:
	Type: yum
	Name: Puppet_6_el7_x86_64
	Label: DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Puppet_6_EL7/Puppet_6_el7_x86_64
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
Content:
	Type: yum
	Name: Foreman_Client_el7
	Label: DIDM_Foreman_Client_EL7_Foreman_Client_el7
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Foreman_Client_EL7/Foreman_Client_el7
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
Content:
	Type: yum
	Name: extras x86_64
	Label: DIDM_CentOS_7_extras_x86_64
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/CentOS_7/extras_x86_64
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
Content:
	Type: yum
	Name: x86_64 stable
	Label: DIDM_Docker_x86_64_stable
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Docker/x86_64_stable
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
Content:
	Type: yum
	Name: x86_64
	Label: DIDM_Epel_EL7_x86_64
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Epel_EL7/x86_64
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
Content:
	Type: yum
	Name: noarch
	Label: DIDM_Ceph_Nautilus_EL7_noarch
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/noarch
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL
--
Content:
	Type: yum
	Name: x86_64
	Label: DIDM_Ceph_Nautilus_EL7_x86_64
	Vendor: Custom
	URL: /DIDM/DC1_Testlab_Ceph/dc1_testlab_Ceph/custom/Ceph_Nautilus_EL7/x86_64
	GPG:
	Enabled: True
	Expires: 1
	Required Tags:
	Arches: ALL

But remember that the keys for the missing CentOS_7 repos are missing as well!

[langesmalle]

@jeremylenz ,

I think the CentOS_7 product/repos are corrupt.

I executed the forced subscription-manager on another test server which has also the CentOS_7 product (with anoter lifecycle env and contentview) .

Initially it had all the repos configured for CentOS_7 (I made a copy of the initial redhat.repo):

[root@dc1-testlab1-one-fe1 yum.repos.d]# cat redhat.repo_copy | egrep "DIDM|baseurl|enabled" | sed "s/bootstrap1/bootstrap/"
[DIDM_Epel_EL7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Epel_EL7/x86_64
enabled = 0
[DIDM_CentOS_7_os_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/os_x86_64
enabled = 1
[DIDM_Ceph_Nautilus_EL7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Ceph_Nautilus_EL7/x86_64
enabled = 1
[DIDM_Qemu_kvm_ev_EL7_Qemu_kvm_ev_el7]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Qemu_kvm_ev_EL7/Qemu_kvm_ev_el7
enabled = 1
[DIDM_CentOS_7_updates_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/updates_x86_64
enabled = 1
[DIDM_CentOS_7_extras_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/extras_x86_64
enabled = 1
[DIDM_OpenNebula_6_4_x_EL7_x84_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/OpenNebula_6_4_x_EL7/x84_64
enabled = 1
[DIDM_Foreman_Client_EL7_Foreman_Client_el7]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Foreman_Client_EL7/Foreman_Client_el7
enabled = 1
[DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Puppet_6_EL7/Puppet_6_el7_x86_64
enabled = 1
[DIDM_CentOS_7_sclo_x86_64_sclo]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/sclo_x86_64_sclo
enabled = 0
[DIDM_CentOS_7_sclo_x86_64_rh]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/sclo_x86_64_rh
enabled = 0

after the forced subscription-manager (same contentview version as the initial register!):

[root@dc1-testlab1-one-fe1 yum.repos.d]# cat redhat.repo | egrep "DIDM|baseurl|enabled" | sed "s/bootstrap1/bootstrap/"
[DIDM_Epel_EL7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Epel_EL7/x86_64
enabled = 0
[DIDM_Ceph_Nautilus_EL7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Ceph_Nautilus_EL7/x86_64
enabled = 1
[DIDM_Qemu_kvm_ev_EL7_Qemu_kvm_ev_el7]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Qemu_kvm_ev_EL7/Qemu_kvm_ev_el7
enabled = 1
[DIDM_Foreman_Client_EL7_Foreman_Client_el7]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Foreman_Client_EL7/Foreman_Client_el7
enabled = 1
[DIDM_CentOS_7_extras_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/CentOS_7/extras_x86_64
enabled = 1
[DIDM_OpenNebula_6_4_x_EL7_x84_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/OpenNebula_6_4_x_EL7/x84_64
enabled = 1
[DIDM_Puppet_6_EL7_Puppet_6_el7_x86_64]
enabled_metadata = 1
baseurl = https://dc1-bootstrap.cumuli.be/pulp/content/DIDM/OpenNebula_DC1_Testlab/OpenNebula_Servers/custom/Puppet_6_EL7/Puppet_6_el7_x86_64
enabled = 1

Only the CentOS_7 extra repo is now available, the others are missing!

So it looks like something became corrupt!
Any suggestion?

[iballou]

@langesmalle, how would you feel about sharing a backup of your Foreman environment so we Katello devs can poke around Candlepin to see what’s wrong?

If you’re okay with it, you would provide the output of foreman-maintain backup online /path/to/ouput

Transferring the backup to us is another question, but I’ll wait to see if you’re interested at all.

[iballou]

An update for anyone following along, we’re looking at a reproducer DB (which took longer to set up than I’d like to admit) and will report back if we see anything out of the ordinary.

[iballou]

So far nothing in the DBs is jumping out at us, we need more info from Candlepin experts so we can get a better understanding about what Candlepin data actually tells sub-man to put a repo in redhat.repo.

Edit:

With some help from Candlepin folks, it looks like the CentOS_7 product in Candlepin is missing product_content for all repositories except the extras repo. That missing product_content means those repositories will not make their way into the redhat.repo file.

Now that we’ve found that, we just need to determine the path forward.

(I’m going to keep replying on this comment because Discourse doesn’t allow more than 3 consecutive replies)

---

@langesmalle I have a cleanup attempt to try:

We need to create the product_content in Candlepin. Let’s try with one repository first for a test:

root = ::Katello::RootRepository.find_by(name: "os x86_64")
::ForemanTasks.sync_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content.cp_content_id)

I think that should be all that is needed… the product content exists already in your Katello repository.
After that, re-register a host and see if it can find that os x86_64 repository in its redhat.repo file.