Subscription-manager is missing repository

gvde · June 2, 2022, 2:48pm

Problem:
I have noticed that one repository is missing on most of our EL8 clients. Or better: it’s disappearing. It is still available on a few but only until I do a subscription refresh --force after which is then disappears from redhat.repo and subscription-manager repos --list.

I didn’t change the configuration of any of those clients. Those, who still have an old redhat.repo can access all rpms in that repository. All clients are subscribed to the product, have the repository enabled in the repository set and obviously the content view in use is published and promoted.

The repository actually assigned to all our EL8 clients. It’s a yum repository which is not mirrored but only contains some rpms we create and upload locally.

I have noticed that the redhat.repo is from April 29 or May 23 on all clients which still show the repository. So it must have been something after that date I guess which caused this. I have updated my main server and content proxy to foreman 3.1.3 from 3.1.2 on May 23. But that only updated some foreman rpms.

If I switch one of the clients missing it already to Library environment with “Organizational View” and refresh then, I can see our private/local el7 repository with subscription-manager repos --list. The el7 repo is part of the same subscription as the missing el8 repo. But the el8 repo is still missing. So entitlement and subscription of the client is fine. And it’s not a content view issue, because in Library/Organizational View there is no content view involved…

Looks a lot like a bug to me… @katello

Expected outcome:
Repository being available.

Foreman and Proxy versions:

# rpm -q katello foreman
katello-4.3.1-1.el7.noarch
foreman-3.1.3-1.el7.noarch

Distribution and version:
CentOS 7.9

Other relevant data:
I am wondering if this has something to with the problem in The /etc/yum.repos.d/redhat.repo is missed on CentOS 7 after registering on Foreman server

jeremylenz · June 2, 2022, 3:06pm

Hi @gvde

just to confirm, the ‘Restrict to OS version’ feature isn’t set for this repository, right?

gvde · June 2, 2022, 3:31pm

No. It hasn’t been set and it isn’t set.

Partha_Aji · June 2, 2022, 3:56pm

@gvde → Any patterns on the Missing Repos? Are they RH Repos or Custom?

gvde · June 2, 2022, 4:16pm

@Partha_Aji As mentioned above: it’s a private/local repository in a local product. We have created a product which has one repository for el7 and one for el8. We have a few locally built RPMs which we manually upload into these repositories, i.e. there is no upstream url.

I don’t know if it is relevant, that is no mirroring repository, but I doubt it as I can see the el7 (but still not the el8) repository if I switch to Library/Organizational View.

gvde · June 2, 2022, 4:58pm

Digging a little deeper: I have checked the entitlement certificate on the clients for our private ORG product. This is how it looks like after the forced refresh on the clients where it’s already missing:

# rct cat-cert /etc/pki/entitlement/7348114868866713206.pem

+-------------------------------------------+
        Entitlement Certificate
+-------------------------------------------+

Certificate:
        Path: /etc/pki/entitlement/7348114868866713206.pem
        Version: 3.4
        Serial: 7348114868866713206
        Start Date: 2020-07-03 14:26:57+00:00
        End Date: 2049-12-01 00:00:00+00:00
        Pool ID: 082c9981730a593d01731512c0320072

Subject:
        CN: 98aa2d6239cb488b9d3bf6fd06b3b15f
        O: ORG

Issuer:
        C: DE
        CN: foreman.example.com
...

Product:
        ID: 963877616277
        Name: ORG
        Version: 
        Arch: ALL
        Tags: 
        Brand Type: 
        Brand Name: 

Order:
        Name: ORG
        Number: 
        SKU: 963877616277
        Contract: 
        Account: 
        Service Type: 
        Roles: 
        Service Level: 
        Usage: 
        Add-ons: 
        Quantity: Unlimited
        Quantity Used: 1
        Socket Limit: 
        RAM Limit: 
        Core Limit: 
        Virt Only: False
        Stacking ID: 
        Warning Period: 0
        Provides Management: False

(Yes the name of the product is our ORG name). It’s missing the Authorized Content URLs and Content sections which I can see on the clients which still have the el8 repository.

I have enabled debug logging in candlepin on my main foreman server and capture the logs during creating of that entitlement certificate:

2022-06-02 16:52:01,278 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.audit.EventSinkImpl - Queuing event: Event [id: null, target: ENTITLEMENT, type: MODIFIED, time: 2022-06-02 16:52:01+0200, entity: 43c37940342e4ba69402e6a331151ffd]
2022-06-02 16:52:01,279 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generating entitlement cert for entitlements
2022-06-02 16:52:01,280 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Persisting new certificate serials
2022-06-02 16:52:01,281 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generating entitlement cert for pool: Pool [id: 082c9981730a593d01731512c0320072, type: NORMAL, product: 963877616277, productName: ORG, quantity: -1] quantity: 1 entitlement id: 98aa2d6239cb488b9d3bf6fd06b3b15f
2022-06-02 16:52:01,281 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Consumer has an environment; checking for promoted content in: Environment [id: 1809b04b0444be4df8aa435dd54cc11a, name: Production/el8-epel8, owner: Owner [id: 082c9981720833470172083368ac0001, key: ORG]]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cbbd01ff, id: 1589352123313, name: PostgreSQL 12 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg12-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cc25021f, id: 1589352127979, name: PostgreSQL 10 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg10-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e250083d019a, id: 1653064272568, name: AlmaLinux-8 - Real Time x86_64 os, label: ORG_almalinux8_plus_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017fbffed9e502e6, id: 1623228212554, name: CentOS-8 Stream - extras x86_64, label: ORG_centos8stream_extras_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cbf3020f, id: 1589352125594, name: PostgreSQL 11 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg11-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018074b7bed30320, id: 1589353644182, name: Extra Packages for Enterprise Linux Modular 8 - x86_64, label: ORG_epel8_epel-modular]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cfa2029a, id: 1592404126102, name: Backup Archiving Recovery Open Sourced 19.2 (CentOS_8), label: ORG_bareos_bareos-com_bareos-19_2-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c9981806407dd018065af8c8f04ab, id: 1625663429486, name: AlmaLinux-8 - AppStream x86_64 os, label: ORG_almalinux8_AppStream_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c6700108, id: 1623228210938, name: CentOS-8 Stream - Plus x86_64, label: ORG_centos8stream_centosplus_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6ca7f019f, id: 1589351695000, name: PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg-common-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c61700f8, id: 1623228205631, name: CentOS-8 Stream - AppStream x86_64, label: ORG_centos8stream_AppStream_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c69d0110, id: 1623228208320, name: CentOS-8 Stream - HighAvailability x86_64, label: ORG_centos8stream_HighAvailability_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cfd402a0, id: 1591882452614, name: Elasticsearch repository for 7.x packages, label: ORG_elastic_elasticsearch-7-x]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017fbffeb96002dd, id: 1648193484725, name: CentOS-8 Stream - extras common x86_64, label: ORG_centos8stream_extras_common_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a81176b5603f4, id: 1625663433430, name: AlmaLinux-8 - HighAvailability x86_64 os, label: ORG_almalinux8_HighAvailability_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a81177a8d0410, id: 1625663437330, name: AlmaLinux-8 - extras x86_64 os, label: ORG_almalinux8_extras_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017edd8317f400b1, id: 1644393805078, name: EPEL-Next 8 - x86_64, label: ORG_epel8_next]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017f968b7960016e, id: 1647498197222, name: PostgreSQL Supplementary ucommon RPMs for RHEL / Rocky 8 - x86_64, label: ORG_postgresql_pgdg-centos8-sysupdates]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073ad42830094, id: 1650558752470, name: Foreman client 3.2 el8, label: ORG_foreman-client_3_2_el8_x86_64]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c6430100, id: 1623228209551, name: CentOS-8 Stream - PowerTools x86_64, label: ORG_centos8stream_PowerTools_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017f0848b1d30339, id: 1645111456085, name: CentOS 8 Stream - Ceph Pacific, label: ORG_centos8-stream-storage_ceph-pacific]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c2280083, id: 1601011894902, name: PostgreSQL 13 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg13-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f563e45017f64dee8cf0081, id: 1593787688196, name: ORG el8, label: ORG_org_el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817c362875017c3b4cf0da0066, id: 1633082470381, name: PostgreSQL 14 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg14-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c0bf0052, id: 1623308982040, name: stable EL8, label: ORG_docker-ce_stable_el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6d07802ae, id: 1592822449058, name: Shibboleth (CentOS_8), label: ORG_shibboleth_shibboleth-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c076004e, id: 1604314814110, name: Elasticsearch repository for 6.x packages, label: ORG_elastic_elasticsearch-6-x]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017f0c5fbe44038b, id: 1645180075449, name: CentOS 8 Stream - Ceph Quincy, label: ORG_centos8-stream-storage_ceph-quincy]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817c5e7528017c5e9725b80007, id: 1589353598586, name: Extra Packages for Enterprise Linux 8 - x86_64, label: ORG_epel8_epel]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817d32b20e017d33b4c325005f, id: 1637250024076, name: Foreman client 3.1 el8, label: ORG_foreman-client_3_1_el8_x86_64]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e24fbd530184, id: 1653064276294, name: AlmaLinux-8 - Plus x86_64 os, label: ORG_almalinux8_RT_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073b029f700fa, id: 1650557669831, name: Puppet 7 Repository el 8 - x86_64, label: ORG_puppet_puppet7_el8_x86_64]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017f9688ff740122, id: 1647497907514, name: PostgreSQL extra packages to support some RPMs in the PostgreSQL RPM repo RHEL / Rocky 8 - x86_64, label: ORG_postgresql_pgdg-rhel8-extras]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818045e635018061a41de91538, id: 1589784079717, name: Puppet 6 Repository el 8 - x86_64, label: ORG_puppet_puppet6_el8_x86_64]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a811763c803e9, id: 1625663431505, name: AlmaLinux-8 - BaseOS x86_64 os, label: ORG_almalinux8_BaseOS_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a811773390401, id: 1625663435428, name: AlmaLinux-8 - PowerTools x86_64 os, label: ORG_almalinux8_PowerTools_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e2502b6f01a5, id: 1653064274495, name: AlmaLinux-8 - ResilientStorage x86_64 os, label: ORG_almalinux8_ResilientStorage_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c5e800f0, id: 1623228207122, name: CentOS-8 Stream - BaseOS x86_64, label: ORG_centos8stream_BaseOS_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e24fecab018f, id: 1653064270559, name: AlmaLinux-8 - Real Time for NFV x86_64 os, label: ORG_almalinux8_NFV_x86_64_os]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c4b400d6, id: 1621248407297, name: Backup Archiving Recovery Open Sourced 20 (CentOS 8), label: ORG_bareos_bareos-com_bareos_20-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073af932000e9, id: 1650558367546, name: CentOS Configmanagement SIG - ansible-29 el8, label: ORG_centos-configmanagement_ansible_29_el8_x86_64]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cf770296, id: 1589982020034, name: ICINGA (stable release for epel) EL8, label: ORG_Icinga_icinga-stable-release-el8]
2022-06-02 16:52:01,282 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] INFO  org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Creating X509 cert for product: Product [uuid: 082c99817f563e45017f64dee9240083, id: 963877616277, name: ORG]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Provided products: [Product [uuid: 082c99817f563e45017f64dee9240083, id: 963877616277, name: ORG]]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.util.X509Util - Skipping content not promoted to environment: Content [uuid: 082c99817f563e45017f64dee8cf0080, id: 1593787668737, name: ORG el7, label: ORG_org_el7]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Consumer has an environment; checking for promoted content in: Environment [id: 1809b04b0444be4df8aa435dd54cc11a, name: Production/el8-epel8, owner: Owner [id: 082c9981720833470172083368ac0001, key: ORG]]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cbbd01ff, id: 1589352123313, name: PostgreSQL 12 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg12-el8]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cc25021f, id: 1589352127979, name: PostgreSQL 10 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg10-el8]
2022-06-02 16:52:01,283 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e250083d019a, id: 1653064272568, name: AlmaLinux-8 - Real Time x86_64 os, label: ORG_almalinux8_plus_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017fbffed9e502e6, id: 1623228212554, name: CentOS-8 Stream - extras x86_64, label: ORG_centos8stream_extras_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cbf3020f, id: 1589352125594, name: PostgreSQL 11 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg11-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018074b7bed30320, id: 1589353644182, name: Extra Packages for Enterprise Linux Modular 8 - x86_64, label: ORG_epel8_epel-modular]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cfa2029a, id: 1592404126102, name: Backup Archiving Recovery Open Sourced 19.2 (CentOS_8), label: ORG_bareos_bareos-com_bareos-19_2-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c9981806407dd018065af8c8f04ab, id: 1625663429486, name: AlmaLinux-8 - AppStream x86_64 os, label: ORG_almalinux8_AppStream_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c6700108, id: 1623228210938, name: CentOS-8 Stream - Plus x86_64, label: ORG_centos8stream_centosplus_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6ca7f019f, id: 1589351695000, name: PostgreSQL common RPMs for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg-common-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c61700f8, id: 1623228205631, name: CentOS-8 Stream - AppStream x86_64, label: ORG_centos8stream_AppStream_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c69d0110, id: 1623228208320, name: CentOS-8 Stream - HighAvailability x86_64, label: ORG_centos8stream_HighAvailability_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cfd402a0, id: 1591882452614, name: Elasticsearch repository for 7.x packages, label: ORG_elastic_elasticsearch-7-x]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017fbffeb96002dd, id: 1648193484725, name: CentOS-8 Stream - extras common x86_64, label: ORG_centos8stream_extras_common_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a81176b5603f4, id: 1625663433430, name: AlmaLinux-8 - HighAvailability x86_64 os, label: ORG_almalinux8_HighAvailability_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a81177a8d0410, id: 1625663437330, name: AlmaLinux-8 - extras x86_64 os, label: ORG_almalinux8_extras_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017edd8317f400b1, id: 1644393805078, name: EPEL-Next 8 - x86_64, label: ORG_epel8_next]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017f968b7960016e, id: 1647498197222, name: PostgreSQL Supplementary ucommon RPMs for RHEL / Rocky 8 - x86_64, label: ORG_postgresql_pgdg-centos8-sysupdates]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073ad42830094, id: 1650558752470, name: Foreman client 3.2 el8, label: ORG_foreman-client_3_2_el8_x86_64]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c6430100, id: 1623228209551, name: CentOS-8 Stream - PowerTools x86_64, label: ORG_centos8stream_PowerTools_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017f0848b1d30339, id: 1645111456085, name: CentOS 8 Stream - Ceph Pacific, label: ORG_centos8-stream-storage_ceph-pacific]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c2280083, id: 1601011894902, name: PostgreSQL 13 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg13-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f563e45017f64dee8cf0081, id: 1593787688196, name: ORG el8, label: ORG_org_el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817c362875017c3b4cf0da0066, id: 1633082470381, name: PostgreSQL 14 for RHEL/CentOS 8 - x86_64, label: ORG_postgresql_pgdg14-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c0bf0052, id: 1623308982040, name: stable EL8, label: ORG_docker-ce_stable_el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6d07802ae, id: 1592822449058, name: Shibboleth (CentOS_8), label: ORG_shibboleth_shibboleth-el8]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c076004e, id: 1604314814110, name: Elasticsearch repository for 6.x packages, label: ORG_elastic_elasticsearch-6-x]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817eca8d18017f0c5fbe44038b, id: 1645180075449, name: CentOS 8 Stream - Ceph Quincy, label: ORG_centos8-stream-storage_ceph-quincy]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817c5e7528017c5e9725b80007, id: 1589353598586, name: Extra Packages for Enterprise Linux 8 - x86_64, label: ORG_epel8_epel]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817d32b20e017d33b4c325005f, id: 1637250024076, name: Foreman client 3.1 el8, label: ORG_foreman-client_3_1_el8_x86_64]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e24fbd530184, id: 1653064276294, name: AlmaLinux-8 - Plus x86_64 os, label: ORG_almalinux8_RT_x86_64_os]
2022-06-02 16:52:01,284 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073b029f700fa, id: 1650557669831, name: Puppet 7 Repository el 8 - x86_64, label: ORG_puppet_puppet7_el8_x86_64]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817f734f35017f9688ff740122, id: 1647497907514, name: PostgreSQL extra packages to support some RPMs in the PostgreSQL RPM repo RHEL / Rocky 8 - x86_64, label: ORG_postgresql_pgdg-rhel8-extras]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818045e635018061a41de91538, id: 1589784079717, name: Puppet 6 Repository el 8 - x86_64, label: ORG_puppet_puppet6_el8_x86_64]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a811763c803e9, id: 1625663431505, name: AlmaLinux-8 - BaseOS x86_64 os, label: ORG_almalinux8_BaseOS_x86_64_os]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80d3c4017a811773390401, id: 1625663435428, name: AlmaLinux-8 - PowerTools x86_64 os, label: ORG_almalinux8_PowerTools_x86_64_os]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e2502b6f01a5, id: 1653064274495, name: AlmaLinux-8 - ResilientStorage x86_64 os, label: ORG_almalinux8_ResilientStorage_x86_64_os]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c5e800f0, id: 1623228207122, name: CentOS-8 Stream - BaseOS x86_64, label: ORG_centos8stream_BaseOS_x86_64_os]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c998180d04c0e0180e24fecab018f, id: 1653064270559, name: AlmaLinux-8 - Real Time for NFV x86_64 os, label: ORG_almalinux8_NFV_x86_64_os]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6c4b400d6, id: 1621248407297, name: Backup Archiving Recovery Open Sourced 20 (CentOS 8), label: ORG_bareos_bareos-com_bareos_20-el8]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99818069662a018073af932000e9, id: 1650558367546, name: CentOS Configmanagement SIG - ansible-29 el8, label: ORG_centos-configmanagement_ansible_29_el8_x86_64]
2022-06-02 16:52:01,285 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter -   promoted content: Content [uuid: 082c99817a80bc55017a80c6cf770296, id: 1589982020034, name: ICINGA (stable release for epel) EL8, label: ORG_Icinga_icinga-stable-release-el8]
2022-06-02 16:52:01,288 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Getting PEM encoded cert.
2022-06-02 16:52:01,288 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generating v3 entitlement data
2022-06-02 16:52:01,291 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Generated cert serial number: 7348114868866713206
2022-06-02 16:52:01,291 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Key: -----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

2022-06-02 16:52:01,291 [thread=http-bio-127.0.0.1-23443-exec-6] [req=5c7a9bf4-4011-404d-b132-4659d6818e7d, org=ORG, csid=] DEBUG org.candlepin.service.impl.DefaultEntitlementCertServiceAdapter - Cert: -----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN ENTITLEMENT DATA-----
...
-----END ENTITLEMENT DATA-----
-----BEGIN RSA SIGNATURE-----
...
-----END RSA SIGNATURE-----

Which all looks good, too. It lists all repositories in the content view of the client including the missing repo:

promoted content: Content [uuid: 082c99817f563e45017f64dee8cf0081, id: 1593787688196, name: ORG el8, label: ORG_org_el8]

It skips the el7 repository which is belongs to the ORG product but is not in the content view:

Skipping content not promoted to environment: Content [uuid: 082c99817f563e45017f64dee8cf0080, id: 1593787668737, name: ORG el7, label: ORG_org_el7]

So those logs look good and I can see that logs follow the same logic for the other subscribed products of the client.

So question is: why doesn’t candlepin add the ORG_org_el8 repository to the entitlement certificate? And why now?

Dirk · June 3, 2022, 9:29am

It reminds me of 1931027 – Entitlement certificate is missing content section for a custom product perhaps you are hitting this bug, too.

gvde · June 3, 2022, 2:07pm

Yes. That seems to be the problem. Content of the candlepin database is messed up.

candlepin=# SELECT cp2_content.uuid, cp2_content.content_id, cp2_content.contenturl, cp2_content.label, cp2_content.name, cp2_content.entity_version, cp2_product_content.product_uuid,
       cp2_product_content.enabled, cp2_product_content.id,
       cp2_products.name
FROM cp2_content LEFT JOIN cp2_product_content ON cp2_content.uuid = cp2_product_content.content_uuid
                 LEFT JOIN cp2_products ON cp2_product_content.product_uuid = cp2_products.uuid
WHERE label LIKE 'ORG_org%' or cp2_products.uuid IS NULL;
               uuid               |  content_id   |            contenturl             |             label              |   name    | entity_version |           product_uuid           | enabled |                id                | name 
----------------------------------+---------------+-----------------------------------+--------------------------------+-----------+----------------+----------------------------------+---------+----------------------------------+------
 082c99817f563e45017f64dee8cf0080 | 1593787668737 | /custom/org/el7                  | ORG_org_el7                  | ORG el7  |     1969237048 | 082c99817f563e45017f64dee9240083 | t       | 082c99817f563e45017f64dee9250084 | ORG
 082c99817a80bc55017a80c6d76f03aa | 1596726054675 | /custom/centos7-storage/samba-411 | ORG_centos7-storage_samba-411 | samba-411 |     1531276742 |                                  |         |                                  |  
 082c99817f563e45017f64dee8cf0081 | 1593787688196 | /custom/org/el8                  | ORG_org_el8                  | ORG el8  |    -1061146451 |                                  |         |                                  | 
 082c9981812506c601812902ec5b0046 | 1593787688196 | /custom/org/el8                  | ORG_org_el8                  | ORG el8  |    -1867196716 |                                  |         |                                  | 
(4 rows)

It seems the cp2_product_content link between the product ORG and the content ORG_org_el8 got lost somehow. That way I also find another repository with the same issue: ORG_centos7-storage_samba-411. I just verified and it fits: an el7 client can get any other repository from the storage product but not the samba-411.

I’ll try to fix it in the database. However, I currently have a duplicated ORG_org_el8 content entry which must have been created when I set the arch restriction of the repository to x86_64 as a test.

I wonder how katello references the candlepin contents so I know which of the redundant entries I can remove…

gvde · June 3, 2022, 2:26pm

O.K. Fixing the database and a subscription-manager refresh --force on the client makes the repository available again.

candlepin=# INSERT INTO cp2_product_content (product_uuid,content_uuid,enabled,id) VALUES ('082c99817f563e45017f64dee9240083','082c9981812506c6018129defeeb005e', 't', '082c998180efbfee0180fba9d0ac0059');
INSERT 0 1
candlepin=# SELECT cp2_content.uuid, cp2_content.content_id, cp2_content.contenturl, cp2_content.label, cp2_content.name, cp2_content.entity_version, cp2_product_content.product_uuid,
       cp2_product_content.enabled, cp2_product_content.id,
       cp2_products.name
FROM cp2_content LEFT JOIN cp2_product_content ON cp2_content.uuid = cp2_product_content.content_uuid
                 LEFT JOIN cp2_products ON cp2_product_content.product_uuid = cp2_products.uuid
WHERE label LIKE 'DKRZ_dkrz%' or cp2_products.uuid IS NULL;
               uuid               |  content_id   |            contenturl             |             label              |   name    | entity_version |           product_uuid           | enabled |                id                | name 
----------------------------------+---------------+-----------------------------------+--------------------------------+-----------+----------------+----------------------------------+---------+----------------------------------+------
 082c99817f563e45017f64dee8cf0080 | 1593787668737 | /custom/dkrz/el7                  | DKRZ_dkrz_el7                  | DKRZ el7  |     1969237048 | 082c99817f563e45017f64dee9240083 | t       | 082c99817f563e45017f64dee9250084 | DKRZ
 082c9981812506c6018129defeeb005e | 1593787688196 | /custom/dkrz/el8                  | DKRZ_dkrz_el8                  | DKRZ el8  |    -1061146451 | 082c99817f563e45017f64dee9240083 | t       | 082c998180efbfee0180fba9d0ac0059 | DKRZ
 082c99817a80bc55017a80c6d76f03aa | 1596726054675 | /custom/centos7-storage/samba-411 | DKRZ_centos7-storage_samba-411 | samba-411 |     1531276742 |                                  |         |                                  | 
 082c99817f563e45017f64dee8cf0081 | 1593787688196 | /custom/dkrz/el8                  | DKRZ_dkrz_el8                  | DKRZ el8  |    -1061146451 |                                  |         |                                  | 
 082c9981812506c601812902ec5b0046 | 1593787688196 | /custom/dkrz/el8                  | DKRZ_dkrz_el8                  | DKRZ el8  |    -1867196716 |                                  |         |                                  | 
(5 rows)

I have also deleted the redundant entry after checking that it’s not referenced anywhere else.

candlepin=# delete from cp2_content where uuid = '082c99817f563e45017f64dee8cf0081';
DELETE 1
candlepin=# delete from cp2_content where uuid = '082c9981812506c601812902ec5b0046';
DELETE 1

and fixed the other repository:

candlepin=# INSERT INTO cp2_product_content (product_uuid,content_uuid,enabled,id) VALUES ('082c99817a80bc55017a80c6d747039d','082c99817a80bc55017a80c6d76f03aa', 't', '082c998180efbfee0180fba9d0ac005a');
INSERT 0 1

It is however bothering me how this could happen. It was working fine till recently. We did not make any changes to the product or repository. We didn’t even upload any content into the repository. How can the database loose the row in the table???

Dirk · June 9, 2022, 3:51pm

This is the big question here which noone found an answer for yet.

wbclark · June 14, 2022, 1:30pm

Hi @gvde , I assume you will see my post on the other support thread, but in case someone finds it here, I have the same requests for data if you can determine the source original modification to these product contents in Candlepin via either Candlepin’s logs or anything in foreman-tasks in the matching date/time range:

For the custom product(s) affected, are there any foreman tasks in the task history specifically related to these products, from the time period where the issue may have occurred?
Can you find any requests referencing those affected productions in the candlepin logs, within the time/date range after the repos were known working, and before they were known broken?

This remediation should work with just the product ID from Katello. In this case it’s product ID 37:

cat << EOF |foreman-rake console
root = Katello::RootRepository.find(37);
ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: root.content_id)
EOF

Wait for the task the complete, then try subscription-manager refresh on the affected content hosts.

Please let us know if that resolves it, and additionally, if there is any indication what is the original source of the change, we’d like to identify that and correct it.

Thanks again for your help on the other related thread,

gvde · June 14, 2022, 1:48pm

There are no tasks. The repository which we found missing is a local upload only repository which we use to provide custom built rpms. There are not tasks, no sync, nothing like that.

We don’t really know how long this situation occurred. It could be days or weeks. We have only noticed at the moment when we uploaded a new rpm and it wasn’t available on a client where we needed it. It’s been a while since we last uploaded something before that, so I don’t really know where to look or what to look for.

George · December 7, 2022, 1:44pm

Found myself having the same issue. Running the command for each repo that was not showing up properly fixed it. Thanks @wbclark !

hjb · June 19, 2023, 12:43pm

I have the same problem. The console command did fix some of the repos but for some I get:

root = Katello::RootRepository.find(76);
Traceback (most recent call last):
        2: from lib/tasks/console.rake:5:in `block in '
        1: from (irb):38
ActiveRecord::RecordNotFound (Couldn't find Katello::RootRepository with 'id'=76)

I looked in katello_root_repositories and the repo is there, and I can’t see relevant differences to other entries which are found. What could prevent find() from finding them? I don’t know where find() is implemented so I cannot look myself.

jeremylenz · June 20, 2023, 12:15pm

That’s odd…

These methods come from Ruby on Rails.

You can also try

Katello::RootRepository.where(id: 76)

or

Katello::RootRepository.where(id: 76).first

And as a side note, .where also allows .to_sql, so you can do

[4] pry(main)> Katello::RootRepository.where(id: 76).to_sql
=> "SELECT \"katello_root_repositories\".* FROM \"katello_root_repositories\" WHERE \"katello_root_repositories\".\"id\" = 76"

I don’t know why find wouldn’t find a table row, if it actually exists.

hjb · June 26, 2023, 10:31am

Thanks for your hints. I didn’t have much time for almost a week. Now I’ve checked katello_root_repositories and the entry 76 actually doesn’t exist. 76 is in katello_repositories and its corresponding root_id is 21. I don’t see an inconsistency there.

That doesn’t solve the problem, however. I think we have missing entries in Candlepin in cp2_product_content. Maybe I can add these manually. But I’d rather delete the Candlepin database completely if there is a way to have it rebuilt automatically.

areyus · July 17, 2023, 1:33pm

I ran into the same issue today and went with @gvde’s solution of inserting the values manually since the foreman-rake commands did not work for me.

The spawned task by the rake command ended with an error: Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433, which makes me fear there are more things broken in the DB.

The only thing I was unsure about is how the id value of the cp2_product_content table needs to be filled. Since the other repos of that product had ids very close together, I took the risk of just selecting an unused id boardering the other ones. There seems to be no DB logic involved, though.
@gvde Do you remember how you picked the ID?

Also, I believe we have encountered this problem in the past, too, and back then just went with recreating the repository in question. Due to us keeping some old CV versions, we could not yet delete the last instance of a repo going missing, and the old affected repo also is missing that entry in cp2_product_content.

For the record, we had this happen on Katello 4.4/Foreman 3.2 on RHEL8.