Entitlement certificate not containing all content

Dirk · October 6, 2020, 3:25pm

Problem: The initial problem was described to me as the SLES systems not getting all their repositories and narrowed it down to it does not get them because they are not included in the entitlement certificates.

One example which I had to anonymize:

# rct cat-cert /etc/pki/entitlement/1151553941477025897.pem 

+-------------------------------------------+
	Entitlement Certificate
+-------------------------------------------+

Certificate:
	Path: /etc/pki/entitlement/1151553941477025897.pem
	Version: 3.4
	Serial: 1151553941477025897
	Start Date: 2020-04-29 09:11:28+00:00
	End Date: 2049-12-01 00:00:00+00:00
	Pool ID: 2c96813b7157c48b0171c53490451e6c

Subject:
	CN: 8523b71cac4c4643aa4ad3a6337400da
	O: ORGANIZATION

Issuer:
	C: US
	CN: CASERVER
	L: Raleigh
	O: Katello
	OU: SomeOrgUnit
	ST: North Carolina

Product:
	ID: 186245269223
	Name: 1220 Public Cloud Module 12 x86_64
	Version: 
	Arch: ALL
	Tags: 
	Brand Type: 
	Brand Name: 

Order:
	Name: 1220 Public Cloud Module 12 x86_64
	Number: 
	SKU: 186245269223
	Contract: 
	Account: 
	Service Level: 
	Service Type: 
	Quantity: Unlimited
	Quantity Used: 1
	Socket Limit: 
	RAM Limit: 
	Core Limit: 
	Virt Only: False
	Stacking ID: 
	Warning Period: 0
	Provides Management: False

Content:
	Type: yum
	Name: 1220 Public Cloud Module 12 x86_64 SLE-Module-Public-Cloud12-Pool for sle-12-x86_64
	Label: ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64
	Vendor: Custom
	URL: /ORGANIZATION/Library/SLES/custom/1220_Public_Cloud_Module_12_x86_64/1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64
	GPG: ../../katello/api/v2/repositories/22470/gpg_key_content
	Enabled: True
	Expires: 1
	Required Tags: 
	Arches: x86_64

For the host at https://FOREMANSERVER/rhsm/consumers/9a1f843c-b83e-43e9-8c21-db631424c699 it looks like:

{
...
        "environmentContent": [
...
            {
                "content": {
                    "created": "2020-04-29T09:11:43+0000",
                    "updated": "2020-07-17T08:18:52+0000",
                    "uuid": "2c96813b72e970f001735bdacb8851c2",
                    "id": "1588151503687",
                    "type": "yum",
                    "label": "ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64",
                    "name": "1220 Public Cloud Module 12 x86_64 SLE-Module-Public-Cloud12-Pool for sle-12-x86_64",
                    "vendor": "Custom",
                    "contentUrl": "/custom/1220_Public_Cloud_Module_12_x86_64/1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64",
                    "requiredTags": null,
                    "gpgUrl": "../../katello/api/v2/repositories/22470/gpg_key_content",
                    "modifiedProductIds": [ ],
                    "arches": "x86_64",
                    "requiredProductIds": [ ],
                    "metadataExpire": 1,
                    "releaseVer": null
                },
                "enabled": null
            },
            {
                "content": {
                    "created": "2020-04-29T09:11:53+0000",
                    "updated": "2020-07-17T08:18:52+0000",
                    "uuid": "2c96813b72e970f001735bdacb8851c3",
                    "id": "1588151513954",
                    "type": "yum",
                    "label": "ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Updates_for_sle-12-x86_64",
                    "name": "1220 Public Cloud Module 12 x86_64 SLE-Module-Public-Cloud12-Updates for sle-12-x86_64",
                    "vendor": "Custom",
                    "contentUrl": "/custom/1220_Public_Cloud_Module_12_x86_64/1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Updates_for_sle-12-x86_64",
                    "requiredTags": null,
                    "gpgUrl": "../../katello/api/v2/repositories/22491/gpg_key_content",
                    "modifiedProductIds": [ ],
                    "arches": "x86_64",
                    "requiredProductIds": [ ],
                    "metadataExpire": 1,
                    "releaseVer": null
                },
                "enabled": null
            },
...
        ]
    },
    "entitlementCount": 11,
}

And the content_overrides at https://FOREMANSERVER/rhsm/consumers/9a1f843c-b83e-43e9-8c21-db631424c699/content_overrides/ (The value gets set for all available repositories via script because of limitations in the subscription-manager version)

[
...
  {

    "created": "2020-09-16T07:19:08+0000",
    "updated": "2020-09-16T07:19:08+0000",
    "contentLabel": "ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64",
    "name": "repo_gpgcheck",
    "value": "0"
  },
...
]

I do not know how the entitlement certificate is generated, but while debugging I tried an update to the latest subscription manager which did not fix the problem, so I think the problem is on the server side.

I could not see the same problem on a RHEL system, so only SLES is affected.

Expected outcome:
The system gets all its repositories. In this example pool and updates for this product, but there are more examples.

Foreman and Proxy versions:
1.24.3

Foreman and Proxy plugin versions:
katello 3.14.1
foreman_scc_manager 1.8.3

Distribution and version:
RHEL 7.9

Dirk · October 7, 2020, 7:29am

I have digged deeper and now I think it is missing a database entry.

# select * from cp2_content where name like '%1220%';
               uuid               |  content_id   |          created           |          updated           |                                                            contenturl                                                           
  |                         gpgurl                          |                                                               label                                                                | metadataexpire |                           
               name                                          | releasever | requiredtags | type | vendor | arches | entity_version | locked 
----------------------------------+---------------+----------------------------+----------------------------+---------------------------------------------------------------------------------------------------------------------------------
--+---------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------+----------------+---------------------------
-------------------------------------------------------------+------------+--------------+------+--------+--------+----------------+--------
 2c96813b72e970f001735bdacb8851c3 | 1588151513954 | 2020-04-29 11:11:53.886+02 | 2020-07-17 10:18:52.424+02 | /custom/1220_Public_Cloud_Module_12_x86_64/1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Updates_for_sle-12-x86_6
4 | ../../katello/api/v2/repositories/22491/gpg_key_content | ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Updates_for_sle-12-x86_64 |              1 | 1220 Public Cloud Module 1
2 x86_64 SLE-Module-Public-Cloud12-Updates for sle-12-x86_64 |            |              | yum  | Custom | x86_64 |     1897444354 |      0
 2c96813b72e970f001735bdacb8851c2 | 1588151503687 | 2020-04-29 11:11:43.642+02 | 2020-07-17 10:18:52.424+02 | /custom/1220_Public_Cloud_Module_12_x86_64/1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64  
  | ../../katello/api/v2/repositories/22470/gpg_key_content | ORGANIZATION_1220_Public_Cloud_Module_12_x86_64_1220_Public_Cloud_Module_12_x86_64_SLE-Module-Public-Cloud12-Pool_for_sle-12-x86_64    |              1 | 1220 Public Cloud Module 1
2 x86_64 SLE-Module-Public-Cloud12-Pool for sle-12-x86_64    |            |              | yum  | Custom | x86_64 |     2128759327 |      0

# select * from cp2_product_content where product_uuid='2c96813b72e970f001735bdacb8d51c4';
           product_uuid           |           content_uuid           | enabled |          created           |          updated           |                id                
----------------------------------+----------------------------------+---------+----------------------------+----------------------------+----------------------------------
 2c96813b72e970f001735bdacb8d51c4 | 2c96813b72e970f001735bdacb8851c2 | t       | 2020-07-17 10:18:52.429+02 | 2020-07-17 10:18:52.429+02 | 2c96813b72e970f001735bdacb8d51c6

For those repositories working, I see a matching number of entries in cp2_product_content.

m-bucher · July 26, 2021, 1:11pm

Maybe a little late for the party, but we had this issue on some of our systems now and created a script that will look for these missing entries in cp2_product_content and can regenerate the missing entries:

Unfortunately, we also were not able to detect the root-cause that removes those entries
The issue has been reported on Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1931027

Dirk · July 26, 2021, 1:43pm

Thanks, I add the information to the bugzilla entry.

ElCoyote27 · December 2, 2022, 9:00pm

Thank you very much for the script. I added this patch to run it on Satellite 6.11 (still on ruby 2.0):

[root@sat6 ~]# diff -u find_missing_candlepin_product_contents.rb.orig find_missing_candlepin_product_contents.rb
--- find_missing_candlepin_product_contents.rb.orig     2022-12-02 15:22:31.192117015 -0500
+++ find_missing_candlepin_product_contents.rb  2022-12-02 15:21:06.200541724 -0500
@@ -10,6 +10,14 @@
 
 require 'set'
 
+unless [].respond_to? :to_h
+  class Array
+    def to_h
+      Hash[self]
+    end
+  end
+end
+
 def query_database(dbname, sql)
   res = []
   IO.popen("su - postgres -c 'psql #{dbname} -t -A -z'", 'w+') do |io|

Unfortunately, even after running it, there are a few inconsistencies left:

[root@sat6 ~]# ruby ./find_missing_candlepin_product_contents.rb 2>&1|tail -6

Product "GitLab Runner"(cp_id=932577792545) has 2 content in foremanDB, but 3 content in candlepinDB
Product "Dell OMSA"(cp_id=757933989515) has 2 content in foremanDB, but 3 content in candlepinDB
Product "Extra Packages"(cp_id=256062657870) has 12 content in foremanDB, but 24 content in candlepinDB

m-bucher · January 11, 2023, 1:11pm

Hi @ElCoyote27,

The fact that there are more entries in candlepinDB for an id than in foremanDB is something I also observed. This might be OK though I have not analyzed this sufficiently to prove it

Feel free to open a PR on github to get your fix into the script.
Have you tried to run the script in the tfm-ruby interpreter, which should be a newer ruby-version?