Product disappeared from subscriptions

Support
1

I recently upgraded by Forman/Katello installation to 3.4/4.6 running on Oracle Linux 8.

After syncing my repositories and publishing new non-composite content views, but not new composite views, a product disappeared from the client’s subscriptions.

The details:

I have a product named “CentOS”. The product has 7 repositories:

# hammer product info --organization=Curia --name CentOS
Id:                21
Name:              CentOS
Label:             centos
Description:       
Sync State (all):  Syncing Complete.
Sync State (last): Syncing Complete.
Sync Plan ID:      
GPG:               
    GPG Key ID: 
    GPG Key:
Organization:      Curia
Content:           
 1) Repo Name:    centos6_contrib
    Url:          /custom/centos/6_contrib
    Content Type: yum
 2) Repo Name:    centos7
    Url:          /custom/centos/7
    Content Type: yum
 3) Repo Name:    centos7_extras
    Url:          /custom/centos/7_extras
    Content Type: yum
 4) Repo Name:    centos7_plus
    Url:          /custom/centos/7_plus
    Content Type: yum
 5) Repo Name:    centos7_updates
    Url:          /custom/centos/7_updates
    Content Type: yum
 6) Repo Name:    centos8_baseos
    Url:          /custom/centos/8_baseos
    Content Type: yum
 7) Repo Name:    centos9_baseos
    Url:          /custom/centos/9_baseos
    Content Type: yum

I have a non-composite content view named “z_centos7” which contains four of those repositories, published to Library:

hammer content-view info --organization Curia --name "z_centos7"
Id:                           79
Name:                         z_centos7
Label:                        centos7
Composite:                    false
Description:                  
Content Host Count:           0
Solve Dependencies:           false
Organization:                 Curia
Yum Repositories:             
 1) Id:    931
    Name:  centos7
    Label: 7
 2) Id:    932
    Name:  centos7_extras
    Label: 7_extras
 3) Id:    933
    Name:  centos7_plus
    Label: 7_plus
 4) Id:    934
    Name:  centos7_updates
    Label: 7_updates
Container Image Repositories: 

OSTree Repositories:          

Lifecycle Environments:       
 1) Id:   7
    Name: Library

I have a composite content view named “CentOS 7” which contains the “z_centos7” and others:

# hammer content-view info --organization Curia --name "CentOS 7"
Id:                           127
Name:                         CentOS 7
Label:                        centos_7
Composite:                    true
Description:                  
Content Host Count:           81
Solve Dependencies:           false
Organization:                 Curia
Yum Repositories:             
 1) Id:    4605
    Name:  docker_ce_el7
    Label: el7
 2) Id:    4621
    Name:  elrepo_el7
    Label: el7
 3) Id:    4622
    Name:  elrepo_el7_extras
    Label: el7_extras
 4) Id:    4630
    Name:  epel_el7
    Label: el7
 5) Id:    4640
    Name:  foreman_el7_plugins
    Label: el7_plugins
 6) Id:    4639
    Name:  foreman_el7_client
    Label: el7_client
 7) Id:    4647
    Name:  megaraid_megacli
    Label: megacli
 8) Id:    4662
    Name:  oracle_java_jre
    Label: jre
 9) Id:    4679
    Name:  puppet_el7
    Label: el7
 10)Id:    4688
    Name:  r1soft_stable
    Label: stable
 11)Id:    4702
    Name:  supermicro_ipmicfg
    Label: ipmicfg
 12)Id:    4728
    Name:  zabbix_el7
    Label: el7
 13)Id:    3435
    Name:  mogodb_el7
    Label: el7
 14)Id:    4611
    Name:  duo_el7
    Label: el7
 15)Id:    4671
    Name:  powershell_el7
    Label: el7
 16)Id:    4908
    Name:  centos7_plus
    Label: 7_plus
 17)Id:    4907
    Name:  centos7_extras
    Label: 7_extras
 18)Id:    4909
    Name:  centos7_updates
    Label: 7_updates
 19)Id:    4906
    Name:  centos7
    Label: 7

I have a host, tdc1cos7.albmolecular.com, whose content view is CentOS 7. The web gui lists is as subscribed to the product “CentOS” but hammer does not. Additionally, while “hammer host subscription product-content” shows that the repositories are available to the client, and the “hammer host subscription attach” command succeeds, it does not change the subscriptions.

# hammer host subscription enabled-repositories --host tdc1cos7.albmolecular.com
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------
ID   | NAME               | LABEL      | CONTENT TYPE | CONTENT VIEW ID | CONTENT VIEW NAME | CONTENT VIEW VERSION | ENVIRONMENT NAME | PRODUCT NAME
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------
4813 | powershell_el7     | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | PowerShell  
1461 | oracle_java_jre    | jre        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Oracle Java 
1454 | supermicro_ipmicfg | ipmicfg    | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Supermicro  
1458 | elrepo_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | ELRepo      
1453 | megaraid_megacli   | megacli    | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | MegaRAID    
1450 | docker_ce_el7      | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Docker CE   
1459 | elrepo_el7_extras  | el7_extras | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | ELRepo      
1452 | puppet_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Puppet      
1449 | r1soft_stable      | stable     | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | r1Soft      
1463 | epel_el7           | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | EPEL        
1451 | foreman_el7_client | el7_client | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Foreman     
1460 | zabbix_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Zabbix      
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------
# hammer host subscription product-content --host tdc1cos7.albmolecular.com
----|-------------------|-------------------------|------------------|---------
ID  | NAME              | LABEL                   | DEFAULT ENABLED? | OVERRIDE
----|-------------------|-------------------------|------------------|---------
82  | centos6_contrib   | curia_centos_6_contrib  | yes              |         
83  | centos7           | curia_centos_7          | yes              |         
84  | centos7_extras    | curia_centos_7_extras   | yes              |         
85  | centos7_plus      | curia_centos_7_plus     | yes              |         
86  | centos7_updates   | curia_centos_7_updates  | yes              |         
87  | centos8_baseos    | curia_centos_8_baseos   | yes              |         
200 | centos9_baseos    | curia_centos_9_baseos   | yes              |         
88  | docker_ce_el7     | curia_docker_ce_el7     | yes              |         
89  | docker_ce_el8     | curia_docker_ce_el8     | yes              |         
201 | docker_ce_el9     | curia_docker_ce_el9     | yes              |         
90  | elrepo_el7        | curia_elrepo_el7        | yes              |         
91  | elrepo_el7_extras | curia_elrepo_el7_extras | yes              |         
92  | elrepo_el8        | curia_elrepo_el8        | yes              |         
93  | elrepo_el8_extras | curia_elrepo_el8_extras | yes              |         
196 | elrepo_el9        | curia_elrepo_el9        | yes              |         
202 | elrepo_el9_extras | curia_elrepo_el9_extras | yes              |         
96  | epel_el6          | curia_epel_el6          | yes              |         
97  | epel_el7          | curia_epel_el7          | yes              |         
98  | epel_el8          | curia_epel_el8          | yes              |         
194 | epel_el9          | curia_epel_el9          | yes              |         
----|-------------------|-------------------------|------------------|---------
Page 1 of 3 (use --page and --per-page for navigation).
# hammer subscription list --organization Curia | grep CentOS
21 | 8a8a82247a6786c8017ab06481b3038f | CentOS               | Physical |          |         |         | 2021/07/16 17:36:54 | 2049/12/01 00:00:00 | Unlimited | 325
# hammer host subscription attach --host tdc1cos7.albmolecular.com --subscription-id 21
Subscription attached to the host successfully.
# hammer host subscription enabled-repositories --host tdc1cos7.albmolecular.com
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------
ID   | NAME               | LABEL      | CONTENT TYPE | CONTENT VIEW ID | CONTENT VIEW NAME | CONTENT VIEW VERSION | ENVIRONMENT NAME | PRODUCT NAME
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------
4813 | powershell_el7     | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | PowerShell  
1461 | oracle_java_jre    | jre        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Oracle Java 
1454 | supermicro_ipmicfg | ipmicfg    | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Supermicro  
1458 | elrepo_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | ELRepo      
1453 | megaraid_megacli   | megacli    | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | MegaRAID    
1450 | docker_ce_el7      | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Docker CE   
1459 | elrepo_el7_extras  | el7_extras | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | ELRepo      
1452 | puppet_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Puppet      
1449 | r1soft_stable      | stable     | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | r1Soft      
1463 | epel_el7           | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | EPEL        
1451 | foreman_el7_client | el7_client | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Foreman     
1460 | zabbix_el7         | el7        | yum          | 127             | CentOS 7          | CentOS 7 20.0        | Test             | Zabbix      
-----|--------------------|------------|--------------|-----------------|-------------------|----------------------|------------------|-------------

I have done complete syncs of all of the CentOS repositories, republished, and even created a new non-composite content view with the same repositories, and a new composite view with only that view. None of this has allowed the clients to subscribe.

I’m out of ideas.

2

I should add a bit more information:

The upgrade was from 3.3/4.5 to 3.4/4.6.

Before the upgrade, the clients listed the centos7 repositories in the output of subscription-manage repos

After the upgrade, after running subscription-manager refresh, the centos7 repositories were still listed in the output of subscription-manager repos.

After syncing the repositories, after running subscription-manager refresh, the centos7 repositories were still listed in the output of subscription-manager repos.

After publishing a new version of the non-composite content view, and not touching the composite view at all, the repositories were no longer listed in the output of subscription-manager repos.

3

Not sure, but it sounds like you are hitting the bug https://bugzilla.redhat.com/show_bug.cgi?id=1931027

Can you verify this and add to the bug if you have some additional findings?

4

Thanks!

Running the following found nine affected repos, and corrected all but two of them:

# sudo -u postgres psql candlepin
could not change directory to "/root": Permission denied
psql (12.11)
Type "help" for help.

candlepin=# SELECT content.* from cp2_content content LEFT JOIN cp2_product_content pc on content.uuid = pc.content_uuid where pc.content_uuid IS NULL;

# foreman-rake console
Loading production environment (Rails 6.1.6.1)
irb(main):001:0> roots = Katello::RootRepository.where(content_id: ['1626457734930', '1626457224897', '1626457524855', '1626459954900', '1626459504897', '1626457224897', '1626457524855', '
1626457584896', '1626457644865'])
irb(main):001:0> roots.each{|root| ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: ro
ot.content_id) }

I’m left with two that are still missing:

# foreman-rake console
Loading production environment (Rails 6.1.6.1)
irb(main):001:0> roots = Katello::RootRepository.where(content_id: ['1626457224897', '1626457524855'])
=> #<ActiveRecord::Relation [#<Katello::RootRepository id: 83, name: "centos7", created_at: "2021-07-16 17:39:54.387955000 +0000", updated_at: "2022-10-19 14:14:19.001662000 +0000", m...
irb(main):002:0> roots.each{|root| ForemanTasks.async_task(::Actions::Candlepin::Product::ContentAdd, owner: root.product.organization.label, product_id: root.product.cp_id, content_id: ro
ot.content_id) }
=> [#<Katello::RootRepository id: 83, name: "centos7", created_at: "2021-07-16 17:39:54.387955000 +0000", updated_at: "2022-10-19 14:14:19.001662000 +0000", major: nil, minor: nil, gpg_key_id: nil, content_id: "1626457224897", arch: "noarch", label: "7", url: "http://mirror.centos.org/centos//7/os/x86_64/", unprotected: true, content_type: "yum", product_id: 21, checksum_type: nil, pulp_scratchpad_checksum_type: nil, docker_upstream_name: nil, download_policy: "immediate", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignorable_content: ["srpm"], include_tags: nil, description: nil, ansible_collection_requirements: nil, http_proxy_policy: "global_default_http_proxy", http_proxy_id: nil, download_concurrency: 5, os_versions: [], ansible_collection_auth_url: nil, ansible_collection_auth_token: nil, retain_package_versions_count: nil, generic_remote_options: nil, upstream_authentication_token: nil, mirroring_policy: "mirror_content_only", exclude_tags: nil, docker_tags_whitelist: nil>, #<Katello::RootRepository id: 84, name: "centos7_extras", created_at: "2021-07-16 17:45:04.198314000 +0000", updated_at: "2022-10-19 14:36:08.119831000 +0000", major: nil, minor: nil, gpg_key_id: nil, content_id: "1626457524855", arch: "noarch", label: "7_extras", url: "http://mirror.centos.org/centos/7/extras/x86_64/", unprotected: true, content_type: "yum", product_id: 21, checksum_type: nil, pulp_scratchpad_checksum_type: nil, docker_upstream_name: nil, download_policy: "immediate", verify_ssl_on_sync: true, upstream_username: nil, upstream_password: nil, deb_releases: nil, deb_components: nil, deb_architectures: nil, ssl_ca_cert_id: nil, ssl_client_cert_id: nil, ssl_client_key_id: nil, ignorable_content: ["srpm"], include_tags: nil, description: nil, ansible_collection_requirements: nil, http_proxy_policy: "global_default_http_proxy", http_proxy_id: nil, download_concurrency: 5, os_versions: [], ansible_collection_auth_url: nil, ansible_collection_auth_token: nil, retain_package_versions_count: 3, generic_remote_options: nil, upstream_authentication_token: nil, mirroring_policy: "mirror_content_only", exclude_tags: nil, docker_tags_whitelist: nil>]
irb(main):003:0> 
# sudo -u postgres psql candlepin
psql (12.11)
Type "help" for help.

candlepin=# SELECT content.* from cp2_content content LEFT JOIN cp2_product_content pc on content.uuid = pc.content_uuid where pc.content_uuid IS NULL;
               uuid               |  content_id   |          created           |          updated           |       contenturl        | gpgurl |         label         | metadataexpire |   
   name      | releasever | requiredtags | type | vendor | arches | entity_version | locked 
----------------------------------+---------------+----------------------------+----------------------------+-------------------------+--------+-----------------------+----------------+----------------+------------+--------------+------+--------+--------+----------------+--------
 8a8a822481f3bf2f0183a9b8d3f3380d | 1626457224897 | 2021-07-16 13:40:24.892-04 | 2022-10-05 15:56:45.939-04 | /custom/centos/7        |        | curia_centos_7        |              1 | centos7        |            |              | yum  | Custom |        |                |      0
 8a8a822481f3bf2f0183a9b90618381c | 1626457524855 | 2021-07-16 13:45:24.849-04 | 2022-10-05 15:56:58.776-04 | /custom/centos/7_extras |        | curia_centos_7_extras |              1 | centos7_extras |            |              | yum  | Custom |        |                |      0
(2 rows)

The link you provided suggests running a script to fix the entitlement certs when foreman-rake console fails. Do you happen to know where the certs are located by default?

5

The entitlement certificates are on the “clients” at /etc/pki/entitlement/.

You can look into it with rct cat-cert /etc/pki/entitlement/7394224347322562830.pem.

6

Thanks! When reading the instructions for the script, I was thinking they were on the server, too. Either way, the script found nothing, and I have two missing repos. Is there way to manually edit the database to fix?

7

Unfortunately I do not know. I initially found the bug and could not manually fix the database, but needed to work around because of time constraints.

8

Thanks for your help! I ended up recreating the product from scratch and adjusting the subscriptions. You helped me identify a few other smaller affected repositories that I had missed.

1 Like