Serious candlepin issues with CentOS Stream 9

gvde · January 4, 2023, 1:00pm

Problem:
There seem to be some serious issues with candlepin if I try to add CentOS Stream 9 repositories. I have had them set up before but I had issues that I could access most repositories through my content view. I was able to “fix” some using rake from here https://community.theforeman.org/t/subscription-manager-is-missing-repository/28886/12 but BaseOS wasn’t available, whatever I did. The task always failed with:

Runtime Error Entity version collision detected: Product [uuid: 082c99aa84b016240184bdcec2b00083, id: 3280049866, name: CentOS Stream 9] != Product [uuid: null, id: 3280049866, name: CentOS Stream 9] at org.candlepin.controller.ProductManager.updateProduct:368

So I have removed the cs9 repos and product and created them again.

I have created a CentOS Stream 9 product and 8 repositories

CentOS Stream 9 - AppStream x86_64
CentOS Stream 9 - BaseOS x86_64
CentOS Stream 9 - CRB x86_64
CentOS Stream 9 - Extras packages x86_64
CentOS Stream 9 - HighAvailability x86_64
CentOS Stream 9 - NFV x86_64
CentOS Stream 9 - ResilientStorage x86_64
CentOS Stream 9 - RT x86_64

However. this creates 17 products in the candlepin database:

candlepin=# select * from cp2_products where name = 'CentOS Stream 9';
               uuid               |          created           |          updated           | multiplier |  product_id  |      name       |    entity_version    | locked | derived_product_uuid 
----------------------------------+----------------------------+----------------------------+------------+--------------+-----------------+----------------------+--------+----------------------
 082c99aa8543683c01857c97fc640536 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:43:33.476+01 |          1 | 966110287218 | CentOS Stream 9 |  -577612996099227665 |      0 | 
 082c99aa8543683c01857c9be4630553 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.475+01 |          1 | 966110287218 | CentOS Stream 9 |  -872587081046238326 |      0 | 
 082c99aa8543683c01857c9be49c0556 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.532+01 |          1 | 966110287218 | CentOS Stream 9 |  -872587622523194665 |      0 | 
 082c99aa8543683c01857c9beb47055b | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.238+01 |          1 | 966110287218 | CentOS Stream 9 | -2894089637275930772 |      0 | 
 082c99aa8543683c01857c9beb83055f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.299+01 |          1 | 966110287218 | CentOS Stream 9 | -2894090569304311547 |      0 | 
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 |          1 | 966110287218 | CentOS Stream 9 | -8650985404925930517 |      0 | 
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 |          1 | 966110287218 | CentOS Stream 9 | -5754411302718292557 |      0 | 
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 |          1 | 966110287218 | CentOS Stream 9 |  5841965837295864547 |      0 | 
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 |          1 | 966110287218 | CentOS Stream 9 | -3150288127042849024 |      0 | 
 082c99aa8543683c01857c9bf1ce0565 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:52.91+01  |          1 | 966110287218 | CentOS Stream 9 | -5905044481564618210 |      0 | 
 082c99aa8543683c01857c9bf237056a | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:53.015+01 |          1 | 966110287218 | CentOS Stream 9 | -5911513726497692087 |      0 | 
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 |          1 | 966110287218 | CentOS Stream 9 | -6626014154148327502 |      0 | 
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 |          1 | 966110287218 | CentOS Stream 9 | -6626013188530987861 |      0 | 
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 |          1 | 966110287218 | CentOS Stream 9 |  7477646669900187352 |      0 | 
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 |          1 | 966110287218 | CentOS Stream 9 | -2687046942021682924 |      0 | 
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 |          1 | 966110287218 | CentOS Stream 9 | -8041775433908294371 |      0 | 
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 |          1 | 966110287218 | CentOS Stream 9 | -8041687594005900268 |      0 | 
(17 rows)

It also created 16 cp2_content entries which are abandoned, i.e. are not linked in cp2_product_content:

candlepin=# select * from cp2_content where uuid not in ( select content_uuid from cp2_product_content );
               uuid               |  content_id   |          created           |          updated           |                 contenturl                  |                         gpgurl                          |                  label                   | metadataexpire |                   name                    | releasever | requiredtags | type | vendor | arches |    entity_version    | locked
----------------------------------+---------------+----------------------------+----------------------------+---------------------------------------------+---------------------------------------------------------+------------------------------------------+----------------+-------------------------------------------+------------+--------------+------+--------+--------+----------------------+--------
 082c99aa8543683c01857c928ad3049b | 1672832256720 | 2023-01-04 12:37:36.723+01 | 2023-01-04 12:37:36.723+01 | /custom/centos9/AppStream_x86_64_os         |                                                         | ORG_centos9_AppStream_x86_64_os         |              1 | CentOS Stream 9 - AppStream x86_64        |            |              | yum  | Custom |        |  5584822885529146356 |      0
 082c99aa8543683c01857c928c0e049e | 1672832256720 | 2023-01-04 12:37:36.723+01 | 2023-01-04 12:37:37.038+01 | /custom/centos9/AppStream_x86_64_os         | ../../katello/api/v2/repositories/24924/gpg_key_content | ORG_centos9_AppStream_x86_64_os         |              1 | CentOS Stream 9 - AppStream x86_64        |            |              | yum  | Custom |        |  5743145117771954534 |      0
 082c99aa8543683c01857c92aa8804a3 | 1672832264841 | 2023-01-04 12:37:44.84+01  | 2023-01-04 12:37:44.84+01  | /custom/centos9/BaseOS_x86_64_os            |                                                         | ORG_centos9_BaseOS_x86_64_os            |              1 | CentOS Stream 9 - BaseOS x86_64           |            |              | yum  | Custom |        |  3000021058833026453 |      0
 082c99aa8543683c01857c92aae704a7 | 1672832264841 | 2023-01-04 12:37:44.84+01  | 2023-01-04 12:37:44.935+01 | /custom/centos9/BaseOS_x86_64_os            | ../../katello/api/v2/repositories/24925/gpg_key_content | ORG_centos9_BaseOS_x86_64_os            |              1 | CentOS Stream 9 - BaseOS x86_64           |            |              | yum  | Custom |        |  2919531475872166168 |      0
 082c99aa8543683c01857c92c24b04ad | 1672832270925 | 2023-01-04 12:37:50.923+01 | 2023-01-04 12:37:50.923+01 | /custom/centos9/CRB_x86_64_os               |                                                         | ORG_centos9_CRB_x86_64_os               |              1 | CentOS Stream 9 - CRB x86_64              |            |              | yum  | Custom |        | -1778292872404726747 |      0
 082c99aa8543683c01857c92c2a304b2 | 1672832270925 | 2023-01-04 12:37:50.923+01 | 2023-01-04 12:37:51.011+01 | /custom/centos9/CRB_x86_64_os               | ../../katello/api/v2/repositories/24926/gpg_key_content | ORG_centos9_CRB_x86_64_os               |              1 | CentOS Stream 9 - CRB x86_64              |            |              | yum  | Custom |        | -1748917070154486343 |      0
 082c99aa8543683c01857c92e5b304c7 | 1672832279990 | 2023-01-04 12:37:59.987+01 | 2023-01-04 12:37:59.987+01 | /custom/centos9/HighAvailability_x86_64_os  |                                                         | ORG_centos9_HighAvailability_x86_64_os  |              1 | CentOS Stream 9 - HighAvailability x86_64 |            |              | yum  | Custom |        | -2098082564269288412 |      0
 082c99aa8543683c01857c92e61104ce | 1672832279990 | 2023-01-04 12:37:59.987+01 | 2023-01-04 12:38:00.081+01 | /custom/centos9/HighAvailability_x86_64_os  | ../../katello/api/v2/repositories/24928/gpg_key_content | ORG_centos9_HighAvailability_x86_64_os  |              1 | CentOS Stream 9 - HighAvailability x86_64 |            |              | yum  | Custom |        | -2197653192011615782 |      0
 082c99aa8543683c01857c93289d04fd | 1672832297123 | 2023-01-04 12:38:17.117+01 | 2023-01-04 12:38:17.117+01 | /custom/centos9/RT_x86_64_os                |                                                         | ORG_centos9_RT_x86_64_os                |              1 | CentOS Stream 9 - RT x86_64               |            |              | yum  | Custom |        | -8820020093328439905 |      0
 082c99aa8543683c01857c9328fe0507 | 1672832297123 | 2023-01-04 12:38:17.117+01 | 2023-01-04 12:38:17.214+01 | /custom/centos9/RT_x86_64_os                | ../../katello/api/v2/repositories/24931/gpg_key_content | ORG_centos9_RT_x86_64_os                |              1 | CentOS Stream 9 - RT x86_64               |            |              | yum  | Custom |        | -8723561878907734803 |      0
 082c99aa8543683c01857c92d85804b9 | 1672832276570 | 2023-01-04 12:37:56.568+01 | 2023-01-04 12:37:56.568+01 | /custom/centos9/extras_x86_64_extras-common |                                                         | ORG_centos9_extras_x86_64_extras-common |              1 | CentOS Stream 9 - Extras packages x86_64  |            |              | yum  | Custom |        |  2443496623228455932 |      0
 082c99aa8543683c01857c92d8e104bf | 1672832276570 | 2023-01-04 12:37:56.568+01 | 2023-01-04 12:37:56.705+01 | /custom/centos9/extras_x86_64_extras-common | ../../katello/api/v2/repositories/24927/gpg_key_content | ORG_centos9_extras_x86_64_extras-common |              1 | CentOS Stream 9 - Extras packages x86_64  |            |              | yum  | Custom |        |  2582737810689797025 |      0
 082c99aa8543683c01857c92f04604d7 | 1672832282699 | 2023-01-04 12:38:02.694+01 | 2023-01-04 12:38:02.694+01 | /custom/centos9/NFV_x86_64_os               |                                                         | ORG_centos9_NFV_x86_64_os               |              1 | CentOS Stream 9 - NFV x86_64              |            |              | yum  | Custom |        | -4735327340539188215 |      0
 082c99aa8543683c01857c92f0a504df | 1672832282699 | 2023-01-04 12:38:02.694+01 | 2023-01-04 12:38:02.789+01 | /custom/centos9/NFV_x86_64_os               | ../../katello/api/v2/repositories/24929/gpg_key_content | ORG_centos9_NFV_x86_64_os               |              1 | CentOS Stream 9 - NFV x86_64              |            |              | yum  | Custom |        | -4725032583070414896 |      0
 082c99aa8543683c01857c9316fc04e9 | 1672832292610 | 2023-01-04 12:38:12.604+01 | 2023-01-04 12:38:12.604+01 | /custom/centos9/ResilientStorage_x86_64_os  |                                                         | ORG_centos9_ResilientStorage_x86_64_os  |              1 | CentOS Stream 9 - ResilientStorage x86_64 |            |              | yum  | Custom |        |  6670013558988485211 |      0
 082c99aa8543683c01857c93176c04f2 | 1672832292610 | 2023-01-04 12:38:12.604+01 | 2023-01-04 12:38:12.716+01 | /custom/centos9/ResilientStorage_x86_64_os  | ../../katello/api/v2/repositories/24930/gpg_key_content | ORG_centos9_ResilientStorage_x86_64_os  |              1 | CentOS Stream 9 - ResilientStorage x86_64 |            |              | yum  | Custom |        |  6656606388198089624 |      0
(16 rows)

It seems to add two new products whenever I have added a new repository, containing the new repository and all previously added, which results in 17 product uuids (1 for the initial empty product, and 2 for each additional of those 8 repositories), resulting in a large number of cp2_product_content associations:

candlepin=# select a.uuid,a.created,a.updated,a.product_id,a.name,a.entity_version,b.enabled,b.created,b.updated,b.id,c.uuid,c.content_id,c.created,c.updated,c.contenturl,c.label,c.name from cp2_products a left join cp2_product_content b on a.uuid = b.product_uuid left join cp2_content c on b.content_uuid = c.uuid where a.name = 'CentOS Stream 9' order by a.uuid,c.uuid;
               uuid               |          created           |          updated           |  product_id  |      name       |    entity_version    | enabled |          created           |          updated           |                id                |               uuid               |  content_id   |          created           |          updated           |                 contenturl                  |                 replace                 |                     name                     
----------------------------------+----------------------------+----------------------------+--------------+-----------------+----------------------+---------+----------------------------+----------------------------+----------------------------------+----------------------------------+---------------+----------------------------+----------------------------+---------------------------------------------+-----------------------------------------+----------------------------------------------
 082c99aa8543683c01857c97fc640536 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:43:33.476+01 | 966110287218 | CentOS Stream 9 |  -577612996099227665 |         |                            |                            |                                  |                                  |               |                            |                            |                                             |                                         | 
 082c99aa8543683c01857c9be4630553 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.475+01 | 966110287218 | CentOS Stream 9 |  -872587081046238326 | t       | 2023-01-04 12:47:49.475+01 | 2023-01-04 12:47:49.475+01 | 082c99aa8543683c01857c9be4630554 | 082c99aa8543683c01857c9be4330552 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.427+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9be49c0556 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.532+01 | 966110287218 | CentOS Stream 9 |  -872587622523194665 | t       | 2023-01-04 12:47:49.532+01 | 2023-01-04 12:47:49.532+01 | 082c99aa8543683c01857c9be49c0557 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9beb47055b | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.238+01 | 966110287218 | CentOS Stream 9 | -2894089637275930772 | t       | 2023-01-04 12:47:51.239+01 | 2023-01-04 12:47:51.239+01 | 082c99aa8543683c01857c9beb47055c | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9beb47055b | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.238+01 | 966110287218 | CentOS Stream 9 | -2894089637275930772 | t       | 2023-01-04 12:47:51.239+01 | 2023-01-04 12:47:51.239+01 | 082c99aa8543683c01857c9beb47055d | 082c99aa8543683c01857c9beb1b055a | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.195+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9beb83055f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.299+01 | 966110287218 | CentOS Stream 9 | -2894090569304311547 | t       | 2023-01-04 12:47:51.299+01 | 2023-01-04 12:47:51.299+01 | 082c99aa8543683c01857c9beb830560 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9beb83055f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.299+01 | 966110287218 | CentOS Stream 9 | -2894090569304311547 | t       | 2023-01-04 12:47:51.299+01 | 2023-01-04 12:47:51.299+01 | 082c99aa8543683c01857c9beb830561 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9bf1ce0565 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:52.91+01  | 966110287218 | CentOS Stream 9 | -5905044481564618210 | t       | 2023-01-04 12:47:52.91+01  | 2023-01-04 12:47:52.91+01  | 082c99aa8543683c01857c9bf1ce0566 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9bf1ce0565 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:52.91+01  | 966110287218 | CentOS Stream 9 | -5905044481564618210 | t       | 2023-01-04 12:47:52.91+01  | 2023-01-04 12:47:52.91+01  | 082c99aa8543683c01857c9bf1ce0567 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9bf1ce0565 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:52.91+01  | 966110287218 | CentOS Stream 9 | -5905044481564618210 | t       | 2023-01-04 12:47:52.91+01  | 2023-01-04 12:47:52.91+01  | 082c99aa8543683c01857c9bf1ce0568 | 082c99aa8543683c01857c9bf19a0564 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:52.858+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9bf237056a | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:53.015+01 | 966110287218 | CentOS Stream 9 | -5911513726497692087 | t       | 2023-01-04 12:47:53.015+01 | 2023-01-04 12:47:53.015+01 | 082c99aa8543683c01857c9bf237056b | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9bf237056a | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:53.015+01 | 966110287218 | CentOS Stream 9 | -5911513726497692087 | t       | 2023-01-04 12:47:53.015+01 | 2023-01-04 12:47:53.015+01 | 082c99aa8543683c01857c9bf237056c | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9bf237056a | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:53.015+01 | 966110287218 | CentOS Stream 9 | -5911513726497692087 | t       | 2023-01-04 12:47:53.015+01 | 2023-01-04 12:47:53.015+01 | 082c99aa8543683c01857c9bf237056d | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 | 966110287218 | CentOS Stream 9 | -6626014154148327502 | t       | 2023-01-04 12:47:54.819+01 | 2023-01-04 12:47:54.819+01 | 082c99aa8543683c01857c9bf9430572 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 | 966110287218 | CentOS Stream 9 | -6626014154148327502 | t       | 2023-01-04 12:47:54.819+01 | 2023-01-04 12:47:54.819+01 | 082c99aa8543683c01857c9bf9430574 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 | 966110287218 | CentOS Stream 9 | -6626014154148327502 | t       | 2023-01-04 12:47:54.819+01 | 2023-01-04 12:47:54.819+01 | 082c99aa8543683c01857c9bf9430575 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 | 966110287218 | CentOS Stream 9 | -6626014154148327502 | t       | 2023-01-04 12:47:54.819+01 | 2023-01-04 12:47:54.819+01 | 082c99aa8543683c01857c9bf9430573 | 082c99aa8543683c01857c9bf9140570 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.772+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 | 966110287218 | CentOS Stream 9 | -6626013188530987861 | t       | 2023-01-04 12:47:54.896+01 | 2023-01-04 12:47:54.896+01 | 082c99aa8543683c01857c9bf9900578 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 | 966110287218 | CentOS Stream 9 | -6626013188530987861 | t       | 2023-01-04 12:47:54.896+01 | 2023-01-04 12:47:54.896+01 | 082c99aa8543683c01857c9bf990057a | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 | 966110287218 | CentOS Stream 9 | -6626013188530987861 | t       | 2023-01-04 12:47:54.896+01 | 2023-01-04 12:47:54.896+01 | 082c99aa8543683c01857c9bf990057b | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 | 966110287218 | CentOS Stream 9 | -6626013188530987861 | t       | 2023-01-04 12:47:54.896+01 | 2023-01-04 12:47:54.896+01 | 082c99aa8543683c01857c9bf9900579 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 | t       | 2023-01-04 12:47:56.734+01 | 2023-01-04 12:47:56.734+01 | 082c99aa8543683c01857c9c00be0580 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 | t       | 2023-01-04 12:47:56.734+01 | 2023-01-04 12:47:56.734+01 | 082c99aa8543683c01857c9c00be0583 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 | t       | 2023-01-04 12:47:56.734+01 | 2023-01-04 12:47:56.734+01 | 082c99aa8543683c01857c9c00be0584 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 | t       | 2023-01-04 12:47:56.734+01 | 2023-01-04 12:47:56.734+01 | 082c99aa8543683c01857c9c00be0581 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 | t       | 2023-01-04 12:47:56.734+01 | 2023-01-04 12:47:56.734+01 | 082c99aa8543683c01857c9c00be0582 | 082c99aa8543683c01857c9c003f057e | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.607+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 | t       | 2023-01-04 12:47:56.795+01 | 2023-01-04 12:47:56.795+01 | 082c99aa8543683c01857c9c00fb0587 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 | t       | 2023-01-04 12:47:56.795+01 | 2023-01-04 12:47:56.795+01 | 082c99aa8543683c01857c9c00fb058a | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 | t       | 2023-01-04 12:47:56.795+01 | 2023-01-04 12:47:56.795+01 | 082c99aa8543683c01857c9c00fb058b | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 | t       | 2023-01-04 12:47:56.795+01 | 2023-01-04 12:47:56.795+01 | 082c99aa8543683c01857c9c00fb0588 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 | t       | 2023-01-04 12:47:56.795+01 | 2023-01-04 12:47:56.795+01 | 082c99aa8543683c01857c9c00fb0589 | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40590 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40594 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40595 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40591 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40592 | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 | t       | 2023-01-04 12:47:58.564+01 | 2023-01-04 12:47:58.564+01 | 082c99aa8543683c01857c9c07e40593 | 082c99aa8543683c01857c9c07b7058e | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.519+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.624+01 | 2023-01-04 12:47:58.624+01 | 082c99aa8543683c01857c9c08200598 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.625+01 | 2023-01-04 12:47:58.625+01 | 082c99aa8543683c01857c9c0821059c | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.625+01 | 2023-01-04 12:47:58.625+01 | 082c99aa8543683c01857c9c0821059d | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.625+01 | 2023-01-04 12:47:58.625+01 | 082c99aa8543683c01857c9c08210599 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.625+01 | 2023-01-04 12:47:58.625+01 | 082c99aa8543683c01857c9c0821059a | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 | t       | 2023-01-04 12:47:58.625+01 | 2023-01-04 12:47:58.625+01 | 082c99aa8543683c01857c9c0821059b | 082c99aa8543683c01857c9c08190596 | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.617+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005ac | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005b1 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005b2 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005ad | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005af | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005b0 | 082c99aa8543683c01857c9c08190596 | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.617+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 | t       | 2023-01-04 12:51:43.056+01 | 2023-01-04 12:51:43.056+01 | 082c99aa8543683c01857c9f74d005ae | 082c99aa8543683c01857c9f74a105aa | 1672833103023 | 2023-01-04 12:51:43.009+01 | 2023-01-04 12:51:43.009+01 | /custom/centos9/CRB_x86_64_os               | ORG_centos9_CRB_x86_64_os               | CentOS Stream 9 - CRB x86_64
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105b5 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105ba | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105bb | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105b6 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105b8 | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105b9 | 082c99aa8543683c01857c9c08190596 | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.617+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 | t       | 2023-01-04 12:51:43.121+01 | 2023-01-04 12:51:43.121+01 | 082c99aa8543683c01857c9f751105b7 | 082c99aa8543683c01857c9f750a05b3 | 1672833103023 | 2023-01-04 12:51:43.009+01 | 2023-01-04 12:51:43.114+01 | /custom/centos9/CRB_x86_64_os               | ORG_centos9_CRB_x86_64_os               | CentOS Stream 9 - CRB x86_64
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c0 | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c6 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c7 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c1 | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c3 | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c4 | 082c99aa8543683c01857c9c08190596 | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.617+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c2 | 082c99aa8543683c01857c9f750a05b3 | 1672833103023 | 2023-01-04 12:51:43.009+01 | 2023-01-04 12:51:43.114+01 | /custom/centos9/CRB_x86_64_os               | ORG_centos9_CRB_x86_64_os               | CentOS Stream 9 - CRB x86_64
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 | t       | 2023-01-04 12:52:20.008+01 | 2023-01-04 12:52:20.008+01 | 082c99aa8543683c01857ca0052805c5 | 082c99aa8543683c01857ca004f905be | 1672833139976 | 2023-01-04 12:52:19.961+01 | 2023-01-04 12:52:19.961+01 | /custom/centos9/extras_x86_64_extras-common | ORG_centos9_extras_x86_64_extras-common | CentOS Stream 9 - Extras packages x86_64
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05ca | 082c99aa8543683c01857c9be4960555 | 1672832869436 | 2023-01-04 12:47:49.427+01 | 2023-01-04 12:47:49.526+01 | /custom/centos9/AppStream_x86_64_os         | ORG_centos9_AppStream_x86_64_os         | CentOS Stream 9 - AppStream x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05d0 | 082c99aa8543683c01857c9beb7d055e | 1672832871205 | 2023-01-04 12:47:51.195+01 | 2023-01-04 12:47:51.293+01 | /custom/centos9/BaseOS_x86_64_os            | ORG_centos9_BaseOS_x86_64_os            | CentOS Stream 9 - BaseOS x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05d1 | 082c99aa8543683c01857c9bf22e0569 | 1672832872868 | 2023-01-04 12:47:52.858+01 | 2023-01-04 12:47:53.006+01 | /custom/centos9/HighAvailability_x86_64_os  | ORG_centos9_HighAvailability_x86_64_os  | CentOS Stream 9 - HighAvailability x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05cb | 082c99aa8543683c01857c9bf9880576 | 1672832874783 | 2023-01-04 12:47:54.772+01 | 2023-01-04 12:47:54.888+01 | /custom/centos9/NFV_x86_64_os               | ORG_centos9_NFV_x86_64_os               | CentOS Stream 9 - NFV x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05cd | 082c99aa8543683c01857c9c00f50585 | 1672832876620 | 2023-01-04 12:47:56.607+01 | 2023-01-04 12:47:56.789+01 | /custom/centos9/RT_x86_64_os                | ORG_centos9_RT_x86_64_os                | CentOS Stream 9 - RT x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05ce | 082c99aa8543683c01857c9c08190596 | 1672832878533 | 2023-01-04 12:47:58.519+01 | 2023-01-04 12:47:58.617+01 | /custom/centos9/ResilientStorage_x86_64_os  | ORG_centos9_ResilientStorage_x86_64_os  | CentOS Stream 9 - ResilientStorage x86_64 os
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05cc | 082c99aa8543683c01857c9f750a05b3 | 1672833103023 | 2023-01-04 12:51:43.009+01 | 2023-01-04 12:51:43.114+01 | /custom/centos9/CRB_x86_64_os               | ORG_centos9_CRB_x86_64_os               | CentOS Stream 9 - CRB x86_64
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | t       | 2023-01-04 12:52:20.077+01 | 2023-01-04 12:52:20.077+01 | 082c99aa8543683c01857ca0056d05cf | 082c99aa8543683c01857ca0056505c8 | 1672833139976 | 2023-01-04 12:52:19.961+01 | 2023-01-04 12:52:20.069+01 | /custom/centos9/extras_x86_64_extras-common | ORG_centos9_extras_x86_64_extras-common | CentOS Stream 9 - Extras packages x86_64
(73 rows)

My other products and repositories are just fine and only appear once in the database (except for some RHEL repos). It’s only the CentOS Stream 9 for which I have noticed this. Is this normal? Does candlepin have a cleanup task running at some point?

It seems only the latest product is referenced from cp_pool:

candlepin=# select a.uuid,a.created,a.updated,a.product_id,a.name,a.entity_version,b.id,b.created,b.updated from cp2_products a left join cp_pool b on a.uuid = b.product_uuid where a.name = 'CentOS Stream 9';
               uuid               |          created           |          updated           |  product_id  |      name       |    entity_version    |                id                |          created           |          updated           
----------------------------------+----------------------------+----------------------------+--------------+-----------------+----------------------+----------------------------------+----------------------------+----------------------------
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 | 966110287218 | CentOS Stream 9 | -8041687594005900268 | 082c99aa8543683c01857c97fc970537 | 2023-01-04 12:43:33.527+01 | 2023-01-04 12:43:33.527+01
 082c99aa8543683c01857c9be4630553 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.475+01 | 966110287218 | CentOS Stream 9 |  -872587081046238326 |                                  |                            | 
 082c99aa8543683c01857c9c07e4058f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.564+01 | 966110287218 | CentOS Stream 9 |  5841965837295864547 |                                  |                            | 
 082c99aa8543683c01857ca0052805bf | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.008+01 | 966110287218 | CentOS Stream 9 | -8041775433908294371 |                                  |                            | 
 082c99aa8543683c01857c9f751105b4 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.121+01 | 966110287218 | CentOS Stream 9 | -2687046942021682924 |                                  |                            | 
 082c99aa8543683c01857c9bf9430571 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.819+01 | 966110287218 | CentOS Stream 9 | -6626014154148327502 |                                  |                            | 
 082c99aa8543683c01857c9beb83055f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.299+01 | 966110287218 | CentOS Stream 9 | -2894090569304311547 |                                  |                            | 
 082c99aa8543683c01857c9be49c0556 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:49.532+01 | 966110287218 | CentOS Stream 9 |  -872587622523194665 |                                  |                            | 
 082c99aa8543683c01857c9f74d005ab | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:51:43.056+01 | 966110287218 | CentOS Stream 9 |  7477646669900187352 |                                  |                            | 
 082c99aa8543683c01857c9bf9900577 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:54.896+01 | 966110287218 | CentOS Stream 9 | -6626013188530987861 |                                  |                            | 
 082c99aa8543683c01857c9c08200597 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:58.624+01 | 966110287218 | CentOS Stream 9 | -3150288127042849024 |                                  |                            | 
 082c99aa8543683c01857c9beb47055b | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:51.238+01 | 966110287218 | CentOS Stream 9 | -2894089637275930772 |                                  |                            | 
 082c99aa8543683c01857c9bf1ce0565 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:52.91+01  | 966110287218 | CentOS Stream 9 | -5905044481564618210 |                                  |                            | 
 082c99aa8543683c01857c9c00be057f | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.734+01 | 966110287218 | CentOS Stream 9 | -8650985404925930517 |                                  |                            | 
 082c99aa8543683c01857c97fc640536 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:43:33.476+01 | 966110287218 | CentOS Stream 9 |  -577612996099227665 |                                  |                            | 
 082c99aa8543683c01857c9c00fb0586 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:56.795+01 | 966110287218 | CentOS Stream 9 | -5754411302718292557 |                                  |                            | 
 082c99aa8543683c01857c9bf237056a | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:47:53.015+01 | 966110287218 | CentOS Stream 9 | -5911513726497692087 |                                  |                            | 
(17 rows)

Expected outcome:
One product, one content per repo.

Foreman and Proxy versions:
candlepin-selinux-4.2.3-1.el8.noarch
foreman-3.4.1-1.el8.noarch
katello-4.6.0-1.el8.noarch
candlepin-4.2.3-1.el8.noarch

Distribution and version:
AlmaLinux 8.7

iballou · January 12, 2023, 3:57pm

Hi @gvde ,

Besides the mess that is being caused in the Candlepin DB, what symptoms are you noticing? Are your CentOS Stream 9 repos besides BaseOS also unavailable to your clients?

We’ll see if this is reproducible and figure out some debugging steps if not.

gvde · January 12, 2023, 5:03pm

Well, it seems is all good now. The client sees all repositories again. Very weird. candlepin database also is looking good. I suspect, candlepin runs some regular maintenance to clear out remnants in the database. It seems it is really creating a new pool, product and content object for the whole product in the database each time you add/remove a repository of a product. Later it the old entries…

I don’t know, though, why I lost the repositories in the beginning…

iballou · January 12, 2023, 9:01pm

Hm that is odd, I’ll see if I can find another opinion about your candlepin DB pastes and the general situation. Glad it’s working now at least.

iballou · January 12, 2023, 9:14pm

Are you able to share the Dynflow console tasks for one of your repository creates that in turn created 2 products? I’m not yet reproducing that here, just seeing 1 product per new repository in the CP DB.

Edit: also would be curious if the Candlepin logs have any duplicate entries. Would help narrow down if it’s a Katello issue or a Candlepin issue.

gvde · January 13, 2023, 7:22am

I just added a test repository to my centos stream 9 product. Result in the database. Before:

candlepin=# select * from cp2_products where name = 'CentOS Stream 9';
               uuid               |          created           |          updated           | multiplier |  product_id  |      name       |    entity_version    | locked | derived_product_uuid 
----------------------------------+----------------------------+----------------------------+------------+--------------+-----------------+----------------------+--------+----------------------
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 |          1 | 966110287218 | CentOS Stream 9 | -8041687594005900268 |      0 | 
(1 row)

after:

candlepin=# select * from cp2_products where name = 'CentOS Stream 9';
               uuid               |          created           |          updated           | multiplier |  product_id  |      name       |    entity_version    | locked | derived_product_uuid 
----------------------------------+----------------------------+----------------------------+------------+--------------+-----------------+----------------------+--------+----------------------
 082c99aa8596b4630185a9f7ed2d0073 | 2023-01-04 12:43:33.476+01 | 2023-01-13 08:11:15.757+01 |          1 | 966110287218 | CentOS Stream 9 |  -678375074838116254 |      0 | 
 082c99aa8596b4630185a9f7eded007e | 2023-01-04 12:43:33.476+01 | 2023-01-13 08:11:15.949+01 |          1 | 966110287218 | CentOS Stream 9 |  -637745127700587034 |      0 | 
 082c99aa8543683c01857ca0056d05c9 | 2023-01-04 12:43:33.476+01 | 2023-01-04 12:52:20.077+01 |          1 | 966110287218 | CentOS Stream 9 | -8041687594005900268 |      0 | 
(3 rows)

I think it first adds the repository without GPG key and then adds the GPG key and each time it creates a new product uuid and abandons the old one which are cleared out later… So it doesn’t really seem to be an issue at all. It must be how candlepin handles it.

So for me, it’s working at the moment. I have “solved” my initial problem by removing the complete product and all repositories and deleting all traces of cs9 from the candlepin database (because I wasn’t able to get the baseos issue fixed) and creating everything again. I was monitoring the candlepin database because I wanted to be sure that it’s working fine and “misinterpreted” the accumulating cp2_products, etc. entries. But eventually, when all repos were added, content synced, content views published and promoted my centos stream 9 test host saw all repositories and happily installed updates again…

So, I think we can close this topic as “non-issue”. I’ll observe when I run into the next candlepin issue again…

gvde · January 13, 2023, 9:21am

Looking through candlepin.log I have found this:

2023-01-13 00:00:00,016 [thread=QuartzScheduler_Worker-15] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Job queued: AsyncJobStatus [id: 082c99aa8596b4630185a836298e0068, name: ExpiredPoolsCleanupJob, key: ExpiredPoolsCleanupJob, state: QUEUED]
2023-01-13 00:00:00,025 [thread=Thread-1036 (ActiveMQ-client-global-threads)] [job=082c99aa8596b4630185a836298e0068, job_key=ExpiredPoolsCleanupJob, org=, csid=] INFO  org.candlepin.async.JobManager - Starting job "ExpiredPoolsCleanupJob" using class: org.candlepin.async.tasks.ExpiredPoolsCleanupJob
2023-01-13 00:00:00,043 [thread=Thread-1036 (ActiveMQ-client-global-threads)] [job=082c99aa8596b4630185a836298e0068, job_key=ExpiredPoolsCleanupJob, org=, csid=] INFO  org.candlepin.async.JobManager - Job "ExpiredPoolsCleanupJob" completed in 18ms
2023-01-13 03:00:00,014 [thread=QuartzScheduler_Worker-4] [=, org=, csid=] INFO  org.candlepin.async.JobManager - Job queued: AsyncJobStatus [id: 082c99aa8596b4630185a8daf50c006c, name: UnmappedGuestEntitlementCleanerJob, key: UnmappedGuestEntitlementCleanerJob, state: QUEUED]
2023-01-13 03:00:00,025 [thread=Thread-1058 (ActiveMQ-client-global-threads)] [job=082c99aa8596b4630185a8daf50c006c, job_key=UnmappedGuestEntitlementCleanerJob, org=, csid=] INFO  org.candlepin.async.JobManager - Starting job "UnmappedGuestEntitlementCleanerJob" using class: org.candlepin.async.tasks.UnmappedGuestEntitlementC>
2023-01-13 03:00:00,046 [thread=Thread-1058 (ActiveMQ-client-global-threads)] [job=082c99aa8596b4630185a8daf50c006c, job_key=UnmappedGuestEntitlementCleanerJob, org=, csid=] INFO  org.candlepin.async.JobManager - Job "UnmappedGuestEntitlementCleanerJob" completed in 22ms

So as I suspected: candlepin does some regular maintenance. I suppose if I check after midnight those additional database rows will disappear… So this is O.K. Of course, it makes candlepin database debugging a whole lot more difficult, if you add or remove repositories as it fills the database with a lot of abandoned rows which will disappear eventually…