"Error message: the server returns an error" during content view publish

Support
,
1

Problem:
Since this morning, I cannot publish two of my four content views anymore. The two affected have filters. I publish a new version for all my content views every weekday morning. It was working just fine on Friday and before. I haven’t changed anything or updated for more than a week now. It coincides, however, with the remove orphan error to the content proxy which also happened the first time this morning. Delete Orphaned Content fails But I am not sure if it’s related as the remove orphan was due to the content proxy while the publish is on the main server.

Error:

Actions::Pulp3::Repository::CopyContent

Input:

{"filter_ids"=>[25, 27, 29],
 "solve_dependencies"=>false,
 "rpm_filenames"=>nil,
 "remove_all"=>true,
 "source_repository_id"=>177,
 "target_repository_id"=>102922,
 "smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>"bf4c673e-8676-4ef4-8522-09890715048e",
 "current_timezone"=>"Europe/Berlin",
 "current_organization_id"=>1,
 "current_location_id"=>2,
 "current_user_id"=>5}

Output:

{}

Exception:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 404
Response headers: {"date"=>"Mon, 29 Aug 2022 12:17:51 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"23", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"bf4c673e-8676-4ef4-8522-09890715048e", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.example.com", "connection"=>"close"}
Response body: {"detail":"Not found."}

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/content_packagegroups_api.rb:163:in `read_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/content_packagegroups_api.rb:110:in `read'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/pulp_content_unit.rb:108:in `pulp_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/pulp_content_unit.rb:122:in `fetch_backend_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/pulp_content_unit.rb:117:in `backend_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/package_group.rb:43:in `package_names_of_type'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/package_group.rb:23:in `default_package_names'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/models/katello/package_group.rb:30:in `package_names'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/katello/util/pulpcore_content_filters.rb:12:in `block in filter_package_groups_by_pulp_href'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/katello/util/pulpcore_content_filters.rb:10:in `reject'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/katello/util/pulpcore_content_filters.rb:10:in `filter_package_groups_by_pulp_href'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:390:in `additional_content_hrefs'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:365:in `copy_content_for_source'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/pulp3/repository/copy_content.rb:18:in `invoke_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:19:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

I have see access to two package groups failing in foreman-ssl_access_ssl.log:

 - - [29/Aug/2022:13:37:32 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"
 - - [29/Aug/2022:14:17:51 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"

Expected outcome:
Successful publish of new content view version.

Foreman and Proxy versions:
foreman-3.2.1-1.el7.noarch
katello-4.4.1-1.el7.noarch
tfm-pulpcore-python3-pulpcore-3.16.12-1.el7.noarch
tfm-pulpcore-python3-pulp-rpm-3.17.7-2.el7.noarch

Distribution and version:
CentOS 7.9

2

I currently see a total of three uuid of package groups missing in the pulpcore database which exist in the foreman database:

foreman=# select * from katello_package_groups where pulp_id like '%9a2f7d22%' or pulp_id like '%a18de06a%' or pulp_id like '%cdc38f34%';
  id  | name |                                   pulp_id                                    |                                                                                   description                                                                                
    |         created_at         |         updated_at         | migrated_pulp3_href | missing_from_migration | ignore_missing_from_migration 
------+------+------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----+----------------------------+----------------------------+---------------------+------------------------+-------------------------------
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | A lightweight desktop environment that works well on low end machines.                                                                                                       
    | 2021-07-07 12:28:00.548595 | 2021-07-07 12:28:00.557375 |                     | f                      | f
  754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | A lightweight desktop environment that works well on low end machines.                                                                                                       
    | 2021-07-07 12:30:29.180661 | 2021-07-07 12:30:29.1843   |                     | f                      | f
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | The KDE Plasma Workspaces, a highly-configurable graphical user interface which includes a panel, desktop, system icons and desktop widgets, and many powerful KDE applicatio
ns. | 2021-11-20 03:40:58.14952  | 2021-11-20 03:40:58.155714 |                     | f                      | f
(3 rows)

In the dynflow console I have identified the error linked to the EPEL 7 (in the one CV) or EPEL 8 (in the second CV) repositories to which the filters apply. So I guess it has something to do with those filters.

I have also checked all ids references from the katello_package_groups table

foreman=# select * from katello_package_groups where name = 'KDE';
  id  | name |                                   pulp_id                                    |                                                                                   description                                                                                
    |         created_at         |         updated_at         |                             migrated_pulp3_href                              | missing_from_migration | ignore_missing_from_migration 
------+------+------------------------------------------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
----+----------------------------+----------------------------+------------------------------------------------------------------------------+------------------------+-------------------------------
 3747 | KDE  | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | The KDE Plasma Workspaces, a highly-configurable graphical user interface which includes a panel, desktop, system icons and desktop widgets, and many powerful KDE applicatio
ns. | 2022-07-30 12:45:26.614927 | 2022-07-30 12:45:26.614931 |                                                                              | f                      | f
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | The KDE Plasma Workspaces, a highly-configurable graphical user interface which includes a panel, desktop, system icons and desktop widgets, and many powerful KDE applicatio
ns. | 2021-11-20 03:40:58.14952  | 2021-11-20 03:40:58.155714 |                                                                              | f                      | f
  304 | KDE  | /pulp/api/v3/content/rpm/packagegroups/2c61de93-8364-447c-a2e5-b0af795a79b7/ | The KDE Plasma Workspaces, a highly-configurable graphical user interface which includes a panel, desktop, system icons and desktop widgets, and many powerful KDE applicatio
ns. | 2020-06-11 06:27:33.829246 | 2020-06-11 06:27:33.830815 | /pulp/api/v3/content/rpm/packagegroups/2c61de93-8364-447c-a2e5-b0af795a79b7/ | f                      | f
  931 | KDE  | /pulp/api/v3/content/rpm/packagegroups/43f79897-76df-411c-82ab-739356421d9a/ | The KDE Plasma Workspaces, a highly-configurable graphical user interface which includes a panel, desktop, system icons and desktop widgets, and many powerful KDE applicatio
ns. | 2021-07-07 12:36:36.453921 | 2021-07-07 12:36:36.456415 |                                                                              | f                      | f
(4 rows)

foreman=# select * from katello_package_groups where name = 'Xfce';
  id  | name |                                   pulp_id                                    |                              description                               |         created_at         |         updated_at         | migrated_pulp3_href | missing_from_migrati
on | ignore_missing_from_migration 
------+------+------------------------------------------------------------------------------+------------------------------------------------------------------------+----------------------------+----------------------------+---------------------+---------------------
---+-------------------------------
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | A lightweight desktop environment that works well on low end machines. | 2021-07-07 12:28:00.548595 | 2021-07-07 12:28:00.557375 |                     | f                   
   | f
 3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | A lightweight desktop environment that works well on low end machines. | 2022-07-30 12:45:26.614919 | 2022-07-30 12:45:26.614923 |                     | f                   
   | f
  754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | A lightweight desktop environment that works well on low end machines. | 2021-07-07 12:30:29.180661 | 2021-07-07 12:30:29.1843   |                     | f                   
   | f
 3767 | Xfce | /pulp/api/v3/content/rpm/packagegroups/178d7527-4ef2-46ef-8ef4-8cefe4c9d17c/ | A lightweight desktop environment that works well on low end machines. | 2022-07-30 12:48:56.819516 | 2022-07-30 12:48:56.819522 |                     | f                   
   | f
(4 rows)

And none of those 8 URIs exist in pulpcore. So there is obviously an inconsistency here. But I don’t know what it has to do with those filters. The filters only allow a very small number of rpms from EPEL into the view. Those have nothing to with KDE or Xfce.

It’s also strange that there is no problem publishing the other to CVs which contain the respective full EPEL repository.

3

Sorry. That was a mixup in the databases on the content proxy and the main server. From those 8 URIs in the foreman database on the main server only those three mentioned before are missing. The other 5 exist in the pulpcore database.

pulpcore=# select content_ptr_id, id, name from rpm_packagegroup  where content_ptr_id in ( 'a18de06a-a198-42d0-a3a4-7838fe9d692b', '2be268c5-ff45-4ff8-96ac-17af587a9cd8', 'cdc38f34-98c2-4a62-b00f-183b78242059', 'a9f014e5-2d09-48bf-864a-b574489dbd8d', '178d7527-4ef2-46ef-8ef4-8cefe4c9d17c', '9a2f7d22-d2e1-45d8-98c3-60130882fc2b', '2c61de93-8364-447c-a2e5-b0af795a79b7', '43f79897-76df-411c-82ab-739356421d9a' );
            content_ptr_id            |      id      | name 
--------------------------------------+--------------+------
 a9f014e5-2d09-48bf-864a-b574489dbd8d | kde-desktop  | KDE
 2be268c5-ff45-4ff8-96ac-17af587a9cd8 | xfce-desktop | Xfce
 43f79897-76df-411c-82ab-739356421d9a | kde-desktop  | KDE
 2c61de93-8364-447c-a2e5-b0af795a79b7 | kde-desktop  | KDE
 178d7527-4ef2-46ef-8ef4-8cefe4c9d17c | xfce-desktop | Xfce
(5 rows)

So it’s back to those three package groups: Xfce twice and KDE once.

4

Copied my “el8” content view, published: error.

Removed all filters from the copy, published: OK.

Added one (first) include filter “etckeeper” for RPM etckeeper* on the EPEL8 repository, published: error.

So basically, once I add a filter, I’ll get an error. As I haven’t changed anything on my server for almost 2 weeks, I guess it must have something to do with the epel7 and epel8 metadata causing cv filters to break?

5

Another observation, although not fully verified yet: on my new server on AlmaLinux 8.6 with

foreman-3.3.0-1.el8.noarch
katello-4.5.0-1.el8.noarch
python39-pulp-rpm-3.17.7-3.el8.noarch
python39-pulpcore-3.18.5-2.el8.noarch

It doesn’t seem to happen. Old and new servers both sync EPEL around the same time, so if there is an issue with the metadata, it should be on both. On the new server I can publish the same content view with the same content and same filters without error. Of course, on the new server I didn’t have the remove orphan issue either…

6

How do I clean up this mess? Those two CVs become unusable. I cannot update them anymore. Is this a @katello issue or a pulpcore issue?

7

I dug into the databases and found that katello_package_groups has a total of 14 references in pulp_id which do not exist in the pulpcore database…

8

Hi @gvde,

My only guess is that, during some sync, these content units were remove from Pulp and Katello did not delete them. The history to check this is likely gone since orphan cleanup removes old repository versions.

Are the offending content units only package groups? It’ll be good to know which units are affected to search for a Katello indexing issue.

It makes sense that you can publish CVs without filters but filtering fails, since we only query for individual units during filtered CV publishing. When there are no filters, we only copy over the repository version.

Here’s what I would do:

  1. Delete the bad package groups from the Katello database that exist inside of Library (non CV) repositories.
  2. Run a complete sync on the repositories that had the bad package groups
  3. Publish your content view again

Let me know if you want help with making console commands for any of those steps above.

9

I created an issue to check Katello content indexing: Bug #35457: Investigation: does Katello properly remove content units when they are removed from a Pulp repository during syncing? - Katello - Foreman

10

Hi @iballou,

I see only three different URIs with 404 during syncs which are the three I have posted in my second post.

That should be those three one. The others I have found via database search don’t seem cause issues so far. I am not sure what you want to delete exactly. The broken package groups are also used in the non-filtered content view repositories in the database. Thus, deleting the package group from the database would remove it from all content views as well?

I tried that in hope it would fix the situation, but it didn’t. But of course, I didn’t delete anything from the database, yet.

Yes, that would be good. I would simply delete them in the database but all those references to all published CV versions scared me off a little bit.

11

Deleting the broken package groups from Katello will completely remove them from all repositories and content views. I thought perhaps they were duplicated, but they aren’t. Since the package groups are broken anyway, that just means they’ll be gone from the UI.

It would be worth double checking that the package groups indeed don’t exist anymore in the upstream repository. If they do, we’ll want to make sure that they are making it into Pulp and that Katello is indexing them properly after deleting the broken ones.

Here are some commands that should help:

# Stick all of your package group IDs in an array
package_group_ids = [<ids>]
repository_ids = []
::Katello::PackageGroup.find(package_group_ids).each do |package_group|
  repository_ids << package_group.repositories.pluck(:id)
  package_group.destroy
end
repository_ids = repository_ids.uniq.flatten.compact
# Indexing content is optional (might take a few minutes), but it may bring back the correct package groups if they did change "pulp_hrefs"
::Katello::Repository.find(repository_ids).each { |r| r.index_content }
1 Like
12

Not sure if it’s related at all but we did fix an issue with additive sync policy on repo not deleting packages in katello when it does so in pulp?

13

Well, as far as I can see:

foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where (a.pulp_id like '%9a2f7d22%' or a.pulp_id like '%a18de06a%' or a.pulp_id like '%cdc38f34%') and relative_path like '%/Library/%' order by relative_path;
  id  | name |                                   pulp_id                                    |                           pulp_id                            |                    relative_path
------+------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
  754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | bf598f28-21a5-4833-be65-4b63faf9aa30                         | ORG/Library/custom/epel8/next
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | bf598f28-21a5-4833-be65-4b63faf9aa30                         | ORG/Library/custom/epel8/next
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-epel8-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270     | ORG/Library/el8-epel8/custom/epel8/epel
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-epel8-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270     | ORG/Library/el8-epel8/custom/epel8/epel
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-epel8-Library-bf598f28-21a5-4833-be65-4b63faf9aa30     | ORG/Library/el8-epel8/custom/epel8/next
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-epel8-Library-bf598f28-21a5-4833-be65-4b63faf9aa30     | ORG/Library/el8-epel8/custom/epel8/next
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 1-el8-test-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270      | ORG/Library/el8-test/custom/epel8/epel
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 1-el8-test-Library-82f0da7e-4598-4ea6-bdd8-65fa7f0b4270      | ORG/Library/el8-test/custom/epel8/epel
(12 rows)

It is group Xfce for EPEL 7 and EPEL 8 and KDE for EPEL 8. Looking at the repository package group metadata for EPEL 7 and EPEL 8 a group with name “Xfce” exists in both EPEL 7 and EPEL 8, and “KDE” exists in EPEL 8. So those group still exist in upstream.

If I search the database for package groups of those names, I’ll find:

foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where (a.name in ( 'Xfce', 'KDE' ) ) and relative_path like '%/Library/custom/%' order by name, c.pulp_id;
  id  | name |                                   pulp_id                                    |               pulp_id                |              relative_path              
------+------+------------------------------------------------------------------------------+--------------------------------------+-----------------------------------------
  931 | KDE  | /pulp/api/v3/content/rpm/packagegroups/43f79897-76df-411c-82ab-739356421d9a/ | 70cf2996-6320-4fcd-a9c5-eda568957879 | DKRZ/Library/custom/centos7/base_x86_64
 3747 | KDE  | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
 1174 | KDE  | /pulp/api/v3/content/rpm/packagegroups/9a2f7d22-d2e1-45d8-98c3-60130882fc2b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
 3747 | KDE  | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
 3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270 | DKRZ/Library/custom/epel8/epel
 3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
  743 | Xfce | /pulp/api/v3/content/rpm/packagegroups/a18de06a-a198-42d0-a3a4-7838fe9d692b/ | bf598f28-21a5-4833-be65-4b63faf9aa30 | DKRZ/Library/custom/epel8/next
 3767 | Xfce | /pulp/api/v3/content/rpm/packagegroups/178d7527-4ef2-46ef-8ef4-8cefe4c9d17c/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | DKRZ/Library/custom/epel7/epel_x86_64
  754 | Xfce | /pulp/api/v3/content/rpm/packagegroups/cdc38f34-98c2-4a62-b00f-183b78242059/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b | DKRZ/Library/custom/epel7/epel_x86_64
(11 rows)

So it seems, for KDE 9a2f7d22-d2e1-45d8-98c3-60130882fc2b is the broken entry and a9f014e5-2d09-48bf-864a-b574489dbd8d is the correct entry. For Xfce there are also correct entries for each broken one.

Those package groups are duplicated in the katello database, the old ones with lower id numbers are the broken ones…

Also interesting: the newer correct entries are a month old:

foreman=# select id, name, pulp_id, created_at, updated_at from katello_package_groups where id in ( 3746, 3747, 3767 );
  id  | name |                                   pulp_id                                    |         created_at         |         updated_at         
------+------+------------------------------------------------------------------------------+----------------------------+----------------------------
 3746 | Xfce | /pulp/api/v3/content/rpm/packagegroups/2be268c5-ff45-4ff8-96ac-17af587a9cd8/ | 2022-07-30 12:45:26.614919 | 2022-07-30 12:45:26.614923
 3747 | KDE  | /pulp/api/v3/content/rpm/packagegroups/a9f014e5-2d09-48bf-864a-b574489dbd8d/ | 2022-07-30 12:45:26.614927 | 2022-07-30 12:45:26.614931
 3767 | Xfce | /pulp/api/v3/content/rpm/packagegroups/178d7527-4ef2-46ef-8ef4-8cefe4c9d17c/ | 2022-07-30 12:48:56.819516 | 2022-07-30 12:48:56.819522
(3 rows)

So I wonder, why the issue only appears now. I’ll try cleaning out later and get back…

1 Like
14

All EPEL repos are set to additive on my server. My problems are related to package groups, though.

15

O.K. Those commands on the three package group ids helped somewhat. Of course, the next publish blew up on a different package group.

From the previous total of 14 packages groups without entries in the pulpcore database 11 have been left:

pulpcore=# select pulp_id,content_ptr_id from public.dblink('dbname=foreman', 'select substring(pulp_id from 40 for 36 ) from  katello_package_groups') as katello_package_groups(pulp_id uuid) full outer  join rpm_packagegroup on katello_package_groups.pulp_id = rpm_packagegroup.content_ptr_id where rpm_packagegroup.content_ptr_id is null;
               pulp_id                | content_ptr_id 
--------------------------------------+----------------
 fa344de9-636e-440a-9aef-7f961330cd22 | 
 1ca37d61-4ba1-43cb-bf65-27705bdaf592 | 
 190562b0-35d5-47fc-993d-834995743951 | 
 47f9919a-59fd-4e2a-b65e-dbc3b8fb8606 | 
 b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8 | 
 229af386-884a-41b2-af76-775f2b465ba8 | 
 b7cfb5f4-d603-4336-8ef0-968a157cbea1 | 
 a55bc85d-d557-43ad-8f3c-0801d1684b88 | 
 d50e7889-64ab-40a7-afe1-1b202829c00e | 
 c3e4825f-fe44-4dcd-a8cd-583d295fd50b | 
 3c91eebf-9b5f-44b5-9d02-d58559821e45 | 
(11 rows)

from a variety of package groups, but all in epel 7 or 8

foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path from katello_package_groups a left join katello_repository_package_groups b on a.id = b.package_group_id left join katello_repositories c on b.repository_id = c.id where ( a.pulp_id in ( '/pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/', '/pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/', '/pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/', '/pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/', '/pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/', '/pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/', '/pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/', '/pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/', '/pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/', '/pulp/api/v3/content/rpm/packagegroups/c3e4825f-fe44-4dcd-a8cd-583d295fd50b/', '/pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/' ) )and relative_path like '%/Library/%' order by relative_path;
  id  |                 name                  |                                   pulp_id                                    |                           pulp_id                            |                    relative_path
------+---------------------------------------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
  782 | Development tools                     | /pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  784 | Cinnamon                              | /pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  766 | Milkymist                             | /pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  773 | Haskell                               | /pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  767 | MATE                                  | /pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  779 | Fedora Packager                       | /pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  769 | LXQt Desktop                          | /pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
  779 | Fedora Packager                       | /pulp/api/v3/content/rpm/packagegroups/b7cfb5f4-d603-4336-8ef0-968a157cbea1/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  784 | Cinnamon                              | /pulp/api/v3/content/rpm/packagegroups/d50e7889-64ab-40a7-afe1-1b202829c00e/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  782 | Development tools                     | /pulp/api/v3/content/rpm/packagegroups/a55bc85d-d557-43ad-8f3c-0801d1684b88/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  773 | Haskell                               | /pulp/api/v3/content/rpm/packagegroups/229af386-884a-41b2-af76-775f2b465ba8/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  766 | Milkymist                             | /pulp/api/v3/content/rpm/packagegroups/190562b0-35d5-47fc-993d-834995743951/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  767 | MATE                                  | /pulp/api/v3/content/rpm/packagegroups/47f9919a-59fd-4e2a-b65e-dbc3b8fb8606/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  769 | LXQt Desktop                          | /pulp/api/v3/content/rpm/packagegroups/b3cf215b-1cda-4f3f-b0e0-20dfa640d2f8/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
  744 | KDE Frameworks 5 Software Development | /pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
  752 | Fedora Packager                       | /pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
 1163 | base-x                                | /pulp/api/v3/content/rpm/packagegroups/c3e4825f-fe44-4dcd-a8cd-583d295fd50b/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
 1169 | KDE Software Development              | /pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/ | 82f0da7e-4598-4ea6-bdd8-65fa7f0b4270                         | ORG/Library/custom/epel8/epel
  752 | Fedora Packager                       | /pulp/api/v3/content/rpm/packagegroups/1ca37d61-4ba1-43cb-bf65-27705bdaf592/ | bf598f28-21a5-4833-be65-4b63faf9aa30                         | ORG/Library/custom/epel8/next
  744 | KDE Frameworks 5 Software Development | /pulp/api/v3/content/rpm/packagegroups/fa344de9-636e-440a-9aef-7f961330cd22/ | bf598f28-21a5-4833-be65-4b63faf9aa30                         | ORG/Library/custom/epel8/next
 1169 | KDE Software Development              | /pulp/api/v3/content/rpm/packagegroups/3c91eebf-9b5f-44b5-9d02-d58559821e45/ | bf598f28-21a5-4833-be65-4b63faf9aa30                         | ORG/Library/custom/epel8/next

So I took those 11 additional package group ids and fed them into the commands. Now they are all gone. Next, I did a complete sync on all three affected repositories (epel7, epel8, epel8-next).

Now, publishing and promoting of my filtered content views are working again!

1 Like
16

Glad to hear it worked! We’ll try to see if we can reproduce the issue ourselves and hopefully find a root cause.

17

I’ll hope you figure this out. I find it rather weird. For whatever reason, it added those duplicate package groups on July 30th. I can’t see why. There were no updates whatsoever before that for more than a week.

I also don’t understand why the issue just pops up now, more than 4 weeks after the duplicates have been created. There have been some updates including pulpcore updates in that time, but nothing after August 17th.

So it puzzles me what triggered this. I guess most likely it came from the epel metadata upstream.

18

If the duplicate package groups were added on July 30th, are there any corresponding repository sync tasks or content view publish tasks?

19

The timestamps for those packages groups match the UTC time of the sync plan for the epel repositories. Tasks older than a month are removed from the history, thus I cannot check for the task itself anymore…

The publish/promote cron job runs in the morning. That is unrelated.

20

It happened again. This morning the publish run for my filtered view “CentOS 7” went into pause state with the same error again:

PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 404
Response headers: {"date"=>"Mon, 12 Sep 2022 02:05:55 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, HEAD, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"23", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"d9233407-5d94-4cc8-ad1b-96467b5baf52", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman.dkrz.de", "connection"=>"close"}
Response body: {"detail":"Not found."}

The requested URL is a package group again:

[12/Sep/2022:04:05:55 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"

Affected group is “Development tools” which was one of the broken ones before:

foreman=# select * from katello_package_groups where pulp_id like '%e69069bf%';
  id  |       name        |                                   pulp_id                                    |           description            |         created_at         |         updated_at         | migrated_pulp3_href | missing_from_migr
ation | ignore_missing_from_migration 
------+-------------------+------------------------------------------------------------------------------+----------------------------------+----------------------------+----------------------------+---------------------+------------------
------+-------------------------------
 3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | A basic development environment. | 2022-07-30 12:48:56.819566 | 2022-07-30 12:48:56.819571 |                     | f                
      | f
(1 row)

At this times it seems this is the only package group affected:

pulpcore=# select pulp_id,content_ptr_id from public.dblink('dbname=foreman', 'select substring(pulp_id from 40 for 36 ) from  katello_package_groups') as katello_package_groups(pulp_id uuid) full outer  join rpm_packagegroup on katello_package_groups.pulp_id = rpm_packagegroup.content_ptr_id where rpm_packagegroup.content_ptr_id is null;
               pulp_id                | content_ptr_id 
--------------------------------------+----------------
 e69069bf-6755-4977-9221-1b7a395e4890 | 
(1 row)

Affected repo is epel7 again, which is the only repository with filters in this CV:

foreman=# select a.id, a.name, a.pulp_id, c.pulp_id, c.relative_path
foreman-#   from katello_package_groups a
foreman-#        left join katello_repository_package_groups b on a.id = b.package_group_id
foreman-#        left join katello_repositories c on b.repository_id = c.id
foreman-#  where ( a.pulp_id in ( '/pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/' ) )
foreman-#        and relative_path like '%/Library/%' order by relative_path;
  id  |       name        |                                   pulp_id                                    |                           pulp_id                            |                    relative_path
------+-------------------+------------------------------------------------------------------------------+--------------------------------------------------------------+-----------------------------------------------------
 3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | 1-centos7-epel7-Library-ff8674d5-eed1-4476-b103-1f31d9d2cc6b | ORG/Library/centos7-epel7/custom/epel7/epel_x86_64
 3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | ff8674d5-eed1-4476-b103-1f31d9d2cc6b                         | ORG/Library/custom/epel7/epel_x86_64
(2 rows)

Checking the tasks running: there is no other task with any error or warning for a couple of days now. The publish is called from a shell script using hammer. It works on all my four content views. First it promotes the version in lifecycle environment Testing to Production. After that it publishes a new version, promoted to Testing. After it runs a purge. This is the script running on Monday through Friday mornings.

#! /bin/bash

echo -n "Publishing and promoting... "
date

for i in 23 11 22 12
do
  hammer content-view info --id $i --fields THIN

  hammer content-view version promote --content-view-id=$i --organization ORG --from-lifecycle-environment Testing --to-lifecycle-environment Production
done

sleep 300

#for i in 4 5 11 12 17 18 19 20
for i in 23 11 22 12
do
  hammer content-view info --id $i --fields THIN

  time hammer --output silent content-view publish --id $i --lifecycle-environments Testing --organization ORG

  hammer --output silent content-view purge --id $i
done

date

ID 11 is the “CentOS 7” content view in question. ID 23 CV is my “EL 8” content view. ID 11 and ID 23 only have a very limited overlap in repositories.

Checking other tasks overlapping with the publish task (just listing in case it might be related):

  1. The smart proxy sync task for the ID 23 CV published/promoted before ended exactly one second before the error. Ended at: 2022-09-12 02:04:54 UTC
  2. The destroy task for the ID 23 CV ended well before: Ended at: 2022-09-12 02:04:37 UTC
  3. The paused task for ID 11 CV started at Started at: 2022-09-12 02:04:43 UTC

So I don’t think it’s an issue with another task running at the same time.

Creation date of the new package group is August 30th.

foreman=# select * from katello_package_groups where name = 'Development tools';
  id  |       name        |                                   pulp_id                                    |           description            |         created_at         |         updated_at         | migrated_pulp3_href | missing_from_migr
ation | ignore_missing_from_migration 
------+-------------------+------------------------------------------------------------------------------+----------------------------------+----------------------------+----------------------------+---------------------+------------------
------+-------------------------------
 3772 | Development tools | /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ | A basic development environment. | 2022-07-30 12:48:56.819566 | 2022-07-30 12:48:56.819571 |                     | f                
      | f
 7237 | Development tools | /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ | A basic development environment. | 2022-08-30 18:48:28.200156 | 2022-08-30 18:48:28.200203 |                     | f                
      | f
(2 rows)

I’ll find it somewhat curious that the new package group is added again on the 30th of a month. And if that’s the true creation date it would also mean that it has been successfully published and promoted in that CV for a couple for times since then. So if I understand correctly, the package group must have been duplicated in pulpcore for two weeks and only this weekend it has been removed from there.

A grep on those two uuids in the http log reveals that both uuids worked for a while.

[root@foreman httpd]# fgrep -h -e 6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc -e e69069bf-6755-4977-9221-1b7a395e4890 foreman-ssl_access_ssl.log-20220* foreman-ssl_access_ssl.log
2001:638:70e:19::1 - - [15/Aug/2022:04:06:16 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [15/Aug/2022:04:06:16 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [16/Aug/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [16/Aug/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [17/Aug/2022:04:07:49 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [17/Aug/2022:04:07:49 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [18/Aug/2022:04:06:13 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [18/Aug/2022:04:06:14 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [19/Aug/2022:04:06:06 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [19/Aug/2022:04:06:06 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [22/Aug/2022:04:07:05 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [22/Aug/2022:04:07:05 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [23/Aug/2022:04:05:39 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [23/Aug/2022:04:05:39 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [24/Aug/2022:04:06:19 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [24/Aug/2022:04:06:19 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [25/Aug/2022:04:06:44 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [25/Aug/2022:04:06:44 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [26/Aug/2022:04:06:51 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [26/Aug/2022:04:06:51 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:26 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [31/Aug/2022:10:08:27 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [01/Sep/2022:04:06:08 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [02/Sep/2022:04:05:31 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [05/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [06/Sep/2022:04:05:46 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:53 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [07/Sep/2022:04:05:54 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [08/Sep/2022:04:06:29 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 200 2771 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [09/Sep/2022:04:06:24 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/6d3f1b3b-29b2-4a5b-81a8-63a26c0170cc/ HTTP/1.1" 200 2773 "-" "OpenAPI-Generator/3.17.4/ruby"
2001:638:70e:19::1 - - [12/Sep/2022:04:05:55 +0200] "GET /pulp/api/v3/content/rpm/packagegroups/e69069bf-6755-4977-9221-1b7a395e4890/ HTTP/1.1" 404 23 "-" "OpenAPI-Generator/3.17.4/ruby"

The 6d3f1b3b appeared the first time on 31/Aug/2022:10:08:27 +0200 which is exactly the publish run I did to fix the issue before in my post 13 above. So basically it seems after I have cleaned up before, I have published successfully but at the same time the duplicate package group popped up.

@iballou Can I get you any other information/logs?