Pulp3 yum repo off-by-one retain package count with additive mirroring policy

Problem:

When a yum repo is configured with “Additive Mirroring” policy and “Retain Package” is set to “2”, Katello contains 3 packages in foreman.katello_rpms and Pulpcore contains 2 packages in pulpcore.rpm_package after numerous syncs with updated packages (data for an example package included below).

Publishing a Content View with an affected repo (EPEL 7 in this case) results in an error (included below).

Expected outcome:

Pulpcore retains 2 “old” packages plus current package, as Katello does.

Foreman and Proxy versions:

  • foreman 3.2.1
  • foreman-proxy 3.2.1

Foreman and Proxy plugin versions:

  • foreman-tasks 6.0.1
  • foreman_ansible 7.1.0
  • foreman_default_hostgroup 6.0.0
  • foreman_discovery 20.0.1
  • foreman_openscap 5.2.1
  • foreman_puppet 3.0.7
  • foreman_remote_execution 6.0.0
  • foreman_templates 9.1.0
  • foreman_vault 1.1.0
  • katello 4.4.1

Distribution and version:

  • CentOS 7.9.2009

Other relevant data:

Here is data from the DB for one of the packages listed in the error.

baresip-jack-2.0.2 is present in foreman.katello_rpms but absent from pulpcore.rpm_package:

foreman.katello_rpms

foreman=# select id,pulp_id,name,epoch,version,release,arch,filename from katello_rpms where name = 'baresip-jack';
   id   |                                 pulp_id                                 |     name     | epoch | version | release |  arch  |              filename               
--------+-------------------------------------------------------------------------+--------------+-------+---------+---------+--------+-------------------------------------
 171029 | /pulp/api/v3/content/rpm/packages/03dcaad6-1d5f-471b-8b57-01fac971968a/ | baresip-jack | 0     | 2.0.2   | 1.el7   | x86_64 | baresip-jack-2.0.2-1.el7.x86_64.rpm
 309806 | /pulp/api/v3/content/rpm/packages/12604dfb-c7f7-4192-b61c-5b06670350ba/ | baresip-jack | 0     | 2.3.0   | 1.el7   | x86_64 | baresip-jack-2.3.0-1.el7.x86_64.rpm
 586725 | /pulp/api/v3/content/rpm/packages/6dc13c48-4d9c-43aa-a478-5cae83eec62f/ | baresip-jack | 0     | 2.4.0   | 1.el7   | x86_64 | baresip-jack-2.4.0-1.el7.x86_64.rpm
(3 rows)

pulpcore.rpm_package

pulpcore=# select content_ptr_id,name,epoch,version,release,arch,"pkgId" from rpm_package where name = 'baresip-jack';
            content_ptr_id            |     name     | epoch | version | release |  arch  |                              pkgId                               
--------------------------------------+--------------+-------+---------+---------+--------+------------------------------------------------------------------
 12604dfb-c7f7-4192-b61c-5b06670350ba | baresip-jack | 0     | 2.3.0   | 1.el7   | x86_64 | 8a39c9d0297a0f063cc9bc7cdcaf96a406e4f4509b28db3046090e48c076fcd6
 6dc13c48-4d9c-43aa-a478-5cae83eec62f | baresip-jack | 0     | 2.4.0   | 1.el7   | x86_64 | d1ecf68d6f6650837ee69760a604a3d512aa0b816af8dcae16370addc39c19f1
(2 rows)

Error:

2022-06-23T20:40:34 [E|bac|55d01f65] Error message: the server returns an error
 55d01f65 | HTTP status code: 400
 55d01f65 | Response headers: {"date"=>"Thu, 23 Jun 2022 20:40:34 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"2898", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"55d01f65-de12-470e-ab8e-19663625bdb0", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 ***REDACTED***", "connection"=>"close"}
 55d01f65 | Response body: ["Could not find the following content units: ['/pulp/api/v3/content/rpm/packages/4bd021f4-40bf-49b1-a8be-3dda56e77785/', '/pulp/api/v3/content/rpm/packages/2b0d2b51-0334-4354-8db0-a09acfbdd4fd/', '/pulp/api/v3/content/rpm/packages/0c88b5d4-0c2d-4043-a06d-d6d7ecaa9263/', '/pulp/api/v3/content/rpm/packages/08a95d60-0cb6-4d12-9471-2fa57dba72d2/', '/pulp/api/v3/content/rpm/packages/12836635-4dae-4e8f-8a66-624e06575e78/', '/pulp/api/v3/content/rpm/packages/49663244-d236-436c-9f52-d0e533e6dcb2/', '/pulp/api/v3/content/rpm/packages/279e74b9-82de-43eb-898c-f9cdede5641f/', '/pulp/api/v3/content/rpm/packages/34acb290-605b-47c5-9f48-b5c44363b78d/', '/pulp/api/v3/content/rpm/packages/44aefd58-7c06-4486-8033-5d467b4280da/', '/pulp/api/v3/content/rpm/packages/01b0c815-3027-454d-80ac-edaf26558d58/', '/pulp/api/v3/content/rpm/packages/113f1762-c02f-4f67-b595-957e3df1bdcc/', '/pulp/api/v3/content/rpm/packages/17866eb0-ad41-408d-a71f-17da5530edb5/', '/pulp/api/v3/content/rpm/packages/3ce6fcf9-cb13-4288-8e87-3e2c7024e0f0/', '/pulp/api/v3/content/rpm/packages/2063e21e-04cb-487c-bc8d-6ff73fd913e4/', '/pulp/api/v3/content/rpm/packages/0e3330ed-efa3-4c6b-84da-e085f2d9cfbf/', '/pulp/api/v3/content/rpm/packages/342fc752-ee7c-42ee-97f5-4bb68c118655/', '/pulp/api/v3/content/rpm/packages/0ac75b90-ab8a-43ac-a377-fd1dd982bb3c/', '/pulp/api/v3/content/rpm/packages/1c511ce3-bc15-4932-999a-67e20cf11aee/', '/pulp/api/v3/content/rpm/packages/29f674a0-d740-4a6e-bf00-bbd8114bc335/', '/pulp/api/v3/content/rpm/packages/01334cde-ddf6-4fb8-903f-e1963440a8f2/', '/pulp/api/v3/content/rpm/packages/3a94f5cd-9a15-4fb3-adde-082439fcd80c/', '/pulp/api/v3/content/rpm/packages/2387ae15-7918-48fa-94ea-adea8b6bf1ae/', '/pulp/api/v3/content/rpm/packages/42b0f53b-1d7a-463c-8748-7bd4f1ab7c88/', '/pulp/api/v3/content/rpm/packages/190c40fa-441e-44ba-ae8e-74757981f6cd/', '/pulp/api/v3/content/rpm/packages/33576d69-a8da-401e-a0c9-b0eb966007dd/', '/pulp/api/v3/content/rpm/packages/4e1ab620-dbb4-4626-93a4-9cd51197250c/', '/pulp/api/v3/content/rpm/packages/03dcaad6-1d5f-471b-8b57-01fac971968a/', '/pulp/api/v3/content/rpm/packages/343ed9d4-a3d9-43fb-abc4-549de1c9e294/', '/pulp/api/v3/content/rpm/packages/31f40788-873c-46f7-a13c-4f0c89a8cea4/', '/pulp/api/v3/content/rpm/packages/436017cf-70ed-44be-8644-6a9b96067210/', '/pulp/api/v3/content/rpm/packages/1d2faed9-3a1e-4412-b487-3b76ca7b7181/', '/pulp/api/v3/content/rpm/packages/516954b0-6344-4573-bd44-d1cab454dad6/', '/pulp/api/v3/content/rpm/packages/1b1c8b6c-5c1d-4d56-a94d-3d20281c21e6/', '/pulp/api/v3/content/rpm/packages/54d0ed52-cafa-4028-95f8-e600f5e3e409/', '/pulp/api/v3/content/rpm/packages/315f79c4-90f4-4c6c-aaa3-7d4dc62beb4d/', '/pulp/api/v3/content/rpm/packages/3833a691-53bf-4ff0-b7f2-988e69e36d74/', '/pulp/api/v3/content/rpm/packages/3d237538-e298-4112-94b2-c51b54005373/', '/pulp/api/v3/content/rpm/packages/43d96d39-c740-46be-b2d7-9858f529b493/']"] (PulpRpmClient::ApiError)
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/repositories_rpm_api.rb:300:in `modify_with_http_info'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.17.4/lib/pulp_rpm_client/api/repositories_rpm_api.rb:244:in `modify'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository.rb:533:in `add_content'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:209:in `block in copy_units'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:208:in `each'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:208:in `each_slice'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:208:in `copy_units'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/services/katello/pulp3/repository/yum.rb:367:in `copy_content_for_source'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/pulp3/repository/copy_content.rb:18:in `invoke_external_task'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/polling.rb:19:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/cancellable.rb:14:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.4.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-6.0.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 55d01f65 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 55d01f65 | [ sidekiq ]
 55d01f65 | [ concurrent-ruby ]

Full list of affected packages from this error:

foreman=# select id,pulp_id,filename from katello_rpms where pulp_id in ('/pulp/api/v3/content/rpm/packages/33576d69-a8da-401e-a0c9-b0eb966007dd/','pulp/api/v3/content/rpm/packages/0e3330ed-efa3-4c6b-84da-e085f2d9cfbf/','/pulp/api/v3/content/rpm/packages/34acb290-605b-47c5-9f48-b5c44363b78d/','/pulp/api/v3/content/rpm/packages/17866eb0-ad41-408d-a71f-17da5530edb5/','/pulp/api/v3/content/rpm/packages/3d237538-e298-4112-94b2-c51b54005373/','/pulp/api/v3/content/rpm/packages/12836635-4dae-4e8f-8a66-624e06575e78/','/pulp/api/v3/content/rpm/packages/342fc752-ee7c-42ee-97f5-4bb68c118655/','/pulp/api/v3/content/rpm/packages/516954b0-6344-4573-bd44-d1cab454dad6/','/pulp/api/v3/content/rpm/packages/3ce6fcf9-cb13-4288-8e87-3e2c7024e0f0/','/pulp/api/v3/content/rpm/packages/0c88b5d4-0c2d-4043-a06d-d6d7ecaa9263/','/pulp/api/v3/content/rpm/packages/2387ae15-7918-48fa-94ea-adea8b6bf1ae/','/pulp/api/v3/content/rpm/packages/42b0f53b-1d7a-463c-8748-7bd4f1ab7c88/','/pulp/api/v3/content/rpm/packages/01334cde-ddf6-4fb8-903f-e1963440a8f2/','/pulp/api/v3/content/rpm/packages/08a95d60-0cb6-4d12-9471-2fa57dba72d2/','/pulp/api/v3/content/rpm/packages/1c511ce3-bc15-4932-999a-67e20cf11aee/','/pulp/api/v3/content/rpm/packages/44aefd58-7c06-4486-8033-5d467b4280da/','/pulp/api/v3/content/rpm/packages/31f40788-873c-46f7-a13c-4f0c89a8cea4/','/pulp/api/v3/content/rpm/packages/29f674a0-d740-4a6e-bf00-bbd8114bc335/','/pulp/api/v3/content/rpm/packages/3a94f5cd-9a15-4fb3-adde-082439fcd80c/','/pulp/api/v3/content/rpm/packages/4e1ab620-dbb4-4626-93a4-9cd51197250c/','/pulp/api/v3/content/rpm/packages/2063e21e-04cb-487c-bc8d-6ff73fd913e4/','/pulp/api/v3/content/rpm/packages/436017cf-70ed-44be-8644-6a9b96067210/','/pulp/api/v3/content/rpm/packages/43d96d39-c740-46be-b2d7-9858f529b493/','/pulp/api/v3/content/rpm/packages/1d2faed9-3a1e-4412-b487-3b76ca7b7181/','/pulp/api/v3/content/rpm/packages/03dcaad6-1d5f-471b-8b57-01fac971968a/','/pulp/api/v3/content/rpm/packages/54d0ed52-cafa-4028-95f8-e600f5e3e409/','/pulp/api/v3/content/rpm/packages/4bd021f4-40bf-49b1-a8be-3dda56e77785/','/pulp/api/v3/content/rpm/packages/190c40fa-441e-44ba-ae8e-74757981f6cd/','/pulp/api/v3/content/rpm/packages/315f79c4-90f4-4c6c-aaa3-7d4dc62beb4d/','/pulp/api/v3/content/rpm/packages/113f1762-c02f-4f67-b595-957e3df1bdcc/','/pulp/api/v3/content/rpm/packages/49663244-d236-436c-9f52-d0e533e6dcb2/','/pulp/api/v3/content/rpm/packages/2b0d2b51-0334-4354-8db0-a09acfbdd4fd/','/pulp/api/v3/content/rpm/packages/343ed9d4-a3d9-43fb-abc4-549de1c9e294/','/pulp/api/v3/content/rpm/packages/01b0c815-3027-454d-80ac-edaf26558d58/','/pulp/api/v3/content/rpm/packages/0ac75b90-ab8a-43ac-a377-fd1dd982bb3c/','/pulp/api/v3/content/rpm/packages/3833a691-53bf-4ff0-b7f2-988e69e36d74/','/pulp/api/v3/content/rpm/packages/1b1c8b6c-5c1d-4d56-a94d-3d20281c21e6/','/pulp/api/v3/content/rpm/packages/279e74b9-82de-43eb-898c-f9cdede5641f/');
   id   |                                 pulp_id                                 |                     filename                     
--------+-------------------------------------------------------------------------+--------------------------------------------------
  17291 | /pulp/api/v3/content/rpm/packages/3a94f5cd-9a15-4fb3-adde-082439fcd80c/ | python2-apprise-0.9.6-1.el7.noarch.rpm
  31422 | /pulp/api/v3/content/rpm/packages/516954b0-6344-4573-bd44-d1cab454dad6/ | rust-std-static-1.59.0-2.el7.x86_64.rpm
  21222 | /pulp/api/v3/content/rpm/packages/42b0f53b-1d7a-463c-8748-7bd4f1ab7c88/ | netatalk-3.1.12-10.el7.x86_64.rpm
  21223 | /pulp/api/v3/content/rpm/packages/33576d69-a8da-401e-a0c9-b0eb966007dd/ | netatalk-devel-3.1.12-10.el7.x86_64.rpm
  24195 | /pulp/api/v3/content/rpm/packages/01b0c815-3027-454d-80ac-edaf26558d58/ | globus-gssapi-gsi-14.17-1.el7.x86_64.rpm
  24211 | /pulp/api/v3/content/rpm/packages/43d96d39-c740-46be-b2d7-9858f529b493/ | globus-gsi-proxy-core-9.6-1.el7.x86_64.rpm
  31201 | /pulp/api/v3/content/rpm/packages/2b0d2b51-0334-4354-8db0-a09acfbdd4fd/ | remmina-1.4.25-1.el7.x86_64.rpm
  31204 | /pulp/api/v3/content/rpm/packages/01334cde-ddf6-4fb8-903f-e1963440a8f2/ | remmina-plugins-vnc-1.4.25-1.el7.x86_64.rpm
  31205 | /pulp/api/v3/content/rpm/packages/190c40fa-441e-44ba-ae8e-74757981f6cd/ | remmina-plugins-spice-1.4.25-1.el7.x86_64.rpm
  31210 | /pulp/api/v3/content/rpm/packages/2063e21e-04cb-487c-bc8d-6ff73fd913e4/ | remmina-gnome-session-1.4.25-1.el7.x86_64.rpm
  31211 | /pulp/api/v3/content/rpm/packages/315f79c4-90f4-4c6c-aaa3-7d4dc62beb4d/ | remmina-devel-1.4.25-1.el7.x86_64.rpm
  31467 | /pulp/api/v3/content/rpm/packages/2387ae15-7918-48fa-94ea-adea8b6bf1ae/ | distribution-gpg-keys-copr-1.67-1.el7.noarch.rpm
  31319 | /pulp/api/v3/content/rpm/packages/12836635-4dae-4e8f-8a66-624e06575e78/ | dmlite-shell-1.15.2-0.el7.x86_64.rpm
  31231 | /pulp/api/v3/content/rpm/packages/342fc752-ee7c-42ee-97f5-4bb68c118655/ | python-dmlite-1.15.2-0.el7.x86_64.rpm
  31322 | /pulp/api/v3/content/rpm/packages/0c88b5d4-0c2d-4043-a06d-d6d7ecaa9263/ | dmlite-plugins-profiler-1.15.2-0.el7.x86_64.rpm
  31324 | /pulp/api/v3/content/rpm/packages/1b1c8b6c-5c1d-4d56-a94d-3d20281c21e6/ | dmlite-plugins-memcache-1.15.2-0.el7.x86_64.rpm
  31325 | /pulp/api/v3/content/rpm/packages/17866eb0-ad41-408d-a71f-17da5530edb5/ | dmlite-plugins-librarian-1.15.2-0.el7.x86_64.rpm
  31331 | /pulp/api/v3/content/rpm/packages/436017cf-70ed-44be-8644-6a9b96067210/ | dmlite-dpmdisk-1.15.2-0.el7.x86_64.rpm
  31470 | /pulp/api/v3/content/rpm/packages/31f40788-873c-46f7-a13c-4f0c89a8cea4/ | clippy-1.59.0-2.el7.x86_64.rpm
  31477 | /pulp/api/v3/content/rpm/packages/1d2faed9-3a1e-4412-b487-3b76ca7b7181/ | cargo-doc-1.59.0-2.el7.noarch.rpm
 107605 | /pulp/api/v3/content/rpm/packages/3ce6fcf9-cb13-4288-8e87-3e2c7024e0f0/ | dmlite-shell-1.15.2-1.el7.x86_64.rpm
 107609 | /pulp/api/v3/content/rpm/packages/44aefd58-7c06-4486-8033-5d467b4280da/ | dmlite-plugins-mysql-1.15.2-1.el7.x86_64.rpm
 107614 | /pulp/api/v3/content/rpm/packages/113f1762-c02f-4f67-b595-957e3df1bdcc/ | dmlite-libs-1.15.2-1.el7.x86_64.rpm
 107625 | /pulp/api/v3/content/rpm/packages/49663244-d236-436c-9f52-d0e533e6dcb2/ | dmlite-apache-httpd-1.15.2-1.el7.x86_64.rpm
 171003 | /pulp/api/v3/content/rpm/packages/0ac75b90-ab8a-43ac-a377-fd1dd982bb3c/ | libre-2.2.2-1.el7.x86_64.rpm
 171014 | /pulp/api/v3/content/rpm/packages/1c511ce3-bc15-4932-999a-67e20cf11aee/ | baresip-2.0.2-1.el7.x86_64.rpm
 171016 | /pulp/api/v3/content/rpm/packages/29f674a0-d740-4a6e-bf00-bbd8114bc335/ | baresip-vp9-2.0.2-1.el7.x86_64.rpm
 171025 | /pulp/api/v3/content/rpm/packages/08a95d60-0cb6-4d12-9471-2fa57dba72d2/ | baresip-plc-2.0.2-1.el7.x86_64.rpm
 171026 | /pulp/api/v3/content/rpm/packages/279e74b9-82de-43eb-898c-f9cdede5641f/ | baresip-opus-2.0.2-1.el7.x86_64.rpm
 171027 | /pulp/api/v3/content/rpm/packages/4bd021f4-40bf-49b1-a8be-3dda56e77785/ | baresip-mqtt-2.0.2-1.el7.x86_64.rpm
 171029 | /pulp/api/v3/content/rpm/packages/03dcaad6-1d5f-471b-8b57-01fac971968a/ | baresip-jack-2.0.2-1.el7.x86_64.rpm
 171035 | /pulp/api/v3/content/rpm/packages/343ed9d4-a3d9-43fb-abc4-549de1c9e294/ | baresip-g722-2.0.2-1.el7.x86_64.rpm
 171036 | /pulp/api/v3/content/rpm/packages/54d0ed52-cafa-4028-95f8-e600f5e3e409/ | baresip-ctrl_dbus-2.0.2-1.el7.x86_64.rpm
 171039 | /pulp/api/v3/content/rpm/packages/34acb290-605b-47c5-9f48-b5c44363b78d/ | baresip-alsa-2.0.2-1.el7.x86_64.rpm
 290778 | /pulp/api/v3/content/rpm/packages/3d237538-e298-4112-94b2-c51b54005373/ | myproxy-6.2.9-8.el7.x86_64.rpm
 290781 | /pulp/api/v3/content/rpm/packages/3833a691-53bf-4ff0-b7f2-988e69e36d74/ | myproxy-libs-6.2.9-8.el7.x86_64.rpm
 290783 | /pulp/api/v3/content/rpm/packages/4e1ab620-dbb4-4626-93a4-9cd51197250c/ | myproxy-devel-6.2.9-8.el7.x86_64.rpm
(37 rows)

Steps taken to achieve clean Content View Publish (destructive to retained packages):

  1. Skip error step via Dynflow console
  2. Resume task in Katello
  3. Allow Content View Publish to complete
  4. Change Mirroring Policy to “Content Only” on affected repo (destructive)
  5. Complete Sync affected repo (destructive)
  6. Delete Content View version where error occurred

I imagine removing the offending entries from foreman.katello_rpms would resolve the issue in a non-destructive manner, until more packages were synced. However, I do not enjoy direct database edits to resolve issues in Katello, and have had my fill of those with much older versions. In my case, loss of retained packages was deemed acceptable after reviewing them.

Thanks for reporting this, we’ll get it checked out.

I haven’t reproduced the CV publish error, but I see that the removed RPMs are still related to the repository after Pulp removes them.

I created this issue: Bug #35120: Retain packages on Repository removes RPMs from Pulp but not from Katello - Katello - Foreman