Unable to publish an incremental CV update

Problem:
After upgrading to Katello 4.3 / Foreman 3.1.1 today I tried to publish an incremental update to my CVs because of the polkit issue.
This was actually the first time I tried this feature so I’m not sure if it worked before the upgrade.

In my case I was trying to apply RLSA-2022:267 to the CV (this was with Rocky Linux here)
(was not able to test it with any other CVs as I didn’t implement the external Errata repo creation for CentOS, or am not using AlmaLinux or others)

As far as I understood it, the job is looking for version 15 of the CV, but as I’m deleting all the versions except the last 3 from time to time, this is not available anymore. (versions 20.0 to 22.0 right now)

I was able to unlock the job by skipping this faulting step (Actions::Pulp3::Repository::MultiCopyUnits) in the Dynflow Console, so at least it doesn’t block any other actions in the CV :slight_smile:

Maybe it’s an issue from my side, am I not supposed to delete old versions of the CVs?
Or can someone reproduce that?

If I missed to add some relevant information I will off course provide it later!

Expected outcome:
Creation of the incremental CV version successfully finishes.

Foreman and Proxy versions:
foreman-3.1.1-1.el8
katello-4.3.0-1.el8

Foreman and Proxy plugin versions:

ansible-collection-theforeman-foreman-3.0.0-1.el8.noarch
candlepin-4.1.8-1.el8.noarch
candlepin-selinux-4.1.8-1.el8.noarch
foreman-cli-3.1.1-1.el8.noarch
foreman-debug-3.1.1-1.el8.noarch
foreman-dynflow-sidekiq-3.1.1-1.el8.noarch
foreman-installer-3.1.1-1.el8.noarch
foreman-installer-katello-3.1.1-1.el8.noarch
foreman-postgresql-3.1.1-1.el8.noarch
foreman-proxy-3.1.1-1.el8.noarch
foreman-release-3.1.1-1.el8.noarch
foreman-selinux-3.1.1-1.el8.noarch
foreman-service-3.1.1-1.el8.noarch
foreman-vmware-3.1.1-1.el8.noarch
katello-ca-consumer-r8-foreman-prod.fritz.box-1.0-1.noarch
katello-certs-tools-2.8.2-1.el8.noarch
katello-client-bootstrap-1.7.7-1.el8.noarch
katello-common-4.3.0-1.el8.noarch
katello-debug-4.3.0-1.el8.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.5-2.el8.noarch
katello-host-tools-tracer-3.5.5-2.el8.noarch
katello-repos-4.3.0-1.el8.noarch
katello-selinux-4.0.2-1.el8.noarch
katello-server-ca-1.0-1.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.7-1.el8.x86_64
python38-pulp-ansible-0.10.1-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
python38-pulp-container-2.9.0-1.el8.noarch
python38-pulp-deb-2.16.1-1.el8.noarch
python38-pulp-file-1.10.1-1.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
python38-pulp-rpm-3.16.1-2.el8.noarch
python38-pulpcore-3.16.1-1.el8.noarch
qpid-proton-c-0.35.0-1.el8.x86_64
r8-foreman-prod.fritz.box-apache-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-client-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-proxy-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-proxy-client-1.0-1.noarch
r8-foreman-prod.fritz.box-puppet-client-1.0-1.noarch
rubygem-foreman-tasks-5.2.0-1.fm3_1.el8.noarch
rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el8.noarch
rubygem-foreman_ansible-7.0.2-1.fm3_1.el8.noarch
rubygem-foreman_ansible_core-4.2.1-1.fm2_5.el8.noarch
rubygem-foreman_bootdisk-19.0.3-1.fm3_1.el8.noarch
rubygem-foreman_maintain-1.0.2-1.el8.noarch
rubygem-foreman_puppet-2.0.0-1.fm3_1.el8.noarch
rubygem-foreman_remote_execution-5.0.1-1.fm3_1.el8.noarch
rubygem-foreman_remote_execution_core-1.4.8-1.el8.noarch
rubygem-foreman_snapshot_management-2.0.1-1.fm2_6.el8.noarch
rubygem-foreman_statistics-2.0.0-1.fm3_1.el8.noarch
rubygem-foreman_templates-9.1.0-1.fm2_6.el8.noarch
rubygem-hammer_cli-3.1.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.1.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.3.4-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.4-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_templates-0.2.0-2.fm2_5.el8.noarch
rubygem-hammer_cli_katello-1.3.1-1.el8.noarch
rubygem-katello-4.3.0-1.el8.noarch
rubygem-pulp_ansible_client-0.10.1-1.el8.noarch
rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
rubygem-pulp_container_client-2.9.0-1.el8.noarch
rubygem-pulp_deb_client-2.16.0-1.el8.noarch
rubygem-pulp_file_client-1.10.0-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
rubygem-pulp_python_client-3.5.2-1.el8.noarch
rubygem-pulp_rpm_client-3.16.1-1.el8.noarch
rubygem-pulpcore_client-3.16.0-1.el8.noarch
rubygem-qpid_proton-0.35.0-1.el8.x86_64
rubygem-smart_proxy_pulp-3.2.0-2.fm3_1.el8.noarch

Distribution and version:
Rocky Linux 8.5

Other relevant data:
So, the actual error is the following:

Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Fri, 28 Jan 2022 00:46:19 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"73", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"4144ea82-7e26-4744-8f39-96c7f6a02c1f", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 <server-fqdn>", "connection"=>"close"}
Response body: ["Version 15 does not exist for repository 'rockylinux8_extras-569931'."]

And the backtrace shows the following:

/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:79:in `copy_content_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:28:in `copy_content'
/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:153:in `copy_content_chunked'
/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:126:in `multi_copy_units'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/multi_copy_units.rb:49:in `invoke_external_task'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:19:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
1 Like

Oh one more thing, this RLSA is part of the BaseOS repo, interestingly it looks to be acting up because of the Extras repo not the BaseOS.

Same problem on Foreman 3.0.1-1 and katello-4.2.1-1 on CentOS7 distribution

Katello 4.2.1 and this hit me too. EVERY repo (redhat, rocky, centos) complained about “Version such and such is not there” (but … it was…).

Job blew up spectacularly. I know doing an errata like this worked flawlessly for me under Katello 3.x.

Hi @lumarel,

It looks like Katello’s database may be out of sync with Pulp. That “Version 15” in the error isn’t referring to CV version 15, but the 15th version of a backend Pulp repo.

Do you recall what version of Katello you were running when you published this content view version? Did this Katello server upgrade up past Katello 3.18 (and as such did the Pulp2->Pulp3 migration)?

Can I please see a few things:

  1. The full input and output of the failed “Actions::Pulp3::Repository::MultiCopyUnits” task
  2. The output of the katello:correct_repositories dry-run:
CONTENT_VIEW=your_content_view_name foreman-rake katello:correct_repositories
  1. The output of the following foreman-rake console command:
# First, find the ID of the content view version that you want to incrementally update.
# You can find this either with hammer or in the UI via the version's URL.
::Katello::ContentViewVersion.find(<ID from above>).repositories.archived.pluck(:id, :version_href)
2 Likes

Hey @iballou,

Thank you for bringing some light into the issue :slight_smile:

That’s actually quite easy to find out which version I created the CV on, because after running Katello 3.18 I decided to competely setup everything from 0, so the version I setup the CVs was 4.1.3 which was already completely Pulp3.

The full output of the “Actions::Pulp3::Repository::MultiCopyUnits” task:

Queue: default

Started at: 2022-01-28 20:39:23 UTC

Ended at: 2022-01-28 20:39:23 UTC

Real time: 0.11s

Execution time (excluding suspended state): 0.11s

Input:

---
repo_map:
  ? - 9
  : dest_repo: 1028
    base_version: 43
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 37
  : dest_repo: 1029
    base_version: 0
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 38
  : dest_repo: 1030
    base_version: 43
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 12
  : dest_repo: 1036
    base_version: 16
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 11
  : dest_repo: 1040
    base_version: 37
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 13
  : dest_repo: 1042
    base_version: 21
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 8
  : dest_repo: 1043
    base_version: 4
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 24
  : dest_repo: 1047
    base_version: 33
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 25
  : dest_repo: 1049
    base_version: 16
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 22
  : dest_repo: 1051
    base_version: 33
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
  ? - 23
  : dest_repo: 1052
    base_version: 19
    content_unit_hrefs:
    - "/pulp/api/v3/content/rpm/advisories/fe4b6f45-703d-4ba4-9b98-8bf3a94b6910/"
unit_map:
  errata:
  - 3534
  rpms: []
dependency_solving: true
incremental_update: 
smart_proxy_id: 1
remote_user: admin
remote_cp_user: admin
current_request_id: 3d361315-fff9-4cdb-8b17-762d57f372ca
current_timezone: Europe/Vienna
current_organization_id: 1
current_location_id: 2
current_user_id: 4

Output:

--- {}

Chunked output:

--- []

Error:

PulpRpmClient::ApiError

Error message: the server returns an error HTTP status code: 400 Response headers: {"date"=>"Fri, 28 Jan 2022 20:39:23 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"71", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"3d361315-fff9-4cdb-8b17-762d57f372ca", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 r8-foreman-prod.fritz.box", "connection"=>"close"} Response body: ["Version 16 does not exist for repository 'el8_epel_modular-511559'."]

---
- "/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api_client.rb:83:in
  `call_api'"
- "/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:79:in
  `copy_content_with_http_info'"
- "/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:28:in
  `copy_content'"
- "/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:153:in
  `copy_content_chunked'"
- "/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:126:in
  `multi_copy_units'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/multi_copy_units.rb:49:in
  `invoke_external_task'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:19:in `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in
  `block in run'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in
  `block in as_remote_user'"
- "/usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in
  `cp_config'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in
  `as_cp_user'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in
  `as_remote_user'"
- "/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
  `block in run'"
- "/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in
  `wrap'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in
  `restore_current_request_id'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in
  `restore_curent_timezone'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
  `restore_current_taxonomies'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in
  `block in run'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in
  `restore_curent_user'"
- "/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in
  `run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels)
  in execute_run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels)
  in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block
  in invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels)
  in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'"
- "/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'"
- "/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block
  in create_with_logging_context'"

Output of the correction dry-run: (some repos might look like a duplicate, but they actually have a different label and are in different Products)

# CONTENT_VIEW=Pottery foreman-rake katello:correct_repositories
Rubocop not loaded.
All operations will be skipped.  Re-run with COMMIT=true to perform corrections.
Processing Repository 1/175: CentOS 8 - BaseOS (63)
Processing Repository 2/175: Docker CE EL 8 - Stable (64)
Processing Repository 3/175: CentOS 8 - PowerTools (65)
Processing Repository 4/175: Extra Packages for Enterprise Linux 8 (66)
Processing Repository 5/175: CentOS 8 - Extras (67)
Processing Repository 6/175: Foreman Client EL8 (68)
Processing Repository 7/175: Microsoft CentOS 8 - Production (69)
Processing Repository 8/175: Elastic EL - 7.x (70)
Processing Repository 9/175: Puppet EL8 (71)
Processing Repository 10/175: CentOS 8 - Plus (72)
Processing Repository 11/175: CentOS 8 - Configmanagement Ansible 2.9 (73)
Processing Repository 12/175: Extra Packages for Enterprise Linux 8 Modular (74)
Processing Repository 13/175: Ripe Atlas Probe CentOS 8 (75)
Processing Repository 14/175: CentOS 8 - AppStream (76)
Processing Repository 15/175: Jenkins EL - Weekly Release (77)
Processing Repository 16/175: RockyLinux 8 - PowerTools (78)
Processing Repository 17/175: RockyLinux 8 - Extras (79)
Processing Repository 18/175: RockyLinux 8 - Plus (80)
Processing Repository 19/175: RockyLinux 8 - BaseOS (81)
Processing Repository 20/175: RockyLinux 8 - AppStream (82)
Processing Repository 21/175: Zabbix RHEL 8 - Non-Supported (83)
Processing Repository 22/175: Zabbix RHEL 8 - Supported (84)
Processing Repository 23/175: Extra Packages for Enterprise Linux 8 Modular (115)
Processing Repository 24/175: Puppet EL8 (116)
Processing Repository 25/175: Microsoft CentOS 8 - Production (117)
Processing Repository 26/175: CentOS 8 - Plus (118)
Processing Repository 27/175: Docker CE EL 8 - Stable (119)
Processing Repository 28/175: CentOS 8 - Configmanagement Ansible 2.9 (120)
Processing Repository 29/175: Elastic EL - 7.x (121)
Processing Repository 30/175: Ripe Atlas Probe CentOS 8 (122)
Processing Repository 31/175: RockyLinux 8 - Extras (123)
Processing Repository 32/175: RockyLinux 8 - AppStream (124)
Processing Repository 33/175: Zabbix RHEL 8 - Non-Supported (125)
Processing Repository 34/175: CentOS 8 - AppStream (126)
Processing Repository 35/175: Zabbix RHEL 8 - Supported (127)
Processing Repository 36/175: Foreman Client EL8 (128)
Processing Repository 37/175: Jenkins EL - Weekly Release (129)
Processing Repository 38/175: RockyLinux 8 - BaseOS (130)
Processing Repository 39/175: CentOS 8 - BaseOS (131)
Processing Repository 40/175: CentOS 8 - Extras (132)
Processing Repository 41/175: Extra Packages for Enterprise Linux 8 (133)
Processing Repository 42/175: CentOS 8 - PowerTools (135)
Processing Repository 43/175: RockyLinux 8 - PowerTools (136)
Processing Repository 44/175: RockyLinux 8 - Plus (137)
Processing Repository 45/175: Extra Packages for Enterprise Linux 8 Modular (138)
Processing Repository 46/175: Puppet EL8 (139)
Processing Repository 47/175: Microsoft CentOS 8 - Production (140)
Processing Repository 48/175: CentOS 8 - Plus (141)
Processing Repository 49/175: Docker CE EL 8 - Stable (142)
Processing Repository 50/175: CentOS 8 - Configmanagement Ansible 2.9 (143)
Processing Repository 51/175: Elastic EL - 7.x (144)
Processing Repository 52/175: Ripe Atlas Probe CentOS 8 (145)
Processing Repository 53/175: RockyLinux 8 - Extras (146)
Processing Repository 54/175: RockyLinux 8 - AppStream (147)
Processing Repository 55/175: Zabbix RHEL 8 - Non-Supported (148)
Processing Repository 56/175: CentOS 8 - AppStream (149)
Processing Repository 57/175: Zabbix RHEL 8 - Supported (150)
Processing Repository 58/175: Foreman Client EL8 (151)
Processing Repository 59/175: Jenkins EL - Weekly Release (152)
Processing Repository 60/175: RockyLinux 8 - BaseOS (153)
Processing Repository 61/175: CentOS 8 - BaseOS (154)
Processing Repository 62/175: CentOS 8 - Extras (155)
Processing Repository 63/175: Extra Packages for Enterprise Linux 8 (156)
Processing Repository 64/175: CentOS 8 - PowerTools (158)
Processing Repository 65/175: RockyLinux 8 - PowerTools (159)
Processing Repository 66/175: RockyLinux 8 - Plus (160)
Processing Repository 67/175: dnsdist (424)
Processing Repository 68/175: PowerDNS Authoritative Server (425)
Processing Repository 69/175: PowerDNS Recursor (426)
Processing Repository 70/175: PowerDNS Recursor (442)
Processing Repository 71/175: PowerDNS Authoritative Server (443)
Processing Repository 72/175: dnsdist (444)
Processing Repository 73/175: PowerDNS Recursor (445)
Processing Repository 74/175: PowerDNS Authoritative Server (446)
Processing Repository 75/175: dnsdist (447)
Processing Repository 76/175: CentOS 8 - AppStream (858)
Processing Repository 77/175: CentOS 8 - BaseOS (859)
Processing Repository 78/175: CentOS 8 - Configmanagement Ansible 2.9 (860)
Processing Repository 79/175: CentOS 8 - Extras (861)
Processing Repository 80/175: CentOS 8 - Plus (862)
Processing Repository 81/175: CentOS 8 - PowerTools (863)
Processing Repository 82/175: Docker CE EL 8 - Stable (864)
Processing Repository 83/175: Elastic EL - 7.x (865)
Processing Repository 84/175: Extra Packages for Enterprise Linux 8 (866)
Processing Repository 85/175: Extra Packages for Enterprise Linux 8 Modular (867)
Processing Repository 86/175: Foreman Client EL8 (868)
Processing Repository 87/175: Jenkins EL - Weekly Release (869)
Processing Repository 88/175: Microsoft CentOS 8 - Production (870)
Processing Repository 89/175: Puppet EL8 (871)
Processing Repository 90/175: Ripe Atlas Probe CentOS 8 (872)
Processing Repository 91/175: RockyLinux 8 - AppStream (873)
Processing Repository 92/175: RockyLinux 8 - Extras (874)
Processing Repository 93/175: RockyLinux 8 - Plus (875)
Processing Repository 94/175: RockyLinux 8 - PowerTools (876)
Processing Repository 95/175: Zabbix RHEL 8 - Non-Supported (877)
Processing Repository 96/175: Zabbix RHEL 8 - Supported (878)
Processing Repository 97/175: RockyLinux 8 - BaseOS (879)
Processing Repository 98/175: dnsdist (880)
Processing Repository 99/175: PowerDNS Authoritative Server (881)
Processing Repository 100/175: PowerDNS Recursor (882)
Processing Repository 101/175: CentOS 8 - AppStream (898)
Processing Repository 102/175: CentOS 8 - BaseOS (899)
Processing Repository 103/175: CentOS 8 - Configmanagement Ansible 2.9 (900)
Processing Repository 104/175: CentOS 8 - Extras (901)
Processing Repository 105/175: CentOS 8 - Plus (902)
Processing Repository 106/175: CentOS 8 - PowerTools (903)
Processing Repository 107/175: Docker CE EL 8 - Stable (904)
Processing Repository 108/175: Elastic EL - 7.x (905)
Processing Repository 109/175: Extra Packages for Enterprise Linux 8 (906)
Processing Repository 110/175: Extra Packages for Enterprise Linux 8 Modular (907)
Processing Repository 111/175: Foreman Client EL8 (908)
Processing Repository 112/175: Jenkins EL - Weekly Release (909)
Processing Repository 113/175: Microsoft CentOS 8 - Production (910)
Processing Repository 114/175: Puppet EL8 (911)
Processing Repository 115/175: Ripe Atlas Probe CentOS 8 (912)
Processing Repository 116/175: RockyLinux 8 - AppStream (913)
Processing Repository 117/175: RockyLinux 8 - Extras (914)
Processing Repository 118/175: RockyLinux 8 - Plus (915)
Processing Repository 119/175: RockyLinux 8 - PowerTools (916)
Processing Repository 120/175: Zabbix RHEL 8 - Non-Supported (917)
Processing Repository 121/175: Zabbix RHEL 8 - Supported (918)
Processing Repository 122/175: RockyLinux 8 - BaseOS (919)
Processing Repository 123/175: dnsdist (920)
Processing Repository 124/175: PowerDNS Authoritative Server (921)
Processing Repository 125/175: PowerDNS Recursor (922)
Processing Repository 126/175: CentOS 8 - AppStream (938)
Processing Repository 127/175: CentOS 8 - BaseOS (939)
Processing Repository 128/175: CentOS 8 - Configmanagement Ansible 2.9 (940)
Processing Repository 129/175: CentOS 8 - Extras (941)
Processing Repository 130/175: CentOS 8 - Plus (942)
Processing Repository 131/175: CentOS 8 - PowerTools (943)
Processing Repository 132/175: Docker CE EL 8 - Stable (944)
Processing Repository 133/175: Elastic EL - 7.x (945)
Processing Repository 134/175: Extra Packages for Enterprise Linux 8 (946)
Processing Repository 135/175: Extra Packages for Enterprise Linux 8 Modular (947)
Processing Repository 136/175: Foreman Client EL8 (948)
Processing Repository 137/175: Jenkins EL - Weekly Release (949)
Processing Repository 138/175: Microsoft CentOS 8 - Production (950)
Processing Repository 139/175: Puppet EL8 (951)
Processing Repository 140/175: Ripe Atlas Probe CentOS 8 (952)
Processing Repository 141/175: RockyLinux 8 - AppStream (953)
Processing Repository 142/175: RockyLinux 8 - Extras (954)
Processing Repository 143/175: RockyLinux 8 - Plus (955)
Processing Repository 144/175: RockyLinux 8 - PowerTools (956)
Processing Repository 145/175: Zabbix RHEL 8 - Non-Supported (957)
Processing Repository 146/175: Zabbix RHEL 8 - Supported (958)
Processing Repository 147/175: RockyLinux 8 - BaseOS (959)
Processing Repository 148/175: dnsdist (960)
Processing Repository 149/175: PowerDNS Authoritative Server (961)
Processing Repository 150/175: PowerDNS Recursor (962)
Processing Repository 151/175: Puppet EL8 (1028)
Processing Repository 152/175: Zabbix RHEL 8 - Non-Supported (1029)
Processing Repository 153/175: Zabbix RHEL 8 - Supported (1030)
Processing Repository 154/175: CentOS 8 - Plus (1031)
Processing Repository 155/175: CentOS 8 - Extras (1032)
Processing Repository 156/175: CentOS 8 - PowerTools (1033)
Processing Repository 157/175: CentOS 8 - BaseOS (1034)
Processing Repository 158/175: CentOS 8 - Configmanagement Ansible 2.9 (1035)
Processing Repository 159/175: Extra Packages for Enterprise Linux 8 Modular (1036)
Processing Repository 160/175: CentOS 8 - AppStream (1037)
Processing Repository 161/175: Docker CE EL 8 - Stable (1038)
Processing Repository 162/175: dnsdist (1039)
Processing Repository 163/175: Extra Packages for Enterprise Linux 8 (1040)
Processing Repository 164/175: PowerDNS Authoritative Server (1041)
Processing Repository 165/175: Jenkins EL - Weekly Release (1042)
Processing Repository 166/175: Foreman Client EL8 (1043)
Processing Repository 167/175: PowerDNS Recursor (1044)
Processing Repository 168/175: RockyLinux 8 - Plus (1045)
Processing Repository 169/175: Ripe Atlas Probe CentOS 8 (1046)
Processing Repository 170/175: RockyLinux 8 - PowerTools (1047)
Processing Repository 171/175: Microsoft CentOS 8 - Production (1048)
Processing Repository 172/175: RockyLinux 8 - Extras (1049)
Processing Repository 173/175: Elastic EL - 7.x (1050)
Processing Repository 174/175: RockyLinux 8 - AppStream (1051)
Processing Repository 175/175: RockyLinux 8 - BaseOS (1052)

And then the pluck:

# foreman-rake console
Rubocop not loaded.
Loading production environment (Rails 6.0.3.7)
irb(main):001:0> ::Katello::ContentViewVersion.find(40).repositories.archived.pluck(:id, :version_href)
=> [[951, "/pulp/api/v3/repositories/rpm/rpm/41651400-1b27-453e-8357-7d336e43a037/versions/43/"], [957, "/pulp/api/v3/repositories/rpm/rpm/f8447452-8426-4f4f-b2bd-6ae4af810702/versions/0/"], [958, "/pulp/api/v3/repositories/rpm/rpm/d15e0544-5453-44a0-a603-775bbf75916b/versions/43/"], [942, "/pulp/api/v3/repositories/rpm/rpm/35fd2ed5-3de1-4df4-aa89-d7ec2dcc17a9/versions/6/"], [941, "/pulp/api/v3/repositories/rpm/rpm/a3f19b67-9230-47e6-b4f7-2cd20afa05a0/versions/4/"], [943, "/pulp/api/v3/repositories/rpm/rpm/96ddda3d-0ef6-437a-83f0-bdf11eb8cf95/versions/9/"], [939, "/pulp/api/v3/repositories/rpm/rpm/62b4f1b2-6e96-4655-8813-8cf458dbb6ac/versions/9/"], [940, "/pulp/api/v3/repositories/rpm/rpm/9e30384a-67d4-4f47-9d3e-41833864e3fd/versions/3/"], [947, "/pulp/api/v3/repositories/rpm/rpm/b0641bd9-35a9-4c73-9556-5f839d6d5052/versions/16/"], [938, "/pulp/api/v3/repositories/rpm/rpm/94731697-e5fc-45ae-b9a3-e387ac7975f8/versions/14/"], [944, "/pulp/api/v3/repositories/rpm/rpm/9dfe4bbe-68f8-49b1-9668-f6ff1b50d902/versions/6/"], [960, "/pulp/api/v3/repositories/rpm/rpm/53f6dd05-3c78-45f3-95b6-de4d4f38be53/versions/1/"], [946, "/pulp/api/v3/repositories/rpm/rpm/0bfecd43-4ae7-4598-84e9-88349c63ff77/versions/37/"], [961, "/pulp/api/v3/repositories/rpm/rpm/7c1589fe-dcf1-4087-8a65-ea3c6e7f755a/versions/3/"], [949, "/pulp/api/v3/repositories/rpm/rpm/53665568-635f-4808-95fd-b9d2c06c24fa/versions/21/"], [948, "/pulp/api/v3/repositories/rpm/rpm/2d254073-5205-4342-8e57-b8e7beb5d265/versions/4/"], [962, "/pulp/api/v3/repositories/rpm/rpm/35f210ad-7a3c-4d67-8dfc-fe3c505c5837/versions/2/"], [955, "/pulp/api/v3/repositories/rpm/rpm/1c6e94bd-37ab-44b2-8038-1b1eca1f0131/versions/9/"], [952, "/pulp/api/v3/repositories/rpm/rpm/04657cfe-a31a-436a-bd02-834e0021aa2d/versions/4/"], [956, "/pulp/api/v3/repositories/rpm/rpm/b34d2025-e228-47b1-a135-a64aab424ac6/versions/33/"], [950, "/pulp/api/v3/repositories/rpm/rpm/30e11fa7-6c35-4026-8adf-3b9c11fc4f0a/versions/17/"], [954, "/pulp/api/v3/repositories/rpm/rpm/4e855069-4680-4e1c-a3e2-101b13cd2f2e/versions/16/"], [945, "/pulp/api/v3/repositories/rpm/rpm/5c0299e6-b9c7-42f8-adaf-d5727f41bb53/versions/10/"], [953, "/pulp/api/v3/repositories/rpm/rpm/0f5fcd53-29ae-4ebc-9cca-c5ce543e120c/versions/33/"], [959, "/pulp/api/v3/repositories/rpm/rpm/1683f5fb-1f72-4d43-a7e6-5a75317cb83b/versions/19/"]]

Thank you for your help! :slight_smile:

I was able to reproduce the issue. Seems to be a legit bug. We’ll take a look and keep you updated on any fixes and/or workarounds.

2 Likes

Okay thank you!

Thank you @iballou I experienced this exact same issue-- this is the first time I’ve tried to do an incremental CV (same reason… PolKit… yay). My install is a fresh 4.1 install, no upgrades from 3.x .

Looking forward to the fix/workaround/etc

1 Like

@iballou Just checkig in about a month later. Any update? Kind of a big deal for me, since using katello to patch my servers (and in emergency cases, “push through” new critical patches as in the case of POLKIT last month) is a really big deal.

I knoww pre-4.xx the incremental updates worked really well for me. Can’t wait to have it working again :slight_smile:

You may not have noticed, but in the other linked thread there iballou shared the issue in Redmine: Bug #34357: Incremental CV update fails with 400 HTTP error - Katello - Foreman
Looks like it will at least be available in Katello 4.5
Maybe we will get a backport, but let’s see :slight_smile:

2 Likes

Oh sweet! I did not see that. Excellent! So yes while 4.5 is likely a ways off, at the same time knowing the bug has been verified and fixed is a HUGE win and I am pleased.

This is after all a community effort, and it would be unrealistic to expect an “instant fix” except for the most critical of errors.

Thank for for the update and letting me know!

2 Likes

Thanks for posting that @lumarel! It’s triaged to Katello 4.3.1, so it’ll make it back that far at least. Katello 4.3.1 should be releasing soon too.

2 Likes

Cool! Thank you for the further update :slight_smile:

Did this make it into 4.4.0 possibly?

Yep it did, interesting, I thought a link would appear here to my other thread :slight_smile:

But there will also be the 4.3.1 release soon if I saw it correctly, which should also have the patch :+1:t2: