Problem:
After upgrading to Katello 4.3 / Foreman 3.1.1 today I tried to publish an incremental update to my CVs because of the polkit issue.
This was actually the first time I tried this feature so I’m not sure if it worked before the upgrade.
In my case I was trying to apply RLSA-2022:267 to the CV (this was with Rocky Linux here)
(was not able to test it with any other CVs as I didn’t implement the external Errata repo creation for CentOS, or am not using AlmaLinux or others)
As far as I understood it, the job is looking for version 15 of the CV, but as I’m deleting all the versions except the last 3 from time to time, this is not available anymore. (versions 20.0 to 22.0 right now)
I was able to unlock the job by skipping this faulting step (Actions::Pulp3::Repository::MultiCopyUnits) in the Dynflow Console, so at least it doesn’t block any other actions in the CV
Maybe it’s an issue from my side, am I not supposed to delete old versions of the CVs?
Or can someone reproduce that?
If I missed to add some relevant information I will off course provide it later!
Expected outcome:
Creation of the incremental CV version successfully finishes.
Foreman and Proxy versions:
foreman-3.1.1-1.el8
katello-4.3.0-1.el8
Foreman and Proxy plugin versions:
ansible-collection-theforeman-foreman-3.0.0-1.el8.noarch
candlepin-4.1.8-1.el8.noarch
candlepin-selinux-4.1.8-1.el8.noarch
foreman-cli-3.1.1-1.el8.noarch
foreman-debug-3.1.1-1.el8.noarch
foreman-dynflow-sidekiq-3.1.1-1.el8.noarch
foreman-installer-3.1.1-1.el8.noarch
foreman-installer-katello-3.1.1-1.el8.noarch
foreman-postgresql-3.1.1-1.el8.noarch
foreman-proxy-3.1.1-1.el8.noarch
foreman-release-3.1.1-1.el8.noarch
foreman-selinux-3.1.1-1.el8.noarch
foreman-service-3.1.1-1.el8.noarch
foreman-vmware-3.1.1-1.el8.noarch
katello-ca-consumer-r8-foreman-prod.fritz.box-1.0-1.noarch
katello-certs-tools-2.8.2-1.el8.noarch
katello-client-bootstrap-1.7.7-1.el8.noarch
katello-common-4.3.0-1.el8.noarch
katello-debug-4.3.0-1.el8.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.5-2.el8.noarch
katello-host-tools-tracer-3.5.5-2.el8.noarch
katello-repos-4.3.0-1.el8.noarch
katello-selinux-4.0.2-1.el8.noarch
katello-server-ca-1.0-1.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.2.7-1.el8.x86_64
python38-pulp-ansible-0.10.1-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
python38-pulp-container-2.9.0-1.el8.noarch
python38-pulp-deb-2.16.1-1.el8.noarch
python38-pulp-file-1.10.1-1.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
python38-pulp-rpm-3.16.1-2.el8.noarch
python38-pulpcore-3.16.1-1.el8.noarch
qpid-proton-c-0.35.0-1.el8.x86_64
r8-foreman-prod.fritz.box-apache-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-client-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-proxy-1.0-1.noarch
r8-foreman-prod.fritz.box-foreman-proxy-client-1.0-1.noarch
r8-foreman-prod.fritz.box-puppet-client-1.0-1.noarch
rubygem-foreman-tasks-5.2.0-1.fm3_1.el8.noarch
rubygem-foreman-tasks-core-0.3.6-1.fm2_5.el8.noarch
rubygem-foreman_ansible-7.0.2-1.fm3_1.el8.noarch
rubygem-foreman_ansible_core-4.2.1-1.fm2_5.el8.noarch
rubygem-foreman_bootdisk-19.0.3-1.fm3_1.el8.noarch
rubygem-foreman_maintain-1.0.2-1.el8.noarch
rubygem-foreman_puppet-2.0.0-1.fm3_1.el8.noarch
rubygem-foreman_remote_execution-5.0.1-1.fm3_1.el8.noarch
rubygem-foreman_remote_execution_core-1.4.8-1.el8.noarch
rubygem-foreman_snapshot_management-2.0.1-1.fm2_6.el8.noarch
rubygem-foreman_statistics-2.0.0-1.fm3_1.el8.noarch
rubygem-foreman_templates-9.1.0-1.fm2_6.el8.noarch
rubygem-hammer_cli-3.1.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.1.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.3.4-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.4-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_templates-0.2.0-2.fm2_5.el8.noarch
rubygem-hammer_cli_katello-1.3.1-1.el8.noarch
rubygem-katello-4.3.0-1.el8.noarch
rubygem-pulp_ansible_client-0.10.1-1.el8.noarch
rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
rubygem-pulp_container_client-2.9.0-1.el8.noarch
rubygem-pulp_deb_client-2.16.0-1.el8.noarch
rubygem-pulp_file_client-1.10.0-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
rubygem-pulp_python_client-3.5.2-1.el8.noarch
rubygem-pulp_rpm_client-3.16.1-1.el8.noarch
rubygem-pulpcore_client-3.16.0-1.el8.noarch
rubygem-qpid_proton-0.35.0-1.el8.x86_64
rubygem-smart_proxy_pulp-3.2.0-2.fm3_1.el8.noarch
Distribution and version:
Rocky Linux 8.5
Other relevant data:
So, the actual error is the following:
Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Fri, 28 Jan 2022 00:46:19 GMT", "server"=>"gunicorn", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"DENY", "content-length"=>"73", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"4144ea82-7e26-4744-8f39-96c7f6a02c1f", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 <server-fqdn>", "connection"=>"close"}
Response body: ["Version 15 does not exist for repository 'rockylinux8_extras-569931'."]
And the backtrace shows the following:
/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api_client.rb:83:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:79:in `copy_content_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.16.1/lib/pulp_rpm_client/api/rpm_copy_api.rb:28:in `copy_content'
/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:153:in `copy_content_chunked'
/usr/share/gems/gems/katello-4.3.0/app/services/katello/pulp3/repository/yum.rb:126:in `multi_copy_units'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/repository/multi_copy_units.rb:49:in `invoke_external_task'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/polling.rb:19:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/cancellable.rb:14:in `run'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/pulp3/abstract_async_task.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.3.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.3.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-5.2.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'