Remove orphan fails

Support
1

Problem:
remove orphan fails

undefined method `all?’ for :_created:Symbol

Expected outcome:

Foreman and Proxy versions:

foreman-tasks The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks. Ivan Nečas 7.0.0
foreman_ansible Ansible integration with Foreman Daniel Lobato Garcia 9.0.1
foreman_remote_execution A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality. Foreman Remote Execution team 8.0.0
foreman_templates Engine to synchronise provisioning templates from GitHub Greg Sutcliffe 9.3.0
katello Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp. N/A 4.6.0

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

  • “/usr/share/gems/gems/pulp_rpm_client-3.17.12/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:146:in
    `list_with_http_info’”
  • “/usr/share/gems/gems/pulp_rpm_client-3.17.12/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:109:in
    `list’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in
    `block in versions_list_for_repository’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:245:in
    `block in fetch_from_list’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in
    `loop’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in
    `fetch_from_list’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in
    `versions_list_for_repository’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:29:in
    `block (2 levels) in orphan_repository_versions’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in
    `collect’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in
    `block in orphan_repository_versions’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in
    `each’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in
    `orphan_repository_versions’”
  • “/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:39:in
    `delete_orphan_repository_versions’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:582:in `block (3 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in
    `block in run’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:40:in
    `block in as_remote_user’”
  • “/usr/share/gems/gems/katello-4.6.0/app/models/katello/concerns/user_extensions.rb:21:in
    `cp_config’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:27:in
    `as_cp_user’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:39:in
    `as_remote_user’”
  • “/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in
    `block in run’”
  • “/usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in
    `wrap’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:17:in `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in
    `restore_current_request_id’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in
    `restore_curent_timezone’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in
    `restore_current_taxonomies’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `block in run’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in
    `restore_curent_user’”
  • “/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in
    `run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:581:in `block (2 levels)
    in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `block in execute_run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `block in with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `catch’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `with_error_handling’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:575:in `execute_run’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
    `block (2 levels) in execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in
    `with_meta_calculation’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
    `block in execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
    `open_action’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
    `execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director.rb:69:in `execute’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
    `block (2 levels) in perform’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in `run_user_code’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
    `block in perform’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
    `with_telemetry’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
    `perform’”
  • “/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in
    `perform’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels)
    in process’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block
    in invoke’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in module:Sidekiq’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels)
    in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog’”
  • “/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread’”
  • “/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block
    in create_with_logging_context’”
2

Having same error with 4.6.0, whenever remove orphans is run.

“undefined method `all?’ for :_created:Symbol”

3

@pulp Looks like a pulp issue to me.

4

when i run manually
foreman-rake katello:delete_orphaned_content RAILS_ENV=production

i see 2 tasks getting created
the first run is fine
the second one fails with above error

5

Could you provide /var/log/messages and /var/log/foreman/production.log from the box? Thanks.

6

It is tracked here: Bug #35657: Remove orphan fails - Foreman

7

The issue is still in the official katello 4.6 release. The task for the main server works fine. The task for the content proxy fails and results in a paused task.

Same error here. Considering where it fails my guess would be it’s an issue in pulp_rpm? @dralley

2022-11-06T22:01:58 [E|bac|] undefined method `all?' for :_created:Symbol (NoMethodError)
 | /usr/share/gems/gems/pulp_rpm_client-3.17.12/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:146:in `list_with_http_info'
 | /usr/share/gems/gems/pulp_rpm_client-3.17.12/lib/pulp_rpm_client/api/repositories_rpm_versions_api.rb:109:in `list'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in `block in versions_list_for_repository'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:245:in `block in fetch_from_list'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in `loop'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in `fetch_from_list'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in `versions_list_for_repository'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:29:in `block (2 levels) in orphan_repository_versions'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in `collect'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in `block in orphan_repository_versions'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in `each'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in `orphan_repository_versions'
 | /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:39:in `delete_orphan_repository_versions'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
 | /usr/share/gems/gems/katello-4.6.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
 | /usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in `wrap'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:17:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
 | /usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in `execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `catch'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `block in execute_run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `block in with_error_handling'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `catch'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `with_error_handling'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:575:in `execute_run'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in `execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director.rb:69:in `execute'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in `run_user_code'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
 | /usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
 | [ sidekiq ]
 | [ concurrent-ruby ]
2022-11-06T22:01:58 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: bff0fd76-0d6e-4a94-a65d-653e1995351c, execution_plan_id: 9d2d7def-7c43-425f-a22a-538d17bd336b} state changed: paused  result: error
8

Same issue here on two environments

9

Probably there is something left behind in the database from before that should not be there, even though the logic bug is resolved.

Do you have the traceback from the inner task, from Pulp?

10

Where would I find that? I looks to me as if there is no pulp task associated.

What I can see is that the main server accesses

[08/Nov/2022:14:26:14 +0000] "GET /pulp/api/v3/repositories/rpm/rpm/?limit=2000&offset=0 HTTP/1.1" 200 390485 "-" "OpenAPI-Generator/3.17.12/ruby"

on the content proxy. And then there is the traceback. So my guess is, that the json returned (which is fine) causes the error on the main server. But I don’t think it started any pulp task. I cannot see anything related in the pulpcore-worker logs on either main server or content proxy at that time.

I can see in the task logs before the upgrade to 4.6 that it generated a long list of pulp tasks. But not now.

11

same issue on katello 4.5

Current versions:

  • foreman-3.3.1
  • katello-4.5.1
  • rubygem-foreman-tasks-6.0.2
  • python39-pulp-certguard-1.5.2-3
  • python39-pulp-cli-0.14.0-4
  • python39-pulp-rpm-3.18.7
  • python39-pulpcore-3.18.10

Logs:

root@foreman01 ~log # grep 23d2b202a2f940bcb90e20851fc33ff5 messages
Nov 14 15:11:19 foreman01 pulpcore-api[448113]: pulp [23d2b202a2f940bcb90e20851fc33ff5]: django.request:WARNING: Not Found: /pulp/api/v3/repositories/container/container/
Nov 14 15:11:19 foreman01 pulpcore-api[448113]: pulp [23d2b202a2f940bcb90e20851fc33ff5]:
12

Redmine issue with fix is above.

13

The fix did not work for me, i think my issue is different:

Nov 16 07:37:45 foreman01 pulpcore-api[547510]: pulp [7a915d0135244b43ab77b8b1bd93dcf7]: - - [16/Nov/2022:06:37:45 +0000] "GET /pulp/api/v3/repositories/container/container/?limit=2000&offset=0 HTTP/1.1" 404 179 "-" "OpenAPI-Generator/2.10.3/ruby"

> /usr/share/gems/gems/pulp_container_client-2.10.3/lib/pulp_container_client/api_client.rb:83:in `call_api'
> /usr/share/gems/gems/pulp_container_client-2.10.3/lib/pulp_container_client/api/repositories_container_api.rb:520:in `list_with_http_info'
> /usr/share/gems/gems/pulp_container_client-2.10.3/lib/pulp_container_client/api/repositories_container_api.rb:454:in `list'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:219:in `block in list_all'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:245:in `block in fetch_from_list'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:240:in `loop'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:240:in `fetch_from_list'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:218:in `list_all'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/api/core.rb:183:in `repository_versions'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/smart_proxy_repository.rb:59:in `block in orphan_repository_versions'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/smart_proxy_repository.rb:57:in `each'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/smart_proxy_repository.rb:57:in `orphan_repository_versions'
> /usr/share/gems/gems/katello-4.5.1/app/services/katello/pulp3/smart_proxy_repository.rb:39:in `delete_orphan_repository_versions'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
> /usr/share/gems/gems/katello-4.5.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
> /usr/share/gems/gems/katello-4.5.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
> /usr/share/gems/gems/activesupport-6.0.4.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in `pass'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in `pass'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
> /usr/share/gems/gems/foreman-tasks-6.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in `call'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in `execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `catch'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in `block in execute_run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `block in with_error_handling'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `catch'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in `with_error_handling'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in `execute_run'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in `execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in `execute'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in `run_user_code'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
> /usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog'
> /usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread'
> /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
14

Fixes #35657 - Fix orphan delete by sjha4 · Pull Request #10340 · Katello/katello · GitHub fixes the remove orphan run for the content proxy, but no it fails on the main server due to an error with pulp_deb:

Actions::Pulp3::OrphanCleanup::DeleteOrphanRepositoryVersions

Input:

{"smart_proxy_id"=>1,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_organization_id"=>nil,
 "current_location_id"=>nil,
 "current_user_id"=>1}

Output:

{}

Exception:

PulpDebClient::ApiError: Error message: the server returns an error
HTTP status code: 500
Response headers: {"date"=>"Thu, 24 Nov 2022 08:12:11 GMT", "server"=>"gunicorn", "content-type"=>"text/html; charset=UTF-8", "x-frame-options"=>"DENY", "content-length"=>"145", "vary"=>"Cookie", "x-content-type-options"=>"nosniff", "referrer-policy"=>"same-origin", "correlation-id"=>"7b661436db6f4000b0320860313ab068", "access-control-expose-headers"=>"Correlation-ID", "via"=>"1.1 foreman8.example.com", "connection"=>"close"}
Response body: 
<!doctype html>
<html lang="en">
<head>
  <title>Server Error (500)</title>
</head>
<body>
  <h1>Server Error (500)</h1><p></p>
</body>
</html>

Backtrace:

/usr/share/gems/gems/pulp_deb_client-2.18.1/lib/pulp_deb_client/api_client.rb:83:in `call_api'
/usr/share/gems/gems/pulp_deb_client-2.18.1/lib/pulp_deb_client/api/repositories_apt_versions_api.rb:196:in `list_with_http_info'
/usr/share/gems/gems/pulp_deb_client-2.18.1/lib/pulp_deb_client/api/repositories_apt_versions_api.rb:109:in `list'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in `block in versions_list_for_repository'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:245:in `block in fetch_from_list'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in `loop'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:240:in `fetch_from_list'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:194:in `versions_list_for_repository'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:187:in `block in repository_versions'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:186:in `collect'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/api/core.rb:186:in `repository_versions'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:59:in `block in orphan_repository_versions'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:57:in `each'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:57:in `orphan_repository_versions'
/usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_repository.rb:39:in `delete_orphan_repository_versions'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.6.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.6.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:192:in `execute_job'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:164:in `block in process'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:109:in `local'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:250:in `stats'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_logger.rb:8:in `call'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/job_retry.rb:74:in `global'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:125:in `block in dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:48:in `with_context'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:124:in `dispatch'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:163:in `process'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:83:in `process_one'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/processor.rb:71:in `run'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:16:in `watchdog'
/usr/share/gems/gems/sidekiq-5.2.10/lib/sidekiq/util.rb:25:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
15

Running 3.4.1 and noticed 28 failed/paused “Remove orphans” tasks. Related to the problem reported in this thread? Safe to just cancel and remove?

image
image1239×901 127 KB

image
image1020×573 42.9 KB

16

After upgrading to Foreman 3.5.1/Katello 4.7 I get the same errors:

image
image1540×1032 281 KB

17

You’ll really need to post the text of the error from the task details error tab. And please post the text and put it into a preformatted text block and don’t post screenshots.

Do you really have 8 content proxies?

Did you upgrade the content proxies as well?

18

Yes, 8 content proxies deployed all also running the latest version.

Some logs from one of errors:

Action:
Actions::Pulp3::OrphanCleanup::DeleteOrphanRepositoryVersions
Input:
{"smart_proxy_id"=>7,
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"UTC",
 "current_organization_id"=>nil,
 "current_location_id"=>nil,
 "current_user_id"=>1}
Output:
{}
Exception:
NoMethodError: undefined method `all?' for :_created:Symbol
Backtrace:
/usr/share/gems/gems/pulp_container_client-2.14.2/lib/pulp_container_client/api/repositories_container_versions_api.rb:146:in `list_with_http_info'
/usr/share/gems/gems/pulp_container_client-2.14.2/lib/pulp_container_client/api/repositories_container_versions_api.rb:109:in `list'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/api/core.rb:194:in `block in versions_list_for_repository'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/api/core.rb:245:in `block in fetch_from_list'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/api/core.rb:240:in `loop'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/api/core.rb:240:in `fetch_from_list'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/api/core.rb:194:in `versions_list_for_repository'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:29:in `block (2 levels) in orphan_repository_versions'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in `collect'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:28:in `block in orphan_repository_versions'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in `each'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb:25:in `orphan_repository_versions'
/usr/share/gems/gems/katello-4.7.0/app/services/katello/pulp3/smart_proxy_repository.rb:39:in `delete_orphan_repository_versions'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/pulp3/orphan_cleanup/delete_orphan_repository_versions.rb:10:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:582:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/usr/share/gems/gems/katello-4.7.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/usr/share/gems/gems/katello-4.7.0/app/lib/actions/middleware/remote_action.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/usr/share/gems/gems/activesupport-6.1.7/lib/active_support/execution_wrapper.rb:91:in `wrap'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/load_setting_values.rb:20:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user'
/usr/share/gems/gems/foreman-tasks-7.1.0/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:581:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:580:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `catch'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:483:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:575:in `execute_run'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/action.rb:296:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:196:in `execute_job'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:164:in `block (2 levels) in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:163:in `block in process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:136:in `block (6 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:112:in `local'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:135:in `block (5 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq.rb:39:in `block in <module:Sidekiq>'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:131:in `block (4 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:257:in `stats'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:126:in `block (3 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:13:in `call'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:125:in `block (2 levels) in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_retry.rb:79:in `global'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:124:in `block in dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/logger.rb:11:in `with'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/job_logger.rb:33:in `prepare'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:123:in `dispatch'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:162:in `process'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:78:in `process_one'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/processor.rb:68:in `run'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:43:in `watchdog'
/usr/share/gems/gems/sidekiq-6.3.1/lib/sidekiq/util.rb:52:in `block in safe_thread'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
19

Looking back in the task history I see that these errors started to come after upgrading from 3.3.1/4.5.1 to 3.4.0/4.6.0 back in November. I also see that this task only completes successfully on the Foreman server.

20

It seems the patch is not in 4.7.0: katello/CHANGELOG.md at KATELLO-4.7 · Katello/katello · GitHub

It’s in the queue for 4.6.1, I think Fixes #35657 - Fix orphan delete (#10340) · Katello/katello@b216bff · GitHub

So I guess in the meantime, all you can do is to apply the change manually, i.e. make the two changes from the pull request to /usr/share/gems/gems/katello-4.6.0/app/services/katello/pulp3/smart_proxy_mirror_repository.rb on the main server.

Then restart foreman services on the main server. That should do it. You’ll have to reapply this if it won’t make it into 4.7.1 either. The rpm in question is rubygem-katello.