Katello 4.0 katello:delete_orphaned_content

Hi

I have problem with proper execution of

foreman-rake katello:delete_orphaned_content RAILS_ENV=production

In logs i get such error

RuntimeError: There was an issue with the backend service pulp: 404 Not Found

I have just remove all pulp2 references with:

foreman-maintain content remove-pulp2

On smart proxies I can see references to pulp2 initated by master :

“GET /pulp/api/v2//status HTTP/1.1” 404 216 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”

Any ideas?

  • Katello 4.0
  • Foreman 2.4

https://projects.theforeman.org/issues/32475
It’s planned to be fixed in Katello 4.0.1.

2 Likes

I have Katello 4.0.1 with the fix in that issue. However, I still see the same error message when running

# foreman-rake katello:delete_orphaned_content RAILS_ENV=production --trace
** Invoke katello:delete_orphaned_content (first_time)
** Invoke dynflow:client (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute dynflow:client
** Execute katello:delete_orphaned_content

Task Log:

There was an issue with the backend service pulp: 404 Not Found

production.log:

2021-06-04T14:37:59 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: c91ebbd2-5f92-4370-8d16-7eeee4b21944, execution_plan_id: 3769b7ea-31fa-4bdf-8027-026991029a3c} state changed: planning 
2021-06-04T14:37:59 [W|app|] 404 Not Found
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in `exception_with_response'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in `return!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:809:in `process_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in `block in transmit'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in `start'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in `transmit'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in `get'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:263:in `backend_status'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:172:in `pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:88:in `block in ping_pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:144:in `exception_watch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:87:in `ping_pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:236:in `ping_services_for_capsule'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:27:in `ping'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `block in plan'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/remote_action.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:513:in `block (2 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:417:in `concurrence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:512:in `block in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:511:in `execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:285:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:284:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/coordinator.rb:326:in `acquire'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:209:in `block in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:204:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:180:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:23:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:29:in `block in trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:27:in `trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:54:in `async_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:18:in `remove_orphan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:12:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `reverse_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
2021-06-04T14:37:59 [W|app|] Skipped pulp_auth check after failed pulp check
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:105:in `block in ping_pulp_with_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:144:in `exception_watch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:101:in `ping_pulp_with_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:239:in `ping_services_for_capsule'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/models/katello/ping.rb:27:in `ping'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `block in plan'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:108:in `as'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:114:in `as_anonymous_admin'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:13:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/remote_action.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:513:in `block (2 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:417:in `concurrence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:512:in `block in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:511:in `execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:285:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:284:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/coordinator.rb:326:in `acquire'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:209:in `block in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:204:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:180:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:23:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:29:in `block in trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:27:in `trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:54:in `async_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:18:in `remove_orphan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:12:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `reverse_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
2021-06-04T14:37:59 [E|bac|] There was an issue with the backend service pulp: 404 Not Found (RuntimeError)
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:17:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:15:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/backend_services_check.rb:15:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/app/lib/actions/middleware/remote_action.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:34:in `with_current_request_id'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_request_id.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:36:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:41:in `with_current_user'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:513:in `block (2 levels) in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:417:in `concurrence'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:512:in `block in execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `block in with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in `with_error_handling'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:511:in `execute_plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:55:in `block in execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:384:in `switch_flow'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:285:in `block in plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:17:in `block in rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/transactions.rb:212:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/transaction_adapters/active_record.rb:6:in `transaction'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:16:in `rollback_on_error'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/common/transaction.rb:6:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in `pass'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:44:in `plan_phase'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan.rb:284:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:211:in `block (2 levels) in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/coordinator.rb:326:in `acquire'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:209:in `block in plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `tap'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:208:in `plan_with_options'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:204:in `plan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/world.rb:180:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:23:in `trigger'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:29:in `block in trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:27:in `trigger_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.0.1/lib/foreman_tasks.rb:54:in `async_task'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:18:in `remove_orphan'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:12:in `block (3 levels) in <top (required)>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `reverse_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.1.1/lib/katello/tasks/delete_orphaned_content.rake:11:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
2021-06-04T14:37:59 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: c91ebbd2-5f92-4370-8d16-7eeee4b21944, execution_plan_id: 3769b7ea-31fa-4bdf-8027-026991029a3c} state changed: stopped  result: error
2021-06-04T14:37:59 [I|bac|] Task {label: Actions::Katello::OrphanCleanup::RemoveOrphans, id: c91ebbd2-5f92-4370-8d16-7eeee4b21944, execution_plan_id: 3769b7ea-31fa-4bdf-8027-026991029a3c} state changed: stopped  result: error
2021-06-04T14:37:59 [E|app|] Smart proxy with ID 6 may be down: There was an issue with the backend service pulp: 404 Not Found

I guess that call is the culprit for this error: it calls a ping on pulp2…

Created issue Bug #32738: Error 404 Not Found on katello:delete_orphaned_content - Katello - Foreman