Problem with removing orphaned packages

Problem:
The Task “Remove orphans” runs into problem
Expected outcome:
Successfully deleting orphaned packages
Foreman and Proxy versions:
Foreman 2.0.3
Foreman and Proxy plugin versions:
katello 3.15.3.1
foreman-tasks 1.1.3
Distribution and version:
CentOS 7
Other relevant data:
Action:

Actions::Katello::OrphanCleanup::RemoveOrphans

Input:

{“services_checked”=>}

Output:

{}

Exception:

RuntimeError: There was an issue with the backend service pulp: Timed out reading data from server

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/middleware/backend_services_check.rb:17:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/middleware/backend_services_check.rb:15:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/middleware/backend_services_check.rb:15:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/app/lib/actions/middleware/remote_action.rb:9:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:10:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:31:in with_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_request_id.rb:9:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:10:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:31:in with_current_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_timezone.rb:9:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:10:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:31:in with_current_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_user.rb:9:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in with_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:36:in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:516:in block (2 levels) in execute_plan’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in switch_flow' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:420:in concurrence’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:515:in block in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in block in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:475:in with_error_handling’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:514:in execute_plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/action.rb:285:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:55:in block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan/steps/plan_step.rb:54:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:286:in block (2 levels) in plan’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:384:in switch_flow' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:374:in with_planning_scope’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:285:in block in plan' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:44:in plan_phase’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:17:in block in rollback_on_error’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in block in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in block in within_new_transaction’
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in within_new_transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in transaction’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/transaction_adapters/active_record.rb:6:in transaction' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:16:in rollback_on_error’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/common/transaction.rb:6:in plan_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/execution_plan.rb:284:in plan’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:211:in block (2 levels) in plan_with_options' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/coordinator.rb:326:in acquire’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:209:in block in plan_with_options' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in tap’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:208:in plan_with_options' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:204:in plan’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.6/lib/dynflow/world.rb:180:in trigger' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:23:in trigger’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:29:in block in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:49:in block in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in block in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in yield_shares’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in permit_concurrent_loads' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:48:in rails_safe_trigger_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:27:in trigger_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.3/lib/foreman_tasks.rb:54:in async_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/tasks/delete_orphaned_content.rake:20:in remove_orphan' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/tasks/delete_orphaned_content.rake:14:in block (3 levels) in <top (required)>’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/tasks/delete_orphaned_content.rake:13:in reverse_each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.3.1/lib/katello/tasks/delete_orphaned_content.rake:13:in block (2 levels) in <top (required)>’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain’
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in load’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `’

Hi,

anyone got an idea ?

Regards
Frank

Hi,

does anyone know how to analyse this problem ?
It looks like as “foreman-rake katello:delete_orphaned_content” triggers the removing orphan packages manually.
this command does not give out any errors and “echo $?” gives 0.
even adding “VERBOSE=true” doesn’t do anything here.
Can someone please help ?
A few hints on how to continue analysing this problem would be very welcome.

Kind Regards
Frank