Problem:
Publish of RHEL8 suddenly fails with NoMethodError: undefined method `’ for nil:NilClass
Expected outcome:
Successfully Published
Foreman and Proxy versions:
foreman-2.3.5-1.el7.noarch
Foreman and Proxy plugin versions:
Distribution and version:
RHEL7
Other relevant data:
[root@ddclvusat001 oper]# systemctl status goferd -l
● goferd.service - Gofer Agent
Loaded: loaded (/usr/lib/systemd/system/goferd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2024-02-27 07:38:24 MST; 7min ago
Main PID: 1792 (python)
Tasks: 8
Memory: 33.2M
CGroup: /system.slice/goferd.service
└─1792 /usr/bin/python /usr/bin/goferd --foreground
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - return fn(*args, **kwargs)
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - File “/usr/lib/python2.7/site-packages/gofer/agent/plugin.py”, line 373, in attach
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - model.setup()
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - File “/usr/lib/python2.7/site-packages/gofer/common.py”, line 346, in _fn
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - return fn(*args, **kwargs)
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - File “/usr/lib/python2.7/site-packages/gofer/agent/plugin.py”, line 567, in setup
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - queue.declare(url)
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - File “/usr/lib/python2.7/site-packages/gofer/messaging/adapter/model.py”, line 45, in _fn
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - raise ModelError(*e.args)
Feb 27 07:39:27 ddclvusat001 goferd[1792]: [ERROR][worker-0] gofer.threadpool:100 - ModelError: Connection amqps://subscription.rhsm.redhat.com:5647 timed out: Opening connection
[root@ddclvusat001 oper]#
Publish for RHEL7 was successful. RHE:L8 failed
Output:
{}
Exception:
NoMethodError: undefined method []' for nil:NilClass Backtrace: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository.rb:199:in lookup_distributor_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository.rb:193:in block in distributor_publish' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository.rb:191:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository.rb:191:in map' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository.rb:191:in distributor_publish’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp/repository/distributor_publish.rb:24:in invoke_external_task' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:84:in initiate_external_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:19:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp/abstract_async_task.rb:45:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/skip_if_matching_content.rb:5:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:68:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
Already the context view is Any locations and still failing
After following Cannot promote to environments after upgrade to Foreman 2.0/Katello 3.15 - #3 by matemikulic
Now it is failing
RestClient::Exceptions::ReadTimeout: Timed out reading data from server
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:733:in rescue in transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in post' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/repository.rb:126:in unit_copy’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/extensions/unit.rb:112:in copy' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/services/katello/pulp/repository/yum.rb:118:in copy_contents’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp/repository/copy_all_units.rb:22:in invoke_external_task' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:84:in initiate_external_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/polling.rb:19:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/cancellable.rb:14:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/pulp/abstract_async_task.rb:45:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.5/app/lib/actions/middleware/remote_action.rb:16:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.6/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/director.rb:68:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.8/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
su - postgres -c “psql -d foreman -c ‘select label,count(label),state,result from foreman_tasks_tasks where state <> ‘'‘stopped’'’ group by label,state,result ORDER BY label;’”
and
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘result == error’ STATES=‘paused’ VERBOSE=true
and publish failed again with the same ```
Timed out reading data from server
RestClient::Exceptions::ReadTimeout: Timed out reading data from server
