Problem:
Foreman Reports paused tasks due to errors. Tasks affected are " Metadata generate" and “Remove orphans”
Expected outcome:
Tasks to complete without errors.
Foreman and Proxy versions:
foreman-installer-katello-2.0.0-1.el7.noarch
foreman-release-scl-7-2.el7.noarch
foreman.example.com-tomcat-1.0-1.noarch
foreman.example.com-qpid-router-server-1.0-1.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-foreman_vmwareannotations-0.0.1-4.fm2_0.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.10-1.fm2_0.el7.noarch
foreman-2.0.0-2.el7.noarch
foreman-vmware-2.0.0-2.el7.noarch
foreman.example.com-foreman-proxy-client-1.0-1.noarch
tfm-rubygem-foreman_templates-8.0.0-1.fm2_0.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-4.el7.noarch
foreman-telemetry-2.0.0-2.el7.noarch
foreman.example.com-qpid-client-cert-1.0-1.noarch
foreman.example.com-puppet-client-1.0-1.noarch
foreman-installer-2.0.0-1.el7.noarch
foreman-release-2.0.0-1.el7.noarch
foreman.example.com-qpid-broker-1.0-2.noarch
foreman.example.com-qpid-router-client-1.0-1.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_0.el7.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_0.el7.noarch
tfm-rubygem-foreman_bootdisk-16.0.0-2.fm2_0.el7.noarch
tfm-rubygem-foreman-tasks-1.1.1-1.fm2_0.el7.noarch
foreman-debug-2.0.0-2.el7.noarch
foreman-ec2-2.0.0-2.el7.noarch
foreman.example.com-foreman-client-1.0-1.noarch
foreman-proxy-2.0.0-1.el7.noarch
tfm-rubygem-foreman_azure_rm-2.0.8-1.fm2_0.el7.noarch
tfm-rubygem-foreman_remote_execution-3.2.1-1.fm2_0.el7.noarch
foreman-dynflow-sidekiq-2.0.0-2.el7.noarch
rubygem-foreman_maintain-0.6.2-1.el7.noarch
foreman.example.com-apache-1.0-1.noarch
tfm-rubygem-foreman_openscap-3.0.0-1.fm2_0.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_0.el7.noarch
foreman-cli-2.0.0-2.el7.noarch
foreman.example.com-foreman-proxy-1.0-1.noarch
tfm-rubygem-hammer_cli_foreman-2.0.2-1.el7.noarch
foreman-selinux-2.0.0-1.el7.noarch
tfm-rubygem-foreman_discovery-16.0.1-2.fm2_0.el7.noarch
tfm-rubygem-foreman_monitoring-1.0.1-4.fm2_0.el7.noarch
tfm-rubygem-hammer_cli_foreman_azure_rm-0.1.2-1.fm2_0.el7.noarch
foreman-postgresql-2.0.0-2.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.3-1.fm2_0.el7.noarch
tfm-rubygem-foreman_cockpit-2.0.3-4.fm1_23.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.1.2-1.fm2_0.el7.noarch
foreman-ovirt-2.0.0-2.el7.noarch
Distribution and version:
CentOS 7.8
Other relevant data:
Error on tasks as follows:
Action: Actions::Pulp3::Repository::CreatePublication
Output: {}
Exception: Faraday::ConnectionFailed: no implicit conversion of OpenSSL::X509::Certificate into String (TypeError)
Trace:
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/ssl_socket.rb:144:in `read'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/ssl_socket.rb:144:in `client_cert_data'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/ssl_socket.rb:68:in `initialize'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/connection.rb:404:in `new'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/connection.rb:404:in `socket'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/connection.rb:101:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/middlewares/mock.rb:48:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/middlewares/instrumentor.rb:26:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/middlewares/base.rb:16:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/middlewares/base.rb:16:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/middlewares/base.rb:16:in `request_call'
/opt/theforeman/tfm/root/usr/share/gems/gems/excon-0.58.0/lib/excon/connection.rb:250:in `request'
/usr/share/foreman/lib/foreman/http_proxy/excon_connection_extension.rb:13:in `request'
/opt/theforeman/tfm/root/usr/share/gems/gems/faraday-0.15.4/lib/faraday/adapter/excon.rb:52:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/faraday-0.15.4/lib/faraday/rack_builder.rb:143:in `build_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/faraday-0.15.4/lib/faraday/connection.rb:387:in `run_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/faraday-0.15.4/lib/faraday/connection.rb:175:in `post'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-0.1.0/lib/pulp_file_client/api_client.rb:67:in `public_send'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-0.1.0/lib/pulp_file_client/api_client.rb:67:in `call_api'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-0.1.0/lib/pulp_file_client/api/publications_file_api.rb:79:in `create_with_http_info'
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_file_client-0.1.0/lib/pulp_file_client/api/publications_file_api.rb:28:in `create'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/services/katello/pulp3/repository.rb:190:in `create_publication'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/pulp3/repository/create_publication.rb:15:in `invoke_external_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/polling.rb:84:in `initiate_external_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/polling.rb:19:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/pulp3/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:563:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/execute_if_contents_changed.rb:5:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.1.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:562:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:561:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:475:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:556:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/director.rb:68:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors.rb:12:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
I dont know if it has something to do with it but sync tasks get stuck every 2-3 days, pressing cancel on the sync tasks reports them as successfull synchronized. Workaround for me is to reboot Foreman every two days.