Problem:
Since updating to the latest Katello it seems our Foreman instance has issues when running certain rake commands related to Katello or when remote execution tries to call back to the Foreman server. From the output below output when running “foreman-rake katello:correct_repositories” I’m assuming something has gone wrong with our SSL setup.
Though we have followed what is in the documentation, and only replaced the --certs-server related options with our certificates and keys. Our SSL setup has worked previously and passed the cert check tool.
So I’m not sure what has happened nor how to fix it (apart from setting SSLVerifyClient to optional_no_ca).
foreman-rake katello:correct_repositories output:
# foreman-rake katello:correct_repositories COMMIT=true --trace
** Invoke katello:correct_repositories (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Invoke environment
** Invoke dynflow:client (first_time)
** Invoke environment
** Execute dynflow:client
** Execute katello:check_ping
** Execute katello:correct_repositories
Processing Repository 1/496: EPEL 8 x86_64 (40)
rake aborted!
NoMethodError: undefined method `code' for #<Faraday::ConnectionFailed:0x0000560bc5f4c988>
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:94:in `rescue in repo_exists?'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:84:in `repo_exists?'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Caused by:
Faraday::ConnectionFailed: SSL_read: tlsv1 alert unknown ca (OpenSSL::SSL::SSLError)
/usr/share/ruby/openssl/buffering.rb:57:in `sysread'
/usr/share/ruby/openssl/buffering.rb:57:in `fill_rbuff'
/usr/share/ruby/openssl/buffering.rb:300:in `eof?'
/usr/share/ruby/openssl/buffering.rb:253:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:85:in `block in readline'
/usr/share/ruby/timeout.rb:95:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:110:in `timeout'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:84:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'
/usr/share/foreman/lib/foreman/http_proxy/excon_connection_extension.rb:13:in `request'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `block in call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/adapter.rb:50:in `connection'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/rack_builder.rb:154:in `build_response'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:516:in `run_request'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:202:in `get'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `public_send'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:702:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:649:in `read'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:88:in `repo_exists?'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Caused by:
Excon::Error::Socket: SSL_read: tlsv1 alert unknown ca (OpenSSL::SSL::SSLError)
/usr/share/ruby/openssl/buffering.rb:57:in `sysread'
/usr/share/ruby/openssl/buffering.rb:57:in `fill_rbuff'
/usr/share/ruby/openssl/buffering.rb:300:in `eof?'
/usr/share/ruby/openssl/buffering.rb:253:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:85:in `block in readline'
/usr/share/ruby/timeout.rb:95:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:110:in `timeout'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:84:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'
/usr/share/foreman/lib/foreman/http_proxy/excon_connection_extension.rb:13:in `request'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `block in call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/adapter.rb:50:in `connection'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/rack_builder.rb:154:in `build_response'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:516:in `run_request'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:202:in `get'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `public_send'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:702:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:649:in `read'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:88:in `repo_exists?'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Caused by:
OpenSSL::SSL::SSLError: SSL_read: tlsv1 alert unknown ca
/usr/share/ruby/openssl/buffering.rb:57:in `sysread'
/usr/share/ruby/openssl/buffering.rb:57:in `fill_rbuff'
/usr/share/ruby/openssl/buffering.rb:300:in `eof?'
/usr/share/ruby/openssl/buffering.rb:253:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:85:in `block in readline'
/usr/share/ruby/timeout.rb:95:in `block in timeout'
/usr/share/ruby/timeout.rb:33:in `block in catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:33:in `catch'
/usr/share/ruby/timeout.rb:110:in `timeout'
/usr/share/gems/gems/excon-0.104.0/lib/excon/socket.rb:84:in `readline'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:73:in `block in parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `loop'
/usr/share/gems/gems/excon-0.104.0/lib/excon/response.rb:72:in `parse'
/usr/share/gems/gems/excon-0.104.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:460:in `response'
/usr/share/gems/gems/excon-0.104.0/lib/excon/connection.rb:291:in `request'
/usr/share/foreman/lib/foreman/http_proxy/excon_connection_extension.rb:13:in `request'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `block in call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/adapter.rb:50:in `connection'
/usr/share/gems/gems/faraday-excon-1.1.0/lib/faraday/adapter/excon.rb:31:in `call'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/rack_builder.rb:154:in `build_response'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:516:in `run_request'
/usr/share/gems/gems/faraday-1.10.2/lib/faraday/connection.rb:202:in `get'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `public_send'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api_client.rb:69:in `call_api'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:702:in `read_with_http_info'
/usr/share/gems/gems/pulp_rpm_client-3.23.0/lib/pulp_rpm_client/api/repositories_rpm_api.rb:649:in `read'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:88:in `repo_exists?'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:41:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block (2 levels) in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:71:in `block in find_each'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:138:in `block in find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:245:in `block in in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `loop'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:229:in `in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/relation/batches.rb:70:in `find_each'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `with_index'
/usr/share/gems/gems/katello-4.11.1/lib/katello/tasks/repository.rake:39:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
/usr/bin/rake:23:in `load'
/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:correct_repositories
foreman-proxy log when running remote execution jobs
4be563a [E] <RuntimeError> Failed performing callback to Foreman server: 403 {
"error": {"message":"Access denied","details":"Missing one of the required permissions: ","missing_permissions":[]}
}
/usr/share/gems/gems/smart_proxy_dynflow-0.9.1/lib/smart_proxy_dynflow/callback.rb:18:in `callback'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.1/lib/smart_proxy_dynflow/action/batch_callback.rb:16:in `run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:589:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.1/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `block in run'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.1/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:51:in `restore_current_request_id'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.1/lib/smart_proxy_dynflow/middleware/keep_current_request_id.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:588:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `catch'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:587:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `catch'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:490:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:582:in `execute_run'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/action.rb:303:in `execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/director.rb:69:in `execute'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/actor.rb:122:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.8.2/lib/dynflow/actor.rb:56:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'