Upgrade Katello 3.5/3.6 with foreman 1.16.1/1.17.1 fails at foreman-rake katello:correct_repositories

Hello,

After all the 1.17.0 issues, tried another upgrade again of katello 3.5/3.6 with foreman 1.16.1/1.17.1
Now the upgrade fails with;

foreman-installer --scenario katello --upgrade
Upgrade Step: correct_repositories (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.

You should use PG::Connection, PG::Result, and PG::Error instead, respectively.

Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
rake aborted!
RestClient::InternalServerError: 500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:31:in `retrieve'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:126:in `repo_exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:61:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:62:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:59:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:59:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories
(See full trace by running task with --trace)
Processing Repository 1/108: base_x86_64 (1)
foreman-rake katello:correct_repositories COMMIT=true failed! Check the output for error!
Upgrade step correct_repositories failed. Check logs for more information.

/var/log/foreman/production.log
2018-06-01T21:15:02  [app] [I] Current user: foreman_admin (administrator)
2018-06-01T21:15:02  [app] [W] 500 Internal Server Error
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:809:in `process_result'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:150:in `backend_status'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:114:in `pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:36:in `block in ping_pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:83:in `exception_watch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:35:in `ping_pulp_without_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:19:in `ping'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/reimport.rake:7:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `<main>'
2018-06-01T21:15:02  [app] [W] Skipped pulp_auth check after failed pulp check
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:52:in `block in ping_pulp_with_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:83:in `exception_watch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:48:in `ping_pulp_with_auth'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:22:in `ping'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/reimport.rake:7:in `block (2 levels) in <top (required)>'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
 | /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `<main>'
2018-06-01T21:15:02  [app] [I] Current user: foreman_api_admin (administrator)


Looks like
http://projects.theforeman.org/issues/23008
http://projects.theforeman.org/issues/21802

Tried to install https://yum.theforeman.org/plugins/nightly/el7/x86_64/tfm-rubygem-foreman-tasks-0.13.1-1.fm1_18.el7.noarch.rpm but then more&more deps needed from nightly.

Any idea how to solve it?

Regards,
Stefan

Re-posting the error with better formatting as it is very difficult to read in plaintext:

Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require' rake aborted! RestClient::InternalServerError: 500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in` exception_with_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in` block in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in` block in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in` execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in` get’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in` call’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:31:in `retrieve'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:126:in` repo_exists?’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:61:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in` block (2 levels) in find_each’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in` block in find_each’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:230:in` block in in_batches’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in` in_batches’
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:62:in` find_each’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:59:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/repository.rake:59:in` block (2 levels) in &lt;top (required)&gt;’
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `&lt;top (required)&gt;’
Tasks: TOP =&gt; katello:correct_repositories
(See full trace by running task with --trace)
Processing Repository 1/108: base_x86_64 (1)
foreman-rake katello:correct_repositories COMMIT=true failed! Check the output for error!
Upgrade step correct_repositories failed. Check logs for more information.

/var/log/foreman/production.log
2018-06-01T21:15:02 [app] [I] Current user: foreman_admin (administrator)
2018-06-01T21:15:02 [app] [W] 500 Internal Server Error
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in` return!’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:809:in `process_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in` block in transmit’
| /opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in` transmit’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in` execute’
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:150:in` backend_status’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:114:in `pulp_without_auth'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:36:in` block in ping_pulp_without_auth’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:83:in `exception_watch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:35:in` ping_pulp_without_auth’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:19:in `ping'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/reimport.rake:7:in` block (2 levels) in &lt;top (required)&gt;’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in` block in invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in` invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in` block in invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in` invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in` invoke_task’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in` run_with_threads’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in` block in run’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in` run’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
| /opt/rh/rh-ruby24/root/usr/bin/rake:23:in` load’
| /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `<main>' 2018-06-01T21:15:02 [app] [W] Skipped pulp_auth check after failed pulp check
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:52:in` block in ping_pulp_with_auth’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:83:in `exception_watch'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:48:in` ping_pulp_with_auth’
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/ping.rb:22:in `ping'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/lib/katello/tasks/reimport.rake:7:in` block (2 levels) in &lt;top (required)&gt;’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in` block in invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in` invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:216:in `block in invoke_prerequisites'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:214:in `invoke_prerequisites'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:193:in` block in invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in` invoke_with_call_chain’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in` invoke_task’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in` each’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in` run_with_threads’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in` block in run’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in` run’
| /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
| /opt/rh/rh-ruby24/root/usr/bin/rake:23:in` load’
| /opt/rh/rh-ruby24/root/usr/bin/rake:23:in `’
2018-06-01T21:15:02 [app] [I] Current user: foreman_api_admin (administrator)

It does not look like this is related to foreman-tasks to me, at least not directly. The error is caused by 500 response originating from katello repository code. I can also see the failed pulp check, it might be worthwhile looking into the pulp status. I am sure someone from @katello will have more suggestions.

Solved;
https://projects.theforeman.org/issues/23805
https://projects.theforeman.org/issues/22768