Upgrade Katello 3.5.0.1 to 3.6 RC1 fails

CapitanBlack · March 24, 2018, 1:07am

Problem:
FGollowing - Foreman :: Plugin Manuals
Upgrade fails at yum update command (after yum clean all )

Expected outcome: Upgrade succesfull

Foreman and Proxy versions: 3.5 / 1.16.0

Foreman and Proxy plugin versions: 3.5 / 1.16.0

Other relevant data:

--> Finished Dependency Resolution
Error: Package: tfm-rubygem-responders-2.4.0-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionpack) < 5.3
Error: Package: tfm-runtime-4.0-1.el7.x86_64 (foreman)
           Requires: tfm-ror51-runtime
Error: Package: tfm-rubygem-responders-2.4.0-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) < 5.3
Error: Package: tfm-rubygem-roadie-rails-1.2.1-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) < 5.2
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rails) >= 5.1.4
Error: Package: tfm-rubygem-anemone-0.7.2-15.el7.noarch (katello)
           Requires: tfm-ror51-rubygem(nokogiri) >= 1.3.0
Error: Package: tfm-rubygem-roadie-3.2.2-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) >= 1.5.0
Error: Package: tfm-rubygem-dynflow-0.8.34-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(concurrent-ruby) >= 1.0
Error: Package: tfm-rubygem-fog-xml-0.1.2-5.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) < 2.0
Error: Package: tfm-rubygem-rails-i18n-5.0.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) >= 5.0
Error: Package: 1:tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(railties) >= 4.2
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionpack) >= 4.0
Error: Package: tfm-rubygem-ldap_fluff-0.4.7-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activesupport)
Error: Package: tfm-rubygem-deep_cloneable-2.2.2-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) < 5.2.0
Error: Package: 1:tfm-rubygem-concurrent-ruby-edge-0.2.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(concurrent-ruby) < 1.1.0
Error: Package: tfm-rubygem-rest-client-2.0.1-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(mime-types) >= 1.16
Error: Package: tfm-rubygem-dynflow-0.8.34-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(concurrent-ruby) < 2.0
Error: Package: python-gofer-qpid-2.7.6-1.el7.noarch (@katello-client)
           Requires: python-gofer = 2.7.6
           Removing: python-gofer-2.7.6-1.el7.noarch (@katello-client)
               python-gofer = 2.7.6-1.el7
           Updated By: python-gofer-2.7.7-3.el7sat.noarch (rhel-7-server-rhv-4-mgmt-agent-rpms)
               python-gofer = 2.7.7-3.el7sat
           Available: python-gofer-1.3.0-1.el7sat.noarch (rhel-7-server-rh-common-rpms)
               python-gofer = 1.3.0-1.el7sat
           Available: python-gofer-2.7.7-2.el7sat.noarch (rhel-7-server-rhv-4-mgmt-agent-rpms)
               python-gofer = 2.7.7-2.el7sat
Error: Package: tfm-rubygem-rack-jsonp-1.3.1-6.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rack)
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 4.0
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(mail) >= 2.7
Error: Package: tfm-rubygem-dynflow-0.8.34-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(multi_json)
Error: Package: tfm-rubygem-friendly_id-5.1.0-4.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 4.0.0
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) >= 4.0
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(mail) < 3.0
Error: Package: tfm-rubygem-audited-4.4.1-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) < 5.2
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(turbolinks) < 3.0
Error: Package: tfm-rubygem-scoped_search-4.1.2-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 4.2.0
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(turbolinks) >= 2.5.4
Error: Package: tfm-rubygem-runcible-2.6.0-2.el7.noarch (katello)
           Requires: tfm-ror51-rubygem(activesupport) >= 3.0.10
Error: Package: tfm-rubygem-passenger-4.0.18-9.12.el7.x86_64 (foreman)
           Requires: tfm-ror51-rubygem(rack)
Error: Package: tfm-rubygem-fog-xml-0.1.2-5.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) >= 1.5.11
Error: Package: tfm-rubygem-responders-2.4.0-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) >= 4.2.0
Error: Package: tfm-rubygem-roadie-3.2.2-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) < 2.0.0
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(sprockets) >= 3
Error: Package: tfm-rubygem-deface-1.2.0-8.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(nokogiri) < 2.0.0
Error: Package: tfm-rubygem-audited-4.4.1-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 4.0
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionpack) < 5.2
Error: Package: tfm-rubygem-rbvmomi-1.10.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(builder) < 4.0
Error: Package: tfm-rubygem-fog-core-1.45.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem-builder
Error: Package: tfm-rubygem-deep_cloneable-2.2.2-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 3.1.0
Error: Package: tfm-rubygem-rbovirt-0.1.4-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem-nokogiri
Error: Package: tfm-rubygem-ancestry-3.0.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 3.2.2
Error: Package: tfm-rubygem-rest-client-2.0.1-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(mime-types) < 4.0
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) < 5.2
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rack) >= 1.5.2
Error: Package: tfm-rubygem-deface-1.2.0-8.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(rails) >= 4.1
Error: Package: tfm-rubygem-sprockets-rails-3.2.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(sprockets) >= 3.0.0
Error: Package: tfm-rubygem-x-editable-rails-1.5.5-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties)
Error: Package: tfm-rubygem-foreman-tasks-0.11.2-1.fm1_17.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(sinatra)
Error: Package: tfm-rubygem-roadie-rails-1.2.1-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) >= 3.0
Error: Package: tfm-rubygem-rbvmomi-1.10.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) < 2.0
Error: Package: tfm-rubygem-webpack-rails-0.9.8-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rails) >= 3.2.0
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rack) < 3
Error: Package: tfm-rubygem-rabl-0.13.1-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activesupport) >= 2.3.14
Error: Package: 1:tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(tilt)
Error: Package: 1:tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(sprockets) < 5
Error: Package: tfm-rubygem-deface-1.2.0-8.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(nokogiri) >= 1.6.0
Error: Package: tfm-rubygem-rbvmomi-1.10.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(nokogiri) >= 1.5
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rails) < 5.2
Error: Package: tfm-rubygem-sprockets-rails-3.2.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionpack) >= 4.0
Error: Package: tfm-rubygem-responders-2.4.0-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionpack) >= 4.2.0
Error: Package: tfm-rubygem-fog-json-1.0.2-5.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(multi_json) >= 1.10
Error: Package: tfm-rubygem-sprockets-rails-3.2.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activesupport) >= 4.0
Error: Package: tfm-rubygem-rails-i18n-5.0.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(i18n) < 1
Error: Package: tfm-rubygem-record_tag_helper-1.0.0-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(actionview)
Error: Package: tfm-rubygem-fog-json-1.0.2-5.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(multi_json) < 2
Error: Package: tfm-rubygem-will_paginate-3.1.5-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord)
Error: Package: foreman-1.17.0-0.3.RC2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(sprockets) < 4
Error: Package: tfm-rubygem-rbvmomi-1.10.0-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(builder) >= 3.0
Error: Package: tfm-rubygem-apipie-rails-0.5.6-2.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(rails) >= 4.1
Error: Package: 1:tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(railties) < 6
Error: Package: 1:tfm-rubygem-concurrent-ruby-edge-0.2.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(concurrent-ruby) >= 1.0.3
Error: Package: 1:tfm-rubygem-angular-rails-templates-1.0.2-2.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(sprockets) >= 3.0
Error: Package: tfm-rubygem-rails-i18n-5.0.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(railties) < 6.0
Error: Package: tfm-rubygem-runcible-2.6.0-2.el7.noarch (katello)
           Requires: tfm-ror51-rubygem(i18n) >= 0.5.0
Error: Package: tfm-rubygem-activerecord-session_store-1.1.0-3.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) < 5.2
Error: Package: tfm-rubygem-wicked-1.3.2-1.el7.noarch (foreman-plugins)
           Requires: tfm-ror51-rubygem(railties) >= 3.0.7
Error: Package: tfm-rubygem-rails-i18n-5.0.4-1.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(i18n) >= 0.7
Error: Package: tfm-rubygem-logging-1.8.2-5.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(multi_json) >= 1.8.4
Error: Package: tfm-rubygem-validates_lengths_from_database-0.5.0-4.el7.noarch (foreman)
           Requires: tfm-ror51-rubygem(activerecord) >= 2.3.2

[skipped] 

 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

fdisk101 · March 24, 2018, 5:16pm

im a newbie but it looks like 1 or 2 repos may be missing have a look at Foreman :: Plugin Manuals
it mentions all the repos you need for a install?
all the best

CapitanBlack · March 25, 2018, 6:46pm

Fixed all dependencies issues - had to run again

yum -y install foreman-release-scl python-django

This was conflicting with katello-client repo
subscription-manager repos --disable=rhel-7-server-rhv-4-mgmt-agent-rpms

CapitanBlack · March 25, 2018, 7:08pm

Upgrade went well except of this last step… Any suggestions? Is that a critical step?

Upgrade Step: correct_repositories (this may take a while) ...
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/appender.rb:139: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/logger.rb:295: warning: constant ::Fixnum is deprecated
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.

You should use PG::Connection, PG::Result, and PG::Error instead, respectively.

Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
rake aborted!
RestClient::InternalServerError: 500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:31:in `retrieve'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:126:in `repo_exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:61:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:62:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_repositories
(See full trace by running task with --trace)
Processing Repository 1/6: EPEL7 (1)
foreman-rake katello:correct_repositories COMMIT=true failed! Check the output for error!
Upgrade step correct_repositories failed. Check logs for more information.

ekohl · March 26, 2018, 9:16am

We shouldn’t have any failures. Can you show the output in /var/log/foreman/production.log? I think that should have more info about the failed rake job.

CapitanBlack · March 26, 2018, 5:08pm

Relevant lines from /var/log/foreman-installer/katello.log

[DEBUG 2018-03-25 10:02:18 main] foreman-rake db:seed finished successfully!
[ INFO 2018-03-25 10:02:18 main] Upgrade Step: correct_repositories (this may take a while) ...
[DEBUG 2018-03-25 10:02:18 main] /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
[DEBUG 2018-03-25 10:02:18 main] /opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
[DEBUG 2018-03-25 10:02:20 main] /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
[DEBUG 2018-03-25 10:02:20 main] /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[DEBUG 2018-03-25 10:02:20 main] /usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
[DEBUG 2018-03-25 10:02:20 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
[DEBUG 2018-03-25 10:02:20 main] /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/appender.rb:139: warning: constant ::Fixnum is deprecated
[DEBUG 2018-03-25 10:02:20 main] /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/logger.rb:295: warning: constant ::Fixnum is deprecated
[DEBUG 2018-03-25 10:02:21 main] API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[DEBUG 2018-03-25 10:02:26 main] The PGconn, PGresult, and PGError constants are deprecated, and will be
[DEBUG 2018-03-25 10:02:26 main] removed as of version 1.0.
[DEBUG 2018-03-25 10:02:26 main]
[DEBUG 2018-03-25 10:02:26 main] You should use PG::Connection, PG::Result, and PG::Error instead, respectively.
[DEBUG 2018-03-25 10:02:26 main]
[DEBUG 2018-03-25 10:02:26 main] Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
[DEBUG 2018-03-25 10:02:27 main] rake aborted!
[DEBUG 2018-03-25 10:02:27 main] RestClient::InternalServerError: 500 Internal Server Error
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in `block in get_response'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:31:in `retrieve'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:126:in `repo_exists?'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:61:in `block (3 levels) in <top (required)>'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `each'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block in find_each'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:230:in `block in in_batches'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `loop'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `in_batches'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:128:in `find_in_batches'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:62:in `find_each'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `with_index'
[DEBUG 2018-03-25 10:02:27 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `block (2 levels) in <top (required)>'
[DEBUG 2018-03-25 10:02:27 main] /opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
[DEBUG 2018-03-25 10:02:27 main] Tasks: TOP => katello:correct_repositories
[DEBUG 2018-03-25 10:02:27 main] (See full trace by running task with --trace)
[DEBUG 2018-03-25 10:02:27 main] Processing Repository 1/6: EPEL7 (1)
[ERROR 2018-03-25 10:02:27 main] foreman-rake katello:correct_repositories COMMIT=true failed! Check the output for error!
[ERROR 2018-03-25 10:02:27 main] Upgrade step correct_repositories failed. Check logs for more information.
[DEBUG 2018-03-25 10:02:27 main] Exit with status code: 1 (signal was 1)
[ERROR 2018-03-25 10:02:27 main] Errors encountered during run:
[ERROR 2018-03-25 10:02:27 main] foreman-rake katello:correct_repositories COMMIT=true failed! Check the output for error!
[ERROR 2018-03-25 10:02:27 main] Upgrade step correct_repositories failed. Check logs for more information.
[DEBUG 2018-03-25 10:02:27 main] Cleaning /tmp/kafo_puppet20180325-13120-ndtzrg.conf
[DEBUG 2018-03-25 10:02:27 main] Cleaning /tmp/kafo_hiera20180325-13120-15oagni
[DEBUG 2018-03-25 10:02:27 main] Cleaning /tmp/kafo_puppet20180325-13120-1r9bp1e.conf
[DEBUG 2018-03-25 10:02:27 main] Cleaning /tmp/default_values.yaml
[ INFO 2018-03-25 10:02:27 main] Installer finished in 305.108201961 seconds

CapitanBlack · March 26, 2018, 5:10pm

… and /var/log/foreman/production.log … Look for “500 Internal Server Error” at 10:02:26

/var/log/foreman/production.log-20180326:2018-03-25 10:02:07  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:07  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:08  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:10  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:10  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:11  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Enqueued CreatePulpDiskSpaceNotifications (Job ID: 717a8c3d-033f-441c-942e-806064a4f7c2) to Dynflow(default)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/v1ar/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Performing CreatePulpDiskSpaceNotifications (Job ID: 846b9c0a-d7f6-489a-b86c-3f3259c3477a) from Dynflow(default)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:14  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Enqueued CreatePulpDiskSpaceNotifications (Job ID: 5b4270c2-7777-4756-8946-bcfe40773cc9) to Dynflow(default) at 2018-03-26 02:02:14 UTC
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Performed CreatePulpDiskSpaceNotifications (Job ID: 846b9c0a-d7f6-489a-b86c-3f3259c3477a) from Dynflow(default) in 330.86ms
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:15  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:16  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [background] [I] Finishing Actions::Candlepin::ListenOnCandlepinEvents c41c0713-b548-42ee-afdf-14bad20507be
/var/log/foreman/production.log-20180326:2018-03-25 10:02:17  [background] [I] Finishing Actions::Candlepin::ListenOnCandlepinEvents c41c0713-b548-42ee-afdf-14bad20507be: attempt 1/10
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating delayed_executor...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating throttle_limiter...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating executor...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] shutting down Dynflow Core ...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [E] ... Dynflow Core has shut down.
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating executor dispatcher...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating client dispatcher...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] stop listening for new events...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:18  [dynflow] [I] start terminating clock...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:22  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:22  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:23  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:26  [app] [I] Current user: foreman_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:26  [app] [W] 500 Internal Server Error
/var/log/foreman/production.log-20180326:2018-03-25 10:02:26  [app] [W] Skipped pulp_auth check after failed pulp check
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [app] [I] Current user: foreman_api_admin (administrator)
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [katello/pulp_rest] [E] RestClient.get "https://inf-foreman-01.mtl.bios.run/pulp/api/v2/repositories/ddfe01b8-9180-4d7e-9835-aebc25f59bf4/?", "Accept"=>"*/*", "Accept-Encoding"=>"gzip, deflate", "User-Agent"=>"rest-client/2.0.1 (linux-gnu x86_64) ruby/2.4.2p198", "accept"=>"application/json", "content_type"=>"application/json"
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [dynflow] [I] start terminating throttle_limiter...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [dynflow] [I] start terminating client dispatcher...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [dynflow] [I] stop listening for new events...
/var/log/foreman/production.log-20180326:2018-03-25 10:02:27  [dynflow] [I] start terminating clock...

CapitanBlack · March 26, 2018, 5:16pm

root@inf-foreman-01:~ # foreman-rake katello:correct_repositories COMMIT=true --trace

/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Fixnum is deprecated
/opt/theforeman/tfm/root/usr/share/gems/gems/sequel-4.20.0/lib/sequel/database/schema_generator.rb:19: warning: constant ::Bignum is deprecated
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
/usr/share/foreman/lib/core_extensions.rb:183: warning: already initialized constant ActiveSupport::MessageEncryptor::DEFAULT_CIPHER
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/message_encryptor.rb:22: warning: previous definition of DEFAULT_CIPHER was here
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/appender.rb:139: warning: constant ::Fixnum is deprecated
** Invoke katello:correct_repositories (first_time)
** Invoke environment (first_time)
** Execute environment
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/logger.rb:295: warning: constant ::Fixnum is deprecated
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
The PGconn, PGresult, and PGError constants are deprecated, and will be
removed as of version 1.0.

You should use PG::Connection, PG::Result, and PG::Error instead, respectively.

Called from /opt/theforeman/tfm/root/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
** Invoke katello:check_ping (first_time)
** Invoke environment
** Execute katello:check_ping
** Execute katello:correct_repositories
Processing Repository 1/6: EPEL7 (1)
rake aborted!
RestClient::InternalServerError: 500 Internal Server Error
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in `block in get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
/opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:31:in `retrieve'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:126:in `repo_exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:61:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block (2 levels) in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:63:in `block in find_each'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:129:in `block in find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:230:in `block in in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `loop'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:214:in `in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:128:in `find_in_batches'
/opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/relation/batches.rb:62:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0.1.rc2/lib/katello/tasks/repository.rake:59:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `block in execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:250:in `execute'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:187:in `invoke_with_call_chain'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/task.rb:180:in `invoke'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:152:in `invoke_task'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block (2 levels) in top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `each'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:108:in `block in top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:117:in `run_with_threads'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:102:in `top_level'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:80:in `block in run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:178:in `standard_exception_handling'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/lib/rake/application.rb:77:in `run'
/opt/rh/rh-ruby24/root/usr/share/gems/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby24/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby24/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => katello:correct_repositories

ekohl · March 26, 2018, 5:20pm

Given the traceback is in runcible that makes me think the 500 error is in pulp rather than foreman. I’m not sure if that’s logged to /var/log/pulp or /var/log/httpd

CapitanBlack · March 26, 2018, 6:37pm

Yes, Pulp is broken after upgrade I like the new release and don’t want roll back to pre-upgrade snapshot - can someone help me to fix this?

ekohl · March 26, 2018, 6:58pm

Logs in the aforementioned locations (/var/log/httpd and /var/log/pulp) are probably more useful. We need specific errors rather than a generic Internal Server Error.

CapitanBlack · March 26, 2018, 6:59pm

Yes, thanks @ekohl - I see some error messages from upgrade date&time in httpd logs. Will post them in a few minutes…

CapitanBlack · March 26, 2018, 7:32pm

I’m not sure why times inside of log file are all messed up - sorry not my fault. Relevant lines from
/var/log/httpd/error_log

error_log:[Sun Mar 25 09:53:17.142835 2018] [mime_magic:error] [pid 988] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 09:53:17.156405 2018] [suexec:notice] [pid 988] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
error_log:[Sun Mar 25 09:53:17.353042 2018] [mime_magic:error] [pid 988] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 09:53:17.353371 2018] [auth_digest:notice] [pid 988] AH01757: generating secret for digest authentication ...
error_log:[Sun Mar 25 09:53:17.499736 2018] [mpm_prefork:notice] [pid 988] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Phusion_Passenger/4.0.53 configured -- resuming normal operations
error_log:[Sun Mar 25 09:53:17.499758 2018] [core:notice] [pid 988] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
error_log:[Sun Mar 25 13:53:20.233517 2018] [:error] [pid 1235] mod_wsgi (pid=1235): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:53:20.233552 2018] [:error] [pid 1235] mod_wsgi (pid=1235): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:53:20.233567 2018] [:error] [pid 1235] Traceback (most recent call last):
error_log:[Sun Mar 25 13:53:20.233549 2018] [:error] [pid 1231] mod_wsgi (pid=1231): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:53:20.233576 2018] [:error] [pid 1235]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:53:20.233576 2018] [:error] [pid 1231] mod_wsgi (pid=1231): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:53:20.233593 2018] [:error] [pid 1231] Traceback (most recent call last):
error_log:[Sun Mar 25 13:53:20.233601 2018] [:error] [pid 1231]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:53:20.233619 2018] [:error] [pid 1235]     application = wsgi_application()
error_log:[Sun Mar 25 13:53:20.233627 2018] [:error] [pid 1235]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:53:20.233638 2018] [:error] [pid 1235]     raise e
error_log:[Sun Mar 25 13:53:20.233641 2018] [:error] [pid 1231]     application = wsgi_application()
error_log:[Sun Mar 25 13:53:20.233657 2018] [:error] [pid 1231]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:53:20.233657 2018] [:error] [pid 1235] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 13:53:20.233670 2018] [:error] [pid 1231]     raise e
error_log:[Sun Mar 25 13:53:20.233678 2018] [:error] [pid 1231] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 13:53:20.278468 2018] [:error] [pid 1228] mod_wsgi (pid=1228): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:53:20.278503 2018] [:error] [pid 1228] mod_wsgi (pid=1228): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:53:20.278519 2018] [:error] [pid 1228] Traceback (most recent call last):
error_log:[Sun Mar 25 13:53:20.278528 2018] [:error] [pid 1228]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:53:20.278571 2018] [:error] [pid 1228]     application = wsgi_application()
error_log:[Sun Mar 25 13:53:20.278595 2018] [:error] [pid 1228]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:53:20.278609 2018] [:error] [pid 1228]     raise e
error_log:[Sun Mar 25 13:53:20.278618 2018] [:error] [pid 1228] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 09:56:50.600061 2018] [mpm_prefork:notice] [pid 988] AH00170: caught SIGWINCH, shutting down gracefully
error_log:[Sun Mar 25 09:59:00.381296 2018] [mime_magic:error] [pid 16934] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 09:59:00.383224 2018] [suexec:notice] [pid 16934] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
error_log:[Sun Mar 25 09:59:00.413613 2018] [mime_magic:error] [pid 16934] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 09:59:00.413626 2018] [auth_digest:notice] [pid 16934] AH01757: generating secret for digest authentication ...
error_log:[Sun Mar 25 09:59:00.517113 2018] [mpm_prefork:notice] [pid 16934] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Phusion_Passenger/4.0.53 configured -- resuming normal operations
error_log:[Sun Mar 25 09:59:00.517144 2018] [core:notice] [pid 16934] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
error_log:[Sun Mar 25 13:59:01.556729 2018] [:error] [pid 16955] mod_wsgi (pid=16955): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:59:01.556765 2018] [:error] [pid 16955] mod_wsgi (pid=16955): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:59:01.556781 2018] [:error] [pid 16955] Traceback (most recent call last):
error_log:[Sun Mar 25 13:59:01.556789 2018] [:error] [pid 16955]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:59:01.556832 2018] [:error] [pid 16955]     application = wsgi_application()
error_log:[Sun Mar 25 13:59:01.556840 2018] [:error] [pid 16955]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:59:01.556851 2018] [:error] [pid 16955]     raise e
error_log:[Sun Mar 25 13:59:01.556859 2018] [:error] [pid 16955] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 13:59:01.586076 2018] [:error] [pid 16956] mod_wsgi (pid=16956): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:59:01.586110 2018] [:error] [pid 16956] mod_wsgi (pid=16956): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:59:01.586148 2018] [:error] [pid 16956] Traceback (most recent call last):
error_log:[Sun Mar 25 13:59:01.586160 2018] [:error] [pid 16956]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:59:01.586204 2018] [:error] [pid 16956]     application = wsgi_application()
error_log:[Sun Mar 25 13:59:01.586211 2018] [:error] [pid 16956]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:59:01.586222 2018] [:error] [pid 16956]     raise e
error_log:[Sun Mar 25 13:59:01.586231 2018] [:error] [pid 16956] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 13:59:01.698970 2018] [:error] [pid 16957] mod_wsgi (pid=16957): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 13:59:01.699013 2018] [:error] [pid 16957] mod_wsgi (pid=16957): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 13:59:01.699029 2018] [:error] [pid 16957] Traceback (most recent call last):
error_log:[Sun Mar 25 13:59:01.699038 2018] [:error] [pid 16957]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 13:59:01.699081 2018] [:error] [pid 16957]     application = wsgi_application()
error_log:[Sun Mar 25 13:59:01.699088 2018] [:error] [pid 16957]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 13:59:01.699099 2018] [:error] [pid 16957]     raise e
error_log:[Sun Mar 25 13:59:01.699108 2018] [:error] [pid 16957] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 10:00:59.110856 2018] [mpm_prefork:notice] [pid 16934] AH00170: caught SIGWINCH, shutting down gracefully
error_log:[Sun Mar 25 10:01:33.102265 2018] [mime_magic:error] [pid 18991] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 10:01:33.104221 2018] [suexec:notice] [pid 18991] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
error_log:[Sun Mar 25 10:01:33.144550 2018] [mime_magic:error] [pid 18991] (2)No such file or directory: AH01515: mod_mime_magic: can't read magic file /etc/httpd/NEVER_EVER_USE
error_log:[Sun Mar 25 10:01:33.144567 2018] [auth_digest:notice] [pid 18991] AH01757: generating secret for digest authentication ...
error_log:[Sun Mar 25 10:01:33.195658 2018] [mpm_prefork:notice] [pid 18991] AH00163: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.2k-fips mod_wsgi/3.4 Python/2.7.5 Phusion_Passenger/4.0.53 configured -- resuming normal operations
error_log:[Sun Mar 25 10:01:33.195679 2018] [core:notice] [pid 18991] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
error_log:[Sun Mar 25 14:01:34.570718 2018] [:error] [pid 19019] mod_wsgi (pid=19019): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 14:01:34.570757 2018] [:error] [pid 19019] mod_wsgi (pid=19019): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 14:01:34.570773 2018] [:error] [pid 19019] Traceback (most recent call last):
error_log:[Sun Mar 25 14:01:34.570759 2018] [:error] [pid 19018] mod_wsgi (pid=19018): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 14:01:34.570782 2018] [:error] [pid 19019]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 14:01:34.570787 2018] [:error] [pid 19018] mod_wsgi (pid=19018): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 14:01:34.570802 2018] [:error] [pid 19018] Traceback (most recent call last):
error_log:[Sun Mar 25 14:01:34.570811 2018] [:error] [pid 19018]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 14:01:34.570826 2018] [:error] [pid 19019]     application = wsgi_application()
error_log:[Sun Mar 25 14:01:34.570834 2018] [:error] [pid 19019]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 14:01:34.570847 2018] [:error] [pid 19019]     raise e
error_log:[Sun Mar 25 14:01:34.570851 2018] [:error] [pid 19018]     application = wsgi_application()
error_log:[Sun Mar 25 14:01:34.570856 2018] [:error] [pid 19019] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 14:01:34.570859 2018] [:error] [pid 19018]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 14:01:34.570871 2018] [:error] [pid 19018]     raise e
error_log:[Sun Mar 25 14:01:34.570880 2018] [:error] [pid 19018] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
error_log:[Sun Mar 25 14:01:34.749206 2018] [:error] [pid 19020] mod_wsgi (pid=19020): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
error_log:[Sun Mar 25 14:01:34.749249 2018] [:error] [pid 19020] mod_wsgi (pid=19020): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
error_log:[Sun Mar 25 14:01:34.749266 2018] [:error] [pid 19020] Traceback (most recent call last):
error_log:[Sun Mar 25 14:01:34.749275 2018] [:error] [pid 19020]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
error_log:[Sun Mar 25 14:01:34.749319 2018] [:error] [pid 19020]     application = wsgi_application()
error_log:[Sun Mar 25 14:01:34.749326 2018] [:error] [pid 19020]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
error_log:[Sun Mar 25 14:01:34.749338 2018] [:error] [pid 19020]     raise e
error_log:[Sun Mar 25 14:01:34.749347 2018] [:error] [pid 19020] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.

ekohl · March 26, 2018, 7:41pm

Normally we stop all pulp processes before migrating. I’d recommend stopping the pulp services, removing the marker file and then rerunning the installer:

systemctl stop pulp_celerybeat pulp_workers pulp_resource_manager pulp_streamer
rm /var/lib/pulp/init.flag
foreman-installer

CapitanBlack · March 26, 2018, 7:43pm

/var/log/httpd/foreman-ssl_error_ssl.log

foreman-ssl_error_ssl.log:[Sun Mar 25 10:02:26.962232 2018] [ssl:warn] [pid 19047] [client <IP-deleted>:34598] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972077 2018] [:error] [pid 19019] [remote <IP-deleted>:0] mod_wsgi (pid=19019): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972093 2018] [:error] [pid 19019] [remote <IP-deleted>:0] mod_wsgi (pid=19019): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972107 2018] [:error] [pid 19019] [remote <IP-deleted>:0] Traceback (most recent call last):
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972118 2018] [:error] [pid 19019] [remote <IP-deleted>:0]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972145 2018] [:error] [pid 19019] [remote <IP-deleted>:0]     application = wsgi_application()
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972155 2018] [:error] [pid 19019] [remote <IP-deleted>:0]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972165 2018] [:error] [pid 19019] [remote <IP-deleted>:0]     raise e
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:26.972175 2018] [:error] [pid 19019] [remote <IP-deleted>:0] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.088964 2018] [:error] [pid 19020] [remote <IP-deleted>:0] mod_wsgi (pid=19020): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.088999 2018] [:error] [pid 19020] [remote <IP-deleted>:0] mod_wsgi (pid=19020): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089012 2018] [:error] [pid 19020] [remote <IP-deleted>:0] Traceback (most recent call last):
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089031 2018] [:error] [pid 19020] [remote <IP-deleted>:0]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089049 2018] [:error] [pid 19020] [remote <IP-deleted>:0]     application = wsgi_application()
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089054 2018] [:error] [pid 19020] [remote <IP-deleted>:0]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089063 2018] [:error] [pid 19020] [remote <IP-deleted>:0]     raise e
foreman-ssl_error_ssl.log:[Sun Mar 25 14:02:27.089071 2018] [:error] [pid 19020] [remote <IP-deleted>:0] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
foreman-ssl_error_ssl.log:[Sun Mar 25 10:04:19.301063 2018] [ssl:warn] [pid 19045] [client <IP-deleted>:34678] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310597 2018] [:error] [pid 19020] [remote <IP-deleted>:88] mod_wsgi (pid=19020): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310613 2018] [:error] [pid 19020] [remote <IP-deleted>:88] mod_wsgi (pid=19020): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310626 2018] [:error] [pid 19020] [remote <IP-deleted>:88] Traceback (most recent call last):
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310636 2018] [:error] [pid 19020] [remote <IP-deleted>:88]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310653 2018] [:error] [pid 19020] [remote <IP-deleted>:88]     application = wsgi_application()
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310658 2018] [:error] [pid 19020] [remote <IP-deleted>:88]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310666 2018] [:error] [pid 19020] [remote <IP-deleted>:88]     raise e
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.310674 2018] [:error] [pid 19020] [remote <IP-deleted>:88] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444024 2018] [:error] [pid 19020] [remote <IP-deleted>:84] mod_wsgi (pid=19020): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444045 2018] [:error] [pid 19020] [remote <IP-deleted>:84] mod_wsgi (pid=19020): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444058 2018] [:error] [pid 19020] [remote <IP-deleted>:84] Traceback (most recent call last):
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444067 2018] [:error] [pid 19020] [remote <IP-deleted>:84]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444081 2018] [:error] [pid 19020] [remote <IP-deleted>:84]     application = wsgi_application()
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444086 2018] [:error] [pid 19020] [remote <IP-deleted>:84]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444095 2018] [:error] [pid 19020] [remote <IP-deleted>:84]     raise e
foreman-ssl_error_ssl.log:[Sun Mar 25 14:04:19.444104 2018] [:error] [pid 19020] [remote <IP-deleted>:84] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.409058 2018] [ssl:error] [pid 19043] [client <IP-deleted>:59214] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.412651 2018] [ssl:error] [pid 20176] [client <IP-deleted>:59215] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.412651 2018] [ssl:error] [pid 19046] [client <IP-deleted>:59217] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.414169 2018] [ssl:error] [pid 19048] [client <IP-deleted>:59218] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.574553 2018] [ssl:error] [pid 19044] [client <IP-deleted>:59208] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:05:51.647003 2018] [ssl:error] [pid 19047] [client <IP-deleted>:59216] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/subscriptions
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:26.959967 2018] [ssl:error] [pid 20175] [client <IP-deleted>:59245] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:27.333740 2018] [ssl:error] [pid 20174] [client <IP-deleted>:59249] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:27.473595 2018] [ssl:error] [pid 19045] [client <IP-deleted>:59250] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:30.541213 2018] [ssl:error] [pid 20176] [client <IP-deleted>:59257] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:30.562050 2018] [ssl:error] [pid 20174] [client <IP-deleted>:59260] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:30.679219 2018] [ssl:error] [pid 19049] [client <IP-deleted>:59258] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:30.817321 2018] [ssl:error] [pid 19046] [client <IP-deleted>:59259] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:06:30.953874 2018] [ssl:error] [pid 19045] [client <IP-deleted>:59261] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:10:53.982205 2018] [ssl:error] [pid 19043] [client <IP-deleted>:59376] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:10:53.992828 2018] [ssl:error] [pid 19046] [client <IP-deleted>:59377] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:10:54.123266 2018] [ssl:error] [pid 19050] [client <IP-deleted>:59378] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:12.088621 2018] [ssl:error] [pid 19049] [client <IP-deleted>:59392] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:12.093422 2018] [ssl:error] [pid 19043] [client <IP-deleted>:59393] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:39.585699 2018] [ssl:error] [pid 20176] [client <IP-deleted>:59402] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:39.861035 2018] [ssl:error] [pid 19049] [client <IP-deleted>:59406] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:45.801590 2018] [ssl:error] [pid 19050] [client <IP-deleted>:59407] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:46.068379 2018] [ssl:error] [pid 20174] [client <IP-deleted>:59411] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories?page=1&per_page=20
foreman-ssl_error_ssl.log:[Sun Mar 25 10:11:51.202915 2018] [ssl:warn] [pid 20175] [client <IP-deleted>:35020] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212560 2018] [:error] [pid 19020] [remote <IP-deleted>:340] mod_wsgi (pid=19020): Target WSGI script '/usr/share/pulp/wsgi/webservices.wsgi' cannot be loaded as Python module.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212574 2018] [:error] [pid 19020] [remote <IP-deleted>:340] mod_wsgi (pid=19020): Exception occurred processing WSGI script '/usr/share/pulp/wsgi/webservices.wsgi'.
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212604 2018] [:error] [pid 19020] [remote <IP-deleted>:340] Traceback (most recent call last):
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212612 2018] [:error] [pid 19020] [remote <IP-deleted>:340]   File "/usr/share/pulp/wsgi/webservices.wsgi", line 19, in <module>
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212626 2018] [:error] [pid 19020] [remote <IP-deleted>:340]     application = wsgi_application()
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212631 2018] [:error] [pid 19020] [remote <IP-deleted>:340]   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py", line 117, in wsgi_application
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212639 2018] [:error] [pid 19020] [remote <IP-deleted>:340]     raise e
foreman-ssl_error_ssl.log:[Sun Mar 25 14:11:51.212648 2018] [:error] [pid 19020] [remote <IP-deleted>:340] InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
foreman-ssl_error_ssl.log:[Sun Mar 25 10:12:34.260633 2018] [ssl:error] [pid 19049] [client <IP-deleted>:59430] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories
foreman-ssl_error_ssl.log:[Sun Mar 25 10:12:34.273111 2018] [ssl:error] [pid 19048] [client <IP-deleted>:59431] AH02261: Re-negotiation handshake failed, referer: https://<hostname-deleted>/products/4/repositories

fdisk101 · March 26, 2018, 8:04pm

Evgeny, did you stop services before the upgrade? if you can roll back and before trying the upgrade use “katello-service stop” to shut down all services before upgrade? (hope i got that right)

CapitanBlack · March 26, 2018, 8:12pm

Actually I did stop my services as said here - Foreman :: Plugin Manuals
Looks like you solution helped @ekohl
I am able to work with my repos again!

THANK YOU!

Lachlan_Musicman · March 26, 2018, 11:13pm

I’ve always been confused by this instruction. If I have a self hosted katello service, and I turn it off, where will my system get it’s updates from? There seems to be a chicken/egg problem here.

fdisk101 · March 27, 2018, 11:57am

Lachlan, does your host not still have the original repo files pointing to the tinternet for updates?

Lachlan_Musicman · March 27, 2018, 9:15pm

The files are there, but we have set enabled=0 - because we want our Katello server in a content view as well.