katello-nightly-rpm-pipeline 299 failed

[jenkins]

Katello nightly pipeline failed:

https://ci.theforeman.org/job/katello-nightly-rpm-pipeline/299/

foreman-katello-nightly-test (passed)
foreman-katello-upgrade-nightly-test (failed)

[evgeni]

the upgrade failed with

[ INFO 2019-12-10T04:23:04 verbose]  Class[Candlepin::Service]: Scheduling refresh of Service[tomcat]
[ WARN 2019-12-10T04:23:05 verbose]  /Stage[main]/Candlepin::Service/Service[tomcat]/ensure: ensure changed 'stopped' to 'running'
[ INFO 2019-12-10T04:23:05 verbose]  /Service[tomcat]: Unscheduling refresh on Service[tomcat]
[ INFO 2019-12-10T04:23:05 verbose]  Foreman::Rake[db:seed]: Scheduling refresh of Exec[foreman-rake-db:seed]
[ WARN 2019-12-10T04:23:51 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
[ WARN 2019-12-10T04:23:51 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: rake aborted!
[ WARN 2019-12-10T04:23:51 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: There was an issue with the backend service candlepin: 404 Not Found
[ WARN 2019-12-10T04:23:51 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]/returns: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.15.0.pre.master/app/lib/actions/middleware/backend_services_check.rb:17:in `block in plan'

My guess would be Candlepin could not re-connect to the new DB, but I didn’t look further.

[tbrisker]

@ehelms might be related to the pg upgrade?

[ehelms]

I investigated this and it’s an unexpected failure in the postgres upgrade that I am working on. Essentially, if the languages of the 9.2 database and 10 database don’t match they will fail. Now why in our automation the PostgreSQL 10 database ends up being created with a different language I don’t know but it’s a worthwhile catch by automation that I should have a fix for today.

[ehelms]

This is targeted at fixing the nightly upgrade breakage:

Always set locale when upgrading postgres