Theforeman 1.10 -> 1.11 upgrade (with katello 2.4 -> 3.0) fails at db:seed

Cristobal_Palmer · February 24, 2017, 4:12pm

Greetings,

I've got a rather old install that I'm trying to step forward one release
at a time from 1.10.x to current. I got stuck at the first upgrade, sadly.

I was following
this: Foreman :: Manual

The package updates installed fine (with a gotcha! – the first time I
failed out because I had repositories for katello 2.4 configured.
Configuring the 3.0 repos and then retrying the 1.10 -> 1.11 package
updates worked… but there was an additional gotcha on the gotcha: I ALSO
tried upgrading katello from 2.4 to 3.0 following
Foreman :: Plugin Manuals before
upgrading foreman… and the first time that failed at the yum update step
saying that the installed foreman was too old. Turns out that "yum upgrade
tfm* ruby* foreman*" step on the foreman instructions was important!).

Anyway, I finally got to where this runs mostly without error:
foreman-installer --scenario katello --upgrade

The errors:

Upgrade Step: Running installer…

/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]:
Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead
of one of [0]

/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]:
/usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
Installing Done [100%]
[…]
Something went wrong! Check the log for ERROR-level output
The full log is at /var/log/foreman-installer/katello.log

Rerunning the failing step gives
me: https://gist.github.com/anonymous/202564d5de17d948ec0772c6d6a446aa

Suggestions most welcome.

···

-- CMP

Cristobal_Palmer · February 24, 2017, 6:49pm

Please find the full katello.log from the upgrade attached.

Pointers to debugging resources I might use also welcome.

katello.log (16.5 KB)

···

On Friday, February 24, 2017 at 11:13:35 AM UTC-5, Cristóbal Palmer wrote: > > > > Rerunning the failing step gives me: > https://gist.github.com/anonymous/202564d5de17d948ec0772c6d6a446aa > >

Jim_Miller · March 26, 2017, 3:06am

Did you make any progress with this? I'm running to the same issue?

Thanks,
Jim

···

On Friday, February 24, 2017 at 10:13:35 AM UTC-6, Cristóbal Palmer wrote: > > Greetings, > > I've got a rather old install that I'm trying to step forward one release > at a time from 1.10.x to current. I got stuck at the first upgrade, sadly. > > I was following this: > https://theforeman.org/manuals/1.11/index.html#3.6Upgrade > > The package updates installed fine (with a gotcha! -- the first time I > failed out because I had repositories for katello 2.4 configured. > Configuring the 3.0 repos and then retrying the 1.10 -> 1.11 package > updates worked... but there was an additional gotcha on the gotcha: I ALSO > tried upgrading katello from 2.4 to 3.0 following > https://theforeman.org/plugins/katello/3.0/upgrade/index.html before > upgrading foreman... and the first time that failed at the yum update step > saying that the installed foreman was too old. Turns out that "yum upgrade > tfm\* ruby\* foreman\*" step on the foreman instructions was important!). > > Anyway, I finally got to where this runs mostly without error: > foreman-installer --scenario katello --upgrade > > The errors: > > Upgrade Step: Running installer... > > /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: > Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead > of one of [0] > > /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: > /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] > Installing Done [100%] > [.......................................................................................................................................................................................................] > Something went wrong! Check the log for ERROR-level output > The full log is at /var/log/foreman-installer/katello.log > > Rerunning the failing step gives me: > https://gist.github.com/anonymous/202564d5de17d948ec0772c6d6a446aa > > Suggestions most welcome. > > -- > CMP >

Mario_Gamboa · March 30, 2017, 9:30am

I have same issue and my solution was upgrade from 2.4 to 3.1 and the
issues gone i don't know if possible works for you

···

On Friday, February 24, 2017 at 5:13:35 PM UTC+1, Cristóbal Palmer wrote: > > Greetings, > > I've got a rather old install that I'm trying to step forward one release > at a time from 1.10.x to current. I got stuck at the first upgrade, sadly. > > I was following this: > https://theforeman.org/manuals/1.11/index.html#3.6Upgrade > > The package updates installed fine (with a gotcha! -- the first time I > failed out because I had repositories for katello 2.4 configured. > Configuring the 3.0 repos and then retrying the 1.10 -> 1.11 package > updates worked... but there was an additional gotcha on the gotcha: I ALSO > tried upgrading katello from 2.4 to 3.0 following > https://theforeman.org/plugins/katello/3.0/upgrade/index.html before > upgrading foreman... and the first time that failed at the yum update step > saying that the installed foreman was too old. Turns out that "yum upgrade > tfm\* ruby\* foreman\*" step on the foreman instructions was important!). > > Anyway, I finally got to where this runs mostly without error: > foreman-installer --scenario katello --upgrade > > The errors: > > Upgrade Step: Running installer... > > /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: > Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead > of one of [0] > > /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: > /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0] > Installing Done [100%] > [.......................................................................................................................................................................................................] > Something went wrong! Check the log for ERROR-level output > The full log is at /var/log/foreman-installer/katello.log > > Rerunning the failing step gives me: > https://gist.github.com/anonymous/202564d5de17d948ec0772c6d6a446aa > > Suggestions most welcome. > > -- > CMP >

Cristobal_Palmer · February 28, 2017, 8:28pm

Pointers to debugging resources I might use also welcome.

I've been chasing errors without much success. The only new clue I have
since last week is that pulp_streamer won't start:
https://gist.github.com/anonymous/c272193901b861e51c1788309c4e397e

Plugging various parts of the log into a popular search engine leads me to
various closed errata that are related to selinux (disabled on this host)
or otherwise not applicable.

I did this:
grep -i pulp_stream /var/log/messages > pulp_streamer.log

And from that I get:

https://gist.github.com/anonymous/e39304026360222ecd20f8a82ce066a9

Pasting same here in the body of this message:

-----8< snip 8<-----

Feb 28 15:19:25 foreman1 pulp_streamer: Executing: /usr/bin/pulp_streamer
–nodaemon --syslog --prefix=pulp_streamer --pidfile= --python
/usr/share/pulp/wsgi/streamer.tac

Feb 28 15:19:25 foreman1 pulp_streamer: Unhandled Error

Feb 28 15:19:25 foreman1 pulp_streamer: Traceback (most recent call last):

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 652,
in run

Feb 28 15:19:25 foreman1 pulp_streamer: runApp(config)

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/scripts/twistd.py", line 23, in
runApp

Feb 28 15:19:25 foreman1 pulp_streamer:
_SomeApplicationRunner(config).run()

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 386,
in run

Feb 28 15:19:25 foreman1 pulp_streamer: self.application =
self.createOrGetApplication()

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 451,
in createOrGetApplication

Feb 28 15:19:25 foreman1 pulp_streamer: application =
getApplication(self.config, passphrase)

Feb 28 15:19:25 foreman1 pulp_streamer: — <exception caught here> —

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/application/app.py", line 462,
in getApplication

Feb 28 15:19:25 foreman1 pulp_streamer: application =
service.loadApplication(filename, style, passphrase)

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/application/service.py", line
405, in loadApplication

Feb 28 15:19:25 foreman1 pulp_streamer: application =
sob.loadValueFromFile(filename, 'application', passphrase)

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib64/python2.7/site-packages/twisted/persisted/sob.py", line 210, in
loadValueFromFile

Feb 28 15:19:25 foreman1 pulp_streamer: exec fileObj in d, d

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/share/pulp/wsgi/streamer.tac", line 13, in <module>

Feb 28 15:19:25 foreman1 pulp_streamer: from pulp.streamer import Streamer,
load_configuration, DEFAULT_CONFIG_FILES

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib/python2.7/site-packages/pulp/streamer/init.py", line 2, in
<module>

Feb 28 15:19:25 foreman1 pulp_streamer: from pulp.streamer.server import
Streamer, Responder, StreamerListener # noqa

Feb 28 15:19:25 foreman1 pulp_streamer: File
"/usr/lib/python2.7/site-packages/pulp/streamer/server.py", line 6, in
<module>

Feb 28 15:19:25 foreman1 pulp_streamer: from mongoengine import
DoesNotExist, NotUniqueError

Feb 28 15:19:25 foreman1 pulp_streamer: exceptions.ImportError: cannot
import name DoesNotExist

Feb 28 15:19:25 foreman1 pulp_streamer: Failed to load application: cannot
import name DoesNotExist

Feb 28 15:19:25 foreman1 systemd[1]: pulp_streamer.service: main process
exited, code=exited, status=1/FAILURE

Feb 28 15:19:25 foreman1 systemd[1]: Unit pulp_streamer.service entered
failed state.

Feb 28 15:19:25 foreman1 systemd[1]: pulp_streamer.service failed.

-----8< snip 8<-----

Any/all help most welcome

···

On Friday, February 24, 2017 at 1:49:43 PM UTC-5, Cristóbal Palmer wrote: