Katello, applying errata: "issue with the backend service pulp: 500 Internal Server Error"

Support
#1

I'm having trouble applying some Errata - after hitting confirm on applying
the Errata to the content hosts, I get an error message:

There was an issue with the backend service pulp: 500 Internal Server

Error

Centos 7.2, Katello 3.0 9.el7, Foreman 1.11.4

hammer ping tells me the pulp; Status is FAIL

systemctl says all pulp services are fine, except pulp-workers which exited
with SUCCESS.

In /var/log/foreman/production.log I see lots of output, but this is the
last dozen or so lines:

> /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/
methodoverride.rb:21:in call' > /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/runtime.rb:17:incall'
>
/opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/cache/strategy/local_cache_m
iddleware.rb:26:in call' > /opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/ action_dispatch/middleware/static.rb:64:inca
ll'
>
/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in
call' > /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/engine.rb:514:incall'
> /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/
application.rb:144:in call' > /opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:inpublic_send'
>
/opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in
method_missing' > /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/builder.rb: 138:incall'
>
/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in
block in call' > /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:50:ineach'
> /opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:
50:in call' > /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:inprocess_request'
>
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in
accept_and_process_next_requ est' > /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb: 109:inmain_loop'
> /usr/share/passenger/phusion_passenger/request_handler.rb:455:in block ( 3 levels) in start_threads' > /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:incall'
> /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/
diagnostic_context.rb:323:in `block in create_with_logging_context'
2016-09-22 14:01:25 [app] [I] Completed 500 Internal Server Error in 378ms
(Views: 0.2ms | ActiveRecord: 188.2ms)

How do I fix this problem?

#2

Ok, I found the solution, but I feel like I had to go the long way around
to find it.

When I did systemctl status httpd, I finally saw the error that was
meaningful:

pulp.server.webservices.application:ERROR: (10601-98880) The database has
not been migrated to the current version. Run pulp-manage-db and restart
the application.
pulp.server.webservices.application:ERROR: (10601-98880) Traceback (most
recent call last):
pulp.server.webservices.application:ERROR: (10601-98880) File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 111, in wsgi_application
pulp.server.webservices.application:ERROR: (10601-98880)
_initialize_web_services()
pulp.server.webservices.application:ERROR: (10601-98880) File
"/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py",
line 74, in _initialize_web_services
pulp.server.webservices.application:ERROR: (10601-98880)
migration_models.check_package_versions()
pulp.server.webservices.application:ERROR: (10601-98880) File
"/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py", line
310, in check_package_versions
pulp.server.webservices.application:ERROR: (10601-98880) raise
Exception(error_message)
pulp.server.webservices.application:ERROR: (10601-98880)
InitializationException: The database has not been migrated to the current
version. Run pulp-manage-db and restart the application.
pulp.server.webservices.application:CRITICAL:

··· *************************************************************

Then doing a

sudo -u apache /usr/bin/pulp-manage-db

and

systemctl restart pulp_resource_manager.service

was sufficient

Cheers
L.

On Thursday, September 22, 2016 at 2:03:40 PM UTC+10, Lachlan Musicman wrote:

I’m having trouble applying some Errata - after hitting confirm on
applying the Errata to the content hosts, I get an error message:

There was an issue with the backend service pulp: 500 Internal Server

Error

Centos 7.2, Katello 3.0 9.el7, Foreman 1.11.4

hammer ping tells me the pulp; Status is FAIL

systemctl says all pulp services are fine, except pulp-workers which
exited with SUCCESS.

In /var/log/foreman/production.log I see lots of output, but this is the
last dozen or so lines:

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/
methodoverride.rb:21:in `call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/runtime.rb:17:in
`call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/activesupport-4.1.5/lib/active_support/cache/strategy/local_cache_m
iddleware.rb:26:in `call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/actionpack-4.1.5/lib/
action_dispatch/middleware/static.rb:64:in `ca
ll’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/sendfile.rb:112:in
`call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/engine.rb:514:in
`call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/
application.rb:144:in `call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in
`public_send’

/opt/rh/rh-ror41/root/usr/share/gems/gems/railties-4.1.5/lib/rails/railtie.rb:194:in
`method_missing’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/builder.
rb:138:in `call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in
`block in call’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in
`each’

/opt/rh/rh-ror41/root/usr/share/gems/gems/rack-1.5.2/lib/rack/urlmap.rb
:50:in `call’

/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in
`process_request’

/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in
`accept_and_process_next_requ
est’

/usr/share/passenger/phusion_passenger/request_handler/thread_handler.
rb:109:in main_loop' /usr/share/passenger/phusion_passenger/request_handler.rb:455:inblock
(3 levels) in start_threads’

/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call’

/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/
diagnostic_context.rb:323:in `block in create_with_logging_context’
2016-09-22 14:01:25 [app] [I] Completed 500 Internal Server Error in 378ms
(Views: 0.2ms | ActiveRecord: 188.2ms)

How do I fix this problem?