Problem upgrading from katello 3.4 to katello 3.5

it4rnd · December 14, 2018, 4:05pm

Problem:
We are trying to update our Foreman installation from katello 3.4 to katello 3.5 (foreman 1.15 to 1.16) via: foreman-installer --scenario katello --upgrade

The update stops with the error:

foreman-rake katello:upgrades:3.0:add_export_distributor failed! Check the output for error!
Upgrade step add_export_distributor failed. Check logs for more information.

Checked the log files but we are unable to determine the cause of the error. Has anybody seen this error before our can point us in the right direction?
Relevant part of the katello.log file is added below.

Expected outcome:

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Other relevant data:

[DEBUG 2018-12-14 15:35:58 main] Importing Subscriptions
[DEBUG 2018-12-14 15:35:58 main] foreman-rake katello:import_subscriptions finished successfully!
[ INFO 2018-12-14 15:35:58 main] Upgrade Step: elasticsearch_message...
[ INFO 2018-12-14 15:35:58 main] Elasticsearch already removed, skipping
[ INFO 2018-12-14 15:35:58 main] Upgrade Step: add_export_distributor (this may take a while) ...
[DEBUG 2018-12-14 15:36:02 main] /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
[DEBUG 2018-12-14 15:36:02 main] /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[DEBUG 2018-12-14 15:36:39 main] rake aborted!
[DEBUG 2018-12-14 15:36:39 main] RestClient::NotFound: 404 Not Found
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.5.0/lib/runcible/base.rb:81:in `block in get_response'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.5.0/lib/runcible/base.rb:80:in `get_response'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.5.0/lib/runcible/base.rb:68:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.5.0/lib/runcible/resources/repository.rb:31:in `retrieve'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.5.0/lib/runcible/extensions/repository.rb:415:in `retrieve_with_details'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/app/lib/actions/pulp/repository/refresh.rb:11:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:473:in `block (3 levels) in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/app/lib/actions/middleware/backend_services_check.rb:23:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/app/lib/actions/middleware/remote_action.rb:9:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:35:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:35:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/world.rb:30:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:472:in `block (2 levels) in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `switch_flow'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:376:in `concurrence'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:471:in `block in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `block in with_error_handling'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `catch'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `with_error_handling'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:470:in `execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:268:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:404:in `plan_action'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/app/lib/actions/katello/repository/refresh_repository.rb:8:in `block in plan'
[DEBUG 2018-12-14 15:36:39 main] /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:93:in `as'
[DEBUG 2018-12-14 15:36:39 main] /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:99:in `as_anonymous_admin'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/app/lib/actions/katello/repository/refresh_repository.rb:6:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:473:in `block (3 levels) in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:35:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:35:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/world.rb:30:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:472:in `block (2 levels) in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `switch_flow'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:376:in `concurrence'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:471:in `block in execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `block in with_error_handling'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `catch'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:431:in `with_error_handling'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:470:in `execute_plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/action.rb:268:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:252:in `block (2 levels) in plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:350:in `switch_flow'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:340:in `with_planning_scope'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:251:in `block in plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:43:in `plan_phase'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:26:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware.rb:18:in `pass'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `block in transaction'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/transaction.rb:184:in `within_new_transaction'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/connection_adapters/abstract/database_statements.rb:213:in `transaction'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/transactions.rb:220:in `transaction'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/stack.rb:22:in `call'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/middleware/world.rb:30:in `execute'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/execution_plan.rb:250:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/world.rb:176:in `block in plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/world.rb:174:in `tap'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/world.rb:174:in `plan'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.34/lib/dynflow/world.rb:150:in `trigger'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.10.9/lib/foreman_tasks.rb:19:in `trigger'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.10.9/lib/foreman_tasks.rb:25:in `block in trigger_task'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.10.9/lib/foreman_tasks.rb:42:in `rails_safe_trigger_task'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.10.9/lib/foreman_tasks.rb:23:in `trigger_task'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.10.9/lib/foreman_tasks.rb:51:in `sync_task'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:9:in `block (5 levels) in <top (required)>'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/batches.rb:51:in `block (2 levels) in find_each'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/batches.rb:51:in `each'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/batches.rb:51:in `block in find_each'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/batches.rb:124:in `find_in_batches'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/relation/batches.rb:50:in `find_each'
[DEBUG 2018-12-14 15:36:39 main] /opt/rh/sclo-ror42/root/usr/share/gems/gems/activerecord-4.2.5.1/lib/active_record/querying.rb:9:in `find_each'
[DEBUG 2018-12-14 15:36:39 main] /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.5.2/lib/katello/tasks/upgrades/3.0/add_export_distributor.rake:8:in `block (4 levels) in <top (required)>'
[DEBUG 2018-12-14 15:36:39 main] Tasks: TOP => katello:upgrades:3.0:add_export_distributor
[DEBUG 2018-12-14 15:36:39 main] (See full trace by running task with --trace)
[DEBUG 2018-12-14 15:36:39 main] Refreshing existing repositories to add export distributor
[ERROR 2018-12-14 15:36:39 main] foreman-rake katello:upgrades:3.0:add_export_distributor failed! Check the output for error!
[ERROR 2018-12-14 15:36:39 main] Upgrade step add_export_distributor failed. Check logs for more information.
[DEBUG 2018-12-14 15:36:39 main] Exit with status code: 1 (signal was 1)
[ERROR 2018-12-14 15:36:39 main] Errors encountered during run:
[ERROR 2018-12-14 15:36:39 main] foreman-rake katello:upgrades:3.0:add_export_distributor failed! Check the output for error!
[ERROR 2018-12-14 15:36:39 main] Upgrade step add_export_distributor failed. Check logs for more information.
[DEBUG 2018-12-14 15:36:39 main] Cleaning /tmp/kafo_puppet20181214-25925-300nr3.conf
[DEBUG 2018-12-14 15:36:39 main] Cleaning /tmp/kafo_hiera20181214-25925-14mlaa8
[DEBUG 2018-12-14 15:36:39 main] Cleaning /tmp/kafo_puppet20181214-25925-1oppp42.conf
[DEBUG 2018-12-14 15:36:39 main] Cleaning /tmp/default_values.yaml
[ INFO 2018-12-14 15:36:39 main] Installer finished in 1147.819968733 seconds

Justin_Sherrill · December 17, 2018, 2:42pm

It looks like there is a mismatch between katello and pulp. Can you try running:

foreman-rake katello:correct_repositories

and then re-run the installer with --upgrade

it4rnd · December 17, 2018, 3:26pm

Thank you for your reply. We were able to run ‘foreman-rake katello:correct_repositories’ without errors but run into the same problem again when re-running the installer with --upgrade.
Do you have any other suggestions?

Justin_Sherrill · December 17, 2018, 4:39pm

Did you happen to save the output? I forgot to mention that it only displays what it will do, but not actually perform any correction. You have to run with COMMIT=true to get it to do anything:

foreman-rake katello:correct_repositories COMMIT=true

apologies for that, can you give that a shot?

it4rnd · December 18, 2018, 3:45pm

We ran the same command again including the “COMMIT=true” and that finished without problems. Unfortunately we ran into the same issue again when re-running the installer with --upgrade.
In the meantime we think that we understand why we get this error. We are running this upgrade on our test server and that setup does not contain the repositories. We think the upgrade tooling is trying to update the repositories as well and as they are not available it fails.
We are currently running the upgrade on our production environment, We now reached the same step and this is taking a very long time but it appears to be working. Will post an update once this step is finished (or the upgrade script stops).

it4rnd · January 2, 2019, 9:30am

As mentioned in my previous post we tried to update our production environment.
We initiated this update on December 20 and it seems to be stuck at the step:
Upgrade Step: add_export_distributor (this may take a while) …
/usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
/usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here

Based on the output of foreman-tail the process appears to be in some kind of loop processing all hosts.

The last entries from /var/log/foreman-installer/katello.log:
[ INFO 2018-12-20 11:14:13 main] Upgrade Step: clean_backend_objects (this may take a while) …
[DEBUG 2018-12-20 11:14:18 main] /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
[DEBUG 2018-12-20 11:14:18 main] /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[DEBUG 2018-12-20 11:14:29 main] 0 orphaned consumer id(s) found in candlepin.
[DEBUG 2018-12-20 11:14:29 main] 0 orphaned consumer id(s) found in pulp.
[DEBUG 2018-12-20 11:14:29 main] foreman-rake katello:clean_backend_objects COMMIT=true finished successfully!
[ INFO 2018-12-20 11:14:29 main] Upgrade Step: add_export_distributor (this may take a while) …
[DEBUG 2018-12-20 11:14:34 main] /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
[DEBUG 2018-12-20 11:14:34 main] /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here

The GUI is showing the updated foreman version (1.16) but it’s unclear what the current status is.

Justin_Sherrill · January 2, 2019, 9:41pm

It sounds like possibly pulp isn’t properly up and running tasks? You might try running:

systemctl stop pulp_workers pulp_celerybeat pulp_resource_manager
systemctl start pulp_workers pulp_celerybeat pulp_resource_manager

And you might want to use pulp-admin to monitor the running tasks to see if any are actually running. You might also want to look in /var/log/messages for any errors from pulp that may indicate why its not processing tasks.

it4rnd · January 3, 2019, 3:07pm

We stopped all foreman processes and reran the upgrade after all processes where started again.
This time the upgrade finished successfully. Thank you for your help.