Install error when upgrading 1.24.3 to 2.0.1

Problem:
Followed Katello upgrade procedure and all was well until the end when it threw the error below. Foreman/Katello seems to be working, but it will not complete a “foreman-installer” run without the error.

Expected outcome:
Installer run completes without error.

Foreman and Proxy versions:
1.24.3 → 2.0.1
Katello
3.14.1 → 3.15.3

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.8

Other relevant data:

[ERROR 2020-08-05T06:21:50 main] Errors encountered during run:
[ERROR 2020-08-05T06:21:50 main]  'python3-django-admin migrate --noinput' returned 1 instead of one of [0]
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:157:in `fail'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/exec.rb:183:in `sync'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:241:in `sync'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:136:in `sync_if_needed'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:90:in `block in perform_changes'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `each'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:89:in `perform_changes'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:267:in `apply'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:287:in `eval_resource'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `call'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block (2 levels) in evaluate'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:191:in `block in evaluate'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:122:in `traverse'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:178:in `evaluate'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:240:in `block (2 levels) in apply'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block in apply'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `apply'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block (2 levels) in apply_catalog'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:545:in `block in thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:544:in `thinmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `block in apply_catalog'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:183:in `apply_catalog'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:399:in `run_internal'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:227:in `block in run'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:210:in `run'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:734:in `exit_on_fail'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:143:in `run'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2020-08-05T06:21:50 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2020-08-05T06:21:50 main]  /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[python3-django-admin migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'python3-django-admin migrate --noinput' returned 1 instead of one of [0]

Looks like we don’t log the ouput on fail, that should be fixed.

If you run PULP_SETTINGS=/etc/pulp/settings.py python3-django-admin migrate --noinput, what do you see?

[root@s2-foreman ~]# PULP_SETTINGS=/etc/pulp/settings.py python3-django-admin migrate --noinput
Traceback (most recent call last):
  File "/bin/python3-django-admin", line 11, in <module>
    load_entry_point('Django==2.2.10', 'console_scripts', 'django-admin')()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 361, in execute
    self.check()
  File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 390, in check
    include_deployment_checks=include_deployment_checks,
  File "/usr/lib/python3.6/site-packages/django/core/management/commands/migrate.py", line 64, in _run_checks
    issues = run_checks(tags=[Tags.database])
  File "/usr/lib/python3.6/site-packages/django/core/checks/registry.py", line 72, in run_checks
    new_errors = check(app_configs=app_configs)
  File "/usr/lib/python3.6/site-packages/django/core/checks/database.py", line 9, in check_database_backends
    for conn in connections.all():
  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 216, in all
    return [self[alias] for alias in self]
  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 213, in __iter__
    return iter(self.databases)
  File "/usr/lib/python3.6/site-packages/django/utils/functional.py", line 80, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/lib/python3.6/site-packages/django/db/utils.py", line 147, in databases
    self._databases = settings.DATABASES
  File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
    self._setup(name)
  File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 64, in _setup
    % (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting DATABASES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.

@ekohl — Any thoughts on this?

@ekohl — It seems that there are problems that I did not initially see. Some may be related to another issue. Content view removal fails on Foreman 2.0.1

Ok, this fixed the installer problem and the related syncing issue.
Thanks @John_Beranek.

Got the link wrong, here is the correct one.