Katello 4.0RC2 installation issue

Problem:
Initial installation of Katello 4.0RC2 fails, with Pulp steps
Expected outcome:
Installation completes
Foreman and Proxy versions:
foreman-2.4.0-0.4.rc2.el8.noarch
foreman-proxy-2.4.0-0.1.rc2.el8.noarch
Foreman and Proxy plugin versions:
katello-4.0.0-0.9.rc2.el8.noarch
Distribution and version:
Oracle Linux 8.3
Other relevant data:

# foreman-installer --scenario katello \
>                   --foreman-initial-organization Example \
>                   --foreman-initial-location "Default Location" \
>                   --foreman-initial-admin-username admin \
>                   --foreman-initial-admin-password "MyComplexPassword" \
>                   --certs-server-cert "/root/foreman_cert/katello_cert.pem" \
>                   --certs-server-key "/root/foreman_cert/foreman_cert_key.pem" \
>                   --certs-server-ca-cert "/root/foreman_cert/cacert.crt"
2021-03-03 11:36:32 [NOTICE] [root] Loading default values from puppet modules...
2021-03-03 11:36:35 [NOTICE] [root] ... finished
2021-03-03 11:36:36 [NOTICE] [root] Running validation checks
Executing: katello-certs-check -c "/root/foreman_cert/katello_cert.pem" -k "/root/foreman_cert/foreman_cert_key.pem" -b "/root/foreman_cert/cacert.crt"
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
tput: No value for $TERM and no -T specified
Checking server certificate encoding: [OK]

Checking expiration of certificate: [OK]

Checking expiration of CA bundle: [OK]

Checking if server certificate has CA:TRUE flag [OK]

Checking for private key passphrase: [OK]

Checking to see if the private key matches the certificate: [OK]

Checking CA bundle against the certificate file: [OK]

Checking CA bundle size: 2
[OK]

Checking Subject Alt Name on certificate [OK]

Checking if any Subject Alt Name on certificate matches the Subject CN
[OK]

Checking Key Usage extension on certificate for Key Encipherment [OK]

Checking for use of shortname as CN
[OK]

Validation succeeded


To install the Katello server with the custom certificates, run:
    foreman-installer --scenario katello \
                      --certs-server-cert "/root/foreman_cert/katello_cert.pem" \
                      --certs-server-key "/root/foreman_cert/foreman_cert_key.pem" \
                      --certs-server-ca-cert "/root/foreman_cert/cacert.crt"

To update the certificates on a currently running Katello installation, run:

    foreman-installer --scenario katello \
                      --certs-server-cert "/root/foreman_cert/katello_cert.pem" \
                      --certs-server-key "/root/foreman_cert/foreman_cert_key.pem" \
                      --certs-server-ca-cert "/root/foreman_cert/cacert.crt" \
                      --certs-update-server --certs-update-server-ca

To use them inside a NEW $FOREMAN_PROXY, rerun this command with -t foreman-proxy
2021-03-03 11:39:05 [NOTICE] [configure] Starting system configuration.
  The total number of configuration tasks may increase during the run.
  Observe logs or specify --verbose-log-level to see individual configuration tasks.
2021-03-03 11:39:25 [NOTICE] [configure] 100 out of 1954 done.
2021-03-03 11:39:38 [NOTICE] [configure] 200 out of 1954 done.
2021-03-03 11:40:10 [NOTICE] [configure] 300 out of 1954 done.
2021-03-03 11:40:36 [NOTICE] [configure] 400 out of 1954 done.
2021-03-03 11:41:21 [NOTICE] [configure] 500 out of 1954 done.
2021-03-03 11:41:28 [NOTICE] [configure] 600 out of 1956 done.
2021-03-03 11:41:28 [NOTICE] [configure] 700 out of 1956 done.
2021-03-03 11:41:30 [NOTICE] [configure] 800 out of 1958 done.
2021-03-03 11:42:04 [NOTICE] [configure] 900 out of 1959 done.
2021-03-03 11:42:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2021-03-03 11:42:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2021-03-03 11:42:40 [NOTICE] [configure] 1000 out of 1961 done.
2021-03-03 11:42:45 [NOTICE] [configure] 1100 out of 1964 done.
2021-03-03 11:42:51 [NOTICE] [configure] 1200 out of 1984 done.
2021-03-03 11:42:55 [NOTICE] [configure] 1300 out of 1985 done.
2021-03-03 11:43:13 [NOTICE] [configure] 1400 out of 1985 done.
2021-03-03 11:48:33 [NOTICE] [configure] 1500 out of 1985 done.
2021-03-03 11:48:33 [NOTICE] [configure] 1600 out of 1985 done.
2021-03-03 11:49:55 [NOTICE] [configure] 1700 out of 1985 done.
2021-03-03 11:49:55 [NOTICE] [configure] 1800 out of 1985 done.
2021-03-03 11:49:58 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2021-03-03 11:49:58 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2021-03-03 11:49:59 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2021-03-03 11:49:59 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2021-03-03 11:49:59 [NOTICE] [configure] 1900 out of 1985 done.
2021-03-03 11:50:01 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

Running " pulpcore-manager collectstatic --noinput" by hand gives:

# pulpcore-manager collectstatic --noinput
/usr/lib/python3.6/site-packages/pulpcore/app/apps.py:72: FutureWarning: The plugin `deb` is missing a version attribute. Starting with pulpcore==3.10, plugins are required to define their version on the PulpPluginAppConfig subclass.
  warnings.warn(msg, FutureWarning)
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 269, in <module>
    CONTENT_ORIGIN
NameError: name 'CONTENT_ORIGIN' is not defined

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/bin/pulpcore-manager", line 11, in <module>
    load_entry_point('pulpcore==3.9.1', 'console_scripts', 'pulpcore-manager')()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 357, in execute
    django.setup()
  File "/usr/lib/python3.6/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/lib/python3.6/site-packages/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/apps.py", line 94, in ready
    self.import_viewsets()
  File "/usr/lib/python3.6/site-packages/pulpcore/app/apps.py", line 127, in import_viewsets
    from pulpcore.app.viewsets import NamedModelViewSet
  File "/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/__init__.py", line 1, in <module>
    from .base import (  # noqa
  File "/usr/lib/python3.6/site-packages/pulpcore/app/viewsets/base.py", line 14, in <module>
    from pulpcore.openapi import PulpAutoSchema
  File "/usr/lib/python3.6/site-packages/pulpcore/openapi/__init__.py", line 25, in <module>
    from pulpcore.app.settings import INSTALLED_PULP_PLUGINS
  File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 273, in <module>
    "CONTENT_ORIGIN is a required setting but it was not configured. This may be caused "
django.core.exceptions.ImproperlyConfigured: CONTENT_ORIGIN is a required setting but it was not configured. This may be caused by invalid read permissions of the settings file. Note that CONTENT_ORIGIN is set by the installer automatically.

Sadly pulpcore-manager doesn’t understand it should read /etc/pulp/settings.py by default. Also, you should not run it as root since that can mess up permissions and create problems in the future. So you should run it as pulp and set PULP_SETTINGS=/etc/pulp/settings.py.

You should also have a more detailed log in /var/log/foreman-installer/katello.log.

2021-03-03 11:42:05 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Starting to evaluate the resource (949 of 1960)
2021-03-03 11:42:05 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Scheduling refresh of Exec[pulpcore-manager collectstatic --noinput]
2021-03-03 11:42:05 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Evaluated in 0.00 seconds
2021-03-03 11:42:05 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Starting to evaluate the resource (950 of 1960)
2021-03-03 11:42:05 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' won't be executed because of failed check 'refreshonly'
2021-03-03 11:42:05 [DEBUG ] [configure] Exec[pulpcore-manager collectstatic --noinput](provider=posix): Executing 'pulpcore-manager collectstatic --noinput'
2021-03-03 11:42:05 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager collectstatic --noinput'
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: Traceback (most recent call last):
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 11, in <module>
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     load_entry_point('pulpcore==3.9.1', 'console_scripts', 'pulpcore-manager')()
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     execute_from_command_line(sys.argv)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from
_command_line
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     utility.execute()
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 325, in execute
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     settings.INSTALLED_APPS
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self._setup(name)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self._wrapped = Settings(settings_module)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     mod = importlib.import_module(self.SETTINGS_MODULE)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     return _bootstrap._gcd_import(name[level:], package, level)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 264, in <module>
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     ENVVAR_FOR_DYNACONF="PULP_SETTINGS",
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 76, in load
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     lazy_settings.populate_obj(django_settings_module)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 141, in __getattr__
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self._setup()
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 186, in _setup
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     default_settings.reload(self._kwargs.get("load_dotenv"))
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 67, in reload
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     start_dotenv(*args, **kwargs)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 53, in start_dotenv
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     or _find_file(".env", project_root=root_path)
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 60, in find_file
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     search_tree.extend(_walk_to_root(work_dir))
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 13, in _walk_to_root
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     raise IOError("Starting path not found")
2021-03-03 11:42:06 [INFO  ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: OSError: Starting path not found
2021-03-03 11:42:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2021-03-03 11:42:06 [ERROR ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
2021-03-03 11:42:06 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Evaluated in 0.52 seconds

So, oddly enough, I ran the pulpcore-manager commands by hand, as advised, and they completed. I then ran the installer again and it completed (with a few warnings):

2021-03-03 14:36:16 [NOTICE] [configure] Starting system configuration.
  The total number of configuration tasks may increase during the run.
  Observe logs or specify --verbose-log-level to see individual configuration tasks.
2021-03-03 14:36:28 [NOTICE] [configure] 100 out of 1954 done.
2021-03-03 14:36:28 [NOTICE] [configure] 200 out of 1954 done.
2021-03-03 14:36:31 [NOTICE] [configure] 300 out of 1954 done.
2021-03-03 14:36:35 [NOTICE] [configure] 400 out of 1954 done.
2021-03-03 14:36:36 [NOTICE] [configure] 500 out of 1954 done.
2021-03-03 14:36:36 [NOTICE] [configure] 600 out of 1956 done.
2021-03-03 14:36:36 [NOTICE] [configure] 700 out of 1956 done.
2021-03-03 14:36:37 [NOTICE] [configure] 800 out of 1958 done.
2021-03-03 14:36:38 [NOTICE] [configure] 900 out of 1959 done.
2021-03-03 14:36:38 [NOTICE] [configure] 1000 out of 1961 done.
2021-03-03 14:36:38 [NOTICE] [configure] 1100 out of 1964 done.
2021-03-03 14:36:38 [NOTICE] [configure] 1200 out of 1965 done.
2021-03-03 14:36:39 [NOTICE] [configure] 1300 out of 1966 done.
2021-03-03 14:37:05 [NOTICE] [configure] 1400 out of 1966 done.
2021-03-03 14:37:05 [NOTICE] [configure] 1500 out of 1966 done.
2021-03-03 14:37:05 [NOTICE] [configure] 1600 out of 1966 done.
2021-03-03 14:37:05 [NOTICE] [configure] 1700 out of 1966 done.
2021-03-03 14:37:05 [NOTICE] [configure] 1800 out of 1966 done.
2021-03-03 14:37:23 [NOTICE] [configure] 1900 out of 1966 done.
2021-03-03 14:37:36 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
=============================================
Upgrade Step 1/16: katello:correct_repositories. This may take a long while.
=============================================
Upgrade Step 2/16: katello:correct_puppet_environments. This may take a long while.
=============================================
Upgrade Step 3/16: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
=============================================
Upgrade Step 4/16: katello:upgrades:3.8:clear_checksum_type. =============================================
Upgrade Step 5/16: katello:upgrades:3.10:clear_invalid_repo_credentials. =============================================
Upgrade Step 6/16: katello:upgrades:3.10:update_gpg_key_urls. Failed upgrade task: katello:upgrades:3.10:update_gpg_key_urls, see logs for more information.
=============================================
Upgrade Step 7/16: katello:upgrades:3.11:import_yum_metadata. Importing Yum Metadata Files
Failed upgrade task: katello:upgrades:3.11:import_yum_metadata, see logs for more information.
=============================================
Upgrade Step 8/16: katello:upgrades:3.11:update_puppet_repos. =============================================
Upgrade Step 9/16: katello:upgrades:3.11:clear_checksum_type. =============================================
Upgrade Step 10/16: katello:upgrades:3.12:remove_pulp2_notifier. Failed upgrade task: katello:upgrades:3.12:remove_pulp2_notifier, see logs for more information.
=============================================
Upgrade Step 11/16: katello:upgrades:3.13:republish_deb_metadata. Input file /var/lib/pulp/0004_deb_repo_republish_candidates.txt was not readable.
You can manually use an alternate input file by running "foreman-rake katello:upgrades:3.13:republish_deb_metadata[<path>]"
=============================================
Upgrade Step 12/16: katello:upgrades:3.15:set_sub_facet_dmi_uuid. =============================================
Upgrade Step 13/16: katello:upgrades:3.15:reindex_rpm_modular. Failed upgrade task: katello:upgrades:3.15:reindex_rpm_modular, see logs for more information.
=============================================
Upgrade Step 14/16: katello:upgrades:3.16:update_applicable_el8_hosts. =============================================
Upgrade Step 15/16: katello:upgrades:3.18:add_cvv_export_history_metadata. =============================================
Upgrade Step 16/16: katello:upgrades:4.0:remove_ostree_puppet_content.   Success!
  * Foreman is running at https://katello.example.com
      Initial credentials are admin / MySuperPassword
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at https://katello.example.com:9090

  The full log is at /var/log/foreman-installer/katello.log

@John_Beranek Thanks for reporting those warnings, they should be safe to ignore for now. I’ve filed Bug #32007: remove old pulp2-based upgrade tasks (warnings on 4.0 install) - Katello - Foreman to remove those tasks for 4.0

For the upgrade, IOError("Starting path not found") appears to be caused when it tries to find .env somewhere. I suspect you are in a directory that the pulp user doesn’t have access to or some other odd thing. This is new to me so I reached out to the Pulp folks. It was suggested to never load .env in production setups. In our installation it doens’t make sense (we configure /etc/pulp/settings.py correct) and can only lead to problems.

This probably doesn’t help you in the short term but I would try to run the installer while in /. Other possibilities may be permissions or SELinux.

OK, so I was running the installer from /root/foreman_cert - which only root would be able to get to.

Can you write every cli command ?
I have the same problem and i dont know, how to initiate correct migration.