Installer fails when enablling pulpcore

jjeffers · December 4, 2020, 1:47pm

While running the installer
$ sudo foreman-installer --foreman-proxy-plugin-pulp-pulpcore-enabled true

the following error is seen

[...] 
2020-12-04 13:46:01 [NOTICE] [configure] 1600 out of 1963 done.
2020-12-04 13:46:01 [NOTICE] [configure] 1700 out of 1963 done.
2020-12-04 13:46:01 [NOTICE] [configure] 1800 out of 1963 done.
2020-12-04 13:46:03 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2020-12-04 13:46:03 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2020-12-04 13:46:03 [NOTICE] [configure] 1900 out of 1963 done.
2020-12-04 13:46:04 [NOTICE] [configure] System configuration has finished.
2020-12-04 13:46:04 [NOTICE] [post] Executing hooks in group post

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman-proxy-content.log
2020-12-04 13:46:04 [NOTICE] [post] All hooks in group post finished

And the relevant parts of the foreman-proxy-content.log:

2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: Traceback (most recent call last):
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/bin/pulpcore-manager", line 11, in <module>
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     load_entry_point('pulpcore==3.7.3', 'console_scripts', 'pulpcore-manager')()
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     execute_from_command_line(sys.argv)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     utility.execute()
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 325, in execute
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     settings.INSTALLED_APPS
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     self._setup(name)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     self._wrapped = Settings(settings_module)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     mod = importlib.import_module(self.SETTINGS_MODULE)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     return _bootstrap._gcd_import(name[level:], package, level)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 248, in <module>
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     ENVVAR_FOR_DYNACONF="PULP_SETTINGS",
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 76, in load
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     lazy_settings.populate_obj(django_settings_module)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 141, in __getattr__
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     self._setup()
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 186, in _setup
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     default_settings.reload(self._kwargs.get("load_dotenv"))
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 67, in reload
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     start_dotenv(*args, **kwargs)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 53, in start_dotenv
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     or _find_file(".env", project_root=root_path)
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 60, in find_file
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     search_tree.extend(_walk_to_root(work_dir))
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 13, in _walk_to_root
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:     raise IOError("Starting path not found")
2020-12-04 13:40:25 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: OSError: Starting path not found
2020-12-04 13:40:25 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing 'pulpcore-manager migrate --noinput'
2020-12-04 13:40:25 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Traceback (most recent call last):
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 11, in <module>
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     load_entry_point('pulpcore==3.7.3', 'console_scripts', 'pulpcore-manager')()
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     execute_from_command_line(sys.argv)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     utility.execute()
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 325, in execute
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     settings.INSTALLED_APPS
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 79, in __getattr__
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self._setup(name)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 66, in _setup
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self._wrapped = Settings(settings_module)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/conf/__init__.py", line 157, in __init__
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     mod = importlib.import_module(self.SETTINGS_MODULE)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib64/python3.6/importlib/__init__.py", line 126, in import_module
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     return _bootstrap._gcd_import(name[level:], package, level)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 994, in _gcd_import
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 971, in _find_and_load
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/settings.py", line 248, in <module>
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     ENVVAR_FOR_DYNACONF="PULP_SETTINGS",
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/contrib/django_dynaconf_v2.py", line 76, in load
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     lazy_settings.populate_obj(django_settings_module)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 141, in __getattr__
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self._setup()
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/base.py", line 186, in _setup
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     default_settings.reload(self._kwargs.get("load_dotenv"))
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 67, in reload
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     start_dotenv(*args, **kwargs)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/default_settings.py", line 53, in start_dotenv
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     or _find_file(".env", project_root=root_path)
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 60, in find_file
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     search_tree.extend(_walk_to_root(work_dir))
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.6/site-packages/dynaconf/utils/files.py", line 13, in _walk_to_root
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     raise IOError("Starting path not found")
2020-12-04 13:40:26 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: OSError: Starting path not found
2020-12-04 13:40:26 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2020-12-04 13:40:26 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]

RPMs present:
foreman-release-2.4.0-0.1.develop.el7.noarch
foreman-installer-2.4.0-0.1.develop.20201202031220gitc214354.el7.noarch
foreman-proxy-2.4.0-0.1.develop.20201201075604git7c818d0.el7.noarch
rubygem-foreman_maintain-0.7.0-1.el7.noarch
centos7-proxy-devel.jjeffers.example.com-foreman-proxy-client-1.0-7.noarch
foreman-debug-2.4.0-0.2.develop.20201201151239git1a144f1.el7.noarch
centos7-proxy-devel.jjeffers.example.com-foreman-proxy-1.0-7.noarch
foreman-proxy-content-3.18.0-0.4.master.el7.noarch
foreman-installer-katello-2.4.0-0.1.develop.20201202031220gitc214354.el7.noarch

ekohl · December 4, 2020, 1:58pm

Do you have an installer log? I wonder if /etc/pulp/settings.py is available prior to running pulpcore-manager. The error suggests it can’t load .env but we don’t care about that. In fact, ideally we would hard disable it because it can load to unexpected results. If it loads it from the current working directory and that’s /root but the user is pulp, it can’t read that. Perhaps it’s a bug in dynaconf that it doesn’t handle that?

jjeffers · December 4, 2020, 2:08pm

Which log file(s) would be helpful here? There are a few candidates in /var/log/foreman-installer/.

ekohl · December 4, 2020, 2:18pm

The one from the installer run. Usually that’s /var/log/foreman-installer/$SCENARIO.log but it could also have rotated.

jjeffers · December 4, 2020, 2:31pm

foreman-proxy-content.log (2.1 MB)

Here is the corresponding log file to a failed installer run.

ekohl · December 4, 2020, 2:40pm

That doesn’t look like the same run. Relevant parts from the log you shared:

2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Config/Concat[pulpcore settings]/Concat_file[pulpcore settings]: Starting to evaluate the resource (910 of 1955)
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Config/Concat[pulpcore settings]/Concat_file[pulpcore settings]: Evaluated in 0.00 seconds
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Config/Concat[pulpcore settings]/File[/etc/pulp/settings.py]: Starting to evaluate the resource (911 of 1955)
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Config/Concat[pulpcore settings]/File[/etc/pulp/settings.py]: Evaluated in 0.00 seconds

2020-12-04 14:26:41 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Starting to evaluate the resource (919 of 1955)
2020-12-04 14:26:41 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Evaluated in 0.00 seconds
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Starting to evaluate the resource (920 of 1955)
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' won't be executed because of failed check 'refreshonly'
2020-12-04 14:26:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Evaluated in 0.00 seconds
2020-12-04 14:26:41 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Starting to evaluate the resource (921 of 1955)
2020-12-04 14:26:41 [DEBUG ] [configure] Pulpcore::Admin[collectstatic --noinput]: Evaluated in 0.00 seconds

2020-12-04 14:26:55 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Starting to evaluate the resource (1853 of 1963)
2020-12-04 14:26:55 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Evaluated in 0.00 seconds
2020-12-04 14:26:55 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Starting to evaluate the resource (1854 of 1963)
2020-12-04 14:26:55 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing check 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
2020-12-04 14:26:55 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
2020-12-04 14:26:58 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: System check identified some issues:
2020-12-04 14:26:58 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:
2020-12-04 14:26:58 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: WARNINGS:
2020-12-04 14:26:58 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: ?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
2020-12-04 14:26:58 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing 'pulpcore-manager migrate --noinput'
2020-12-04 14:26:58 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2020-12-04 14:27:18 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: executed successfully
2020-12-04 14:27:18 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: The container Pulpcore::Admin[migrate --noinput] will propagate my refresh event
2020-12-04 14:27:18 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Evaluated in 22.88 seconds
2020-12-04 14:27:18 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Starting to evaluate the resource (1855 of 1963)
2020-12-04 14:27:18 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: The container Class[Pulpcore::Database] will propagate my refresh event
2020-12-04 14:27:18 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Evaluated in 0.00 seconds
2020-12-04 14:27:18 [DEBUG ] [configure] Pulpcore::Admin[reset-admin-password --random]: Starting to evaluate the resource (1856 of 1963)
2020-12-04 14:27:18 [DEBUG ] [configure] Pulpcore::Admin[reset-admin-password --random]: Evaluated in 0.00 seconds
2020-12-04 14:27:18 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]: Starting to evaluate the resource (1857 of 1963)
2020-12-04 14:27:18 [DEBUG ] [configure] Exec[pulpcore-manager reset-admin-password --random](provider=posix): Executing check 'pulpcore-manager dumpdata auth.User | grep "\"username\": \"admin\""'
2020-12-04 14:27:18 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager dumpdata auth.User | grep "\"username\": \"admin\""'
2020-12-04 14:27:20 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]/unless: System check identified some issues:
2020-12-04 14:27:20 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]/unless:
2020-12-04 14:27:20 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]/unless: WARNINGS:
2020-12-04 14:27:20 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]/unless: ?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
2020-12-04 14:27:20 [DEBUG ] [configure] Exec[pulpcore-manager reset-admin-password --random](provider=posix): Executing 'pulpcore-manager reset-admin-password --random'
2020-12-04 14:27:20 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager reset-admin-password --random'
2020-12-04 14:27:23 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]/returns: executed successfully
2020-12-04 14:27:23 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]: The container Pulpcore::Admin[reset-admin-password --random] will propagate my refresh event
2020-12-04 14:27:23 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[reset-admin-password --random]/Exec[pulpcore-manager reset-admin-password --random]: Evaluated in 4.76 seconds
2020-12-04 14:27:23 [DEBUG ] [configure] Pulpcore::Admin[reset-admin-password --random]: Starting to evaluate the resource (1858 of 1963)
2020-12-04 14:27:23 [DEBUG ] [configure] Pulpcore::Admin[reset-admin-password --random]: The container Class[Pulpcore::Database] will propagate my refresh event
2020-12-04 14:27:23 [DEBUG ] [configure] Pulpcore::Admin[reset-admin-password --random]: Evaluated in 0.00 seconds

That all looks perfectly in order.

jjeffers · December 4, 2020, 3:16pm

Reran the installer again after removing every log file. There is only log file present with any data. This time the installer failed for a different reason:

snippet from log:
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: System check identified some issues:
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: WARNINGS:
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: ?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend').
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: No planned migration operations.
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: ‘pulpcore-manager migrate --noinput’ won’t be executed because of failed check ‘unless’
2020-12-04 15:13:41 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Evaluated in 2.73 seconds
2020-12-04 15:13:41 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Starting to evaluate the resource (1855 of 1963)

foreman-proxy-content.log (2.1 MB)

ekohl · December 4, 2020, 3:34pm

It is migrated now. You probably ran into some dependency bug which is why I asked for the exact log of the run. Now that you’ve removed the log files you can’t say anything useful and you can’t reproduce the original problem since it converged into a correct state.