Foreman 3.3 / Katello 4.5 upgrade failed [pulpcore-manager migrate --noinput]

Problem:
I ran into the following error when trying to upgrade from Foreman 3.2 / Katello 4.4

2022-11-01 10:58:29 [NOTICE] [configure] Starting system configuration.
2022-11-01 10:59:14 [NOTICE] [configure] 250 configuration steps out of 1662 steps complete.
2022-11-01 11:00:42 [NOTICE] [configure] 500 configuration steps out of 1663 steps complete.
2022-11-01 11:00:53 [NOTICE] [configure] 750 configuration steps out of 1667 steps complete.
2022-11-01 11:01:40 [NOTICE] [configure] 1000 configuration steps out of 1712 steps complete.
2022-11-01 11:01:52 [NOTICE] [configure] 1250 configuration steps out of 1712 steps complete.
2022-11-01 11:15:03 [NOTICE] [configure] 1500 configuration steps out of 1712 steps complete.
2022-11-01 11:22:36 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 11:22:36 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 11:22:51 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Failed to call refresh: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 11:22:51 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 11:22:55 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

This is the result if I try running the command directly:

[pulp@katello ~]$ PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager migrate --noinput
System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
Operations to perform:
  Apply all migrations: admin, ansible, auth, certguard, container, contenttypes, core, deb, file, guardian, python, rpm, sessions
Running migrations:
  Applying core.0077_move_remote_url_credentials...Traceback (most recent call last):
  File "/usr/bin/pulpcore-manager", line 33, in <module>
    sys.exit(load_entry_point('pulpcore==3.18.10', 'console_scripts', 'pulpcore-manager')())
  File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
    res = handle_func(*args, **kwargs)
  File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
    post_migrate_state = executor.migrate(
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
    state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
    state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
    state = migration.apply(state, schema_editor)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
    operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
  File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
    self.code(from_state.apps, schema_editor)
  File "/usr/lib/python3.9/site-packages/pulpcore/app/migrations/0077_move_remote_url_credentials.py", line 19, in move_remote_url_credentials
    _, url_split = url.netloc.rsplit("@", maxsplit=1)
ValueError: not enough values to unpack (expected 2, got 1)

Expected outcome:
Katello upgrade is successfull

Foreman and Proxy versions:
foreman-3.3.1-1.el8.noarch

Foreman and Proxy plugin versions:
katello-4.5.1-1.el8.noarch

Distribution and version:
Red Hat Enterprise Linux release 8.6 (Ootpa)

Other relevant data:
Installer LOG

2022-11-01 13:15:46 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Starting to evaluate the resource (1563 of 1674)
2022-11-01 13:15:46 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Evaluated in 0.00 seconds
2022-11-01 13:15:46 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Starting to evaluate the resource (1564 of 1674)
2022-11-01 13:15:46 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing check 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
2022-11-01 13:15:46 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --plan | grep "No planned migration operations"'
2022-11-01 13:15:51 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: System check identified some issues:
2022-11-01 13:15:51 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless:
2022-11-01 13:15:51 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: WARNINGS:
2022-11-01 13:15:51 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/unless: ?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('djan
go.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
2022-11-01 13:15:51 [DEBUG ] [configure] Exec[pulpcore-manager migrate --noinput](provider=posix): Executing 'pulpcore-manager migrate --noinput'
2022-11-01 13:15:51 [DEBUG ] [configure] Executing with uid=pulp: 'pulpcore-manager migrate --noinput'
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: System check identified some issues:
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: WARNINGS:
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: ?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: `AUTHENTICATION_BACKENDS = ('dja
ngo.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend')`.
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Operations to perform:
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   Apply all migrations: admin, ansible, auth, certguard, container, contenttypes, core, deb, file, guardian, python, rpm, sessions
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: Running migrations:
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   Applying core.0077_move_remote_url_credentials...Traceback (most recent call last):
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 33, in <module>
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     sys.exit(load_entry_point('pulpcore==3.18.10', 'console_scripts', 'pulpcore-manager')())
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     execute_from_command_line(sys.argv)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     utility.execute()
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.fetch_command(subcommand).run_from_argv(self.argv)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.execute(*args, **cmd_options)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     output = self.handle(*args, **options)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/base.py", line 89, in wrapped
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     res = handle_func(*args, **kwargs)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/core/management/commands/migrate.py", line 244, in handle
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     post_migrate_state = executor.migrate(
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 117, in migrate
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = self._migrate_all_forwards(state, plan, full_plan, fake=fake, fake_initial=fake_initial)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 147, in _migrate_all_forwards
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = self.apply_migration(state, migration, fake=fake, fake_initial=fake_initial)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/executor.py", line 227, in apply_migration
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     state = migration.apply(state, schema_editor)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/migration.py", line 126, in apply
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     operation.database_forwards(self.app_label, schema_editor, old_state, project_state)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/django/db/migrations/operations/special.py", line 190, in database_forwards
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     self.code(from_state.apps, schema_editor)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:   File "/usr/lib/python3.9/site-packages/pulpcore/app/migrations/0077_move_remote_url_credentials.py", line 19, in move_remote_url_credentials
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns:     _, url_split = url.netloc.rsplit("@", maxsplit=1)
2022-11-01 13:15:57 [INFO  ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: ValueError: not enough values to unpack (expected 2, got 1)
2022-11-01 13:15:57 [ERROR ] [configure] 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 13:15:57 [ERROR ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]/returns: change from 'notrun' to ['0'] failed: 'pulpcore-manager migrate --noinput' returned 1 instead of one of [0]
2022-11-01 13:15:57 [DEBUG ] [configure] /Stage[main]/Pulpcore::Database/Pulpcore::Admin[migrate --noinput]/Exec[pulpcore-manager migrate --noinput]: Evaluated in 10.66 seconds
2022-11-01 13:15:57 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Starting to evaluate the resource (1565 of 1674)
2022-11-01 13:15:57 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Resource is being skipped, unscheduling all events
2022-11-01 13:15:57 [DEBUG ] [configure] Pulpcore::Admin[migrate --noinput]: Evaluated in 0.00 seconds

Hi,

i am sitting in the same boat, but for me its Foreman/Katello 3.3/4.5 to 3.4/4.6.

It also failed with this:

?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend').

The guardian warning is harmless.

@iballou could you have a look here?

Thanks to @sjansen for being my guinea pig, we managed to produce the following patch that should fix the issue: Fix migrating Remotes with @ in path by evgeni · Pull Request #3410 · pulp/pulpcore · GitHub

2 Likes

Thanks @evgeni !

Thanks @evgeni the patch worked, I was able to upgrade to Foreman 3.3 & Katello 4.5 and then to Foreman 3.4 & Katello 4.6

Looks like this was the repo that caused it for me: Index of /results/@qpid/qpid/epel-7-x86_64/

Yeah, for @sjansen it also was a copr repo.