[ContentMigration] content prepare fails with Katello::Errors::Pulp3Error: 'NoneType' object has no attribute 'replace'

senetm · April 14, 2021, 6:10pm

Problem:
When starting foreman-maintain content prepare it failes with:
Katello::Errors::Pulp3Error: ‘NoneType’ object has no attribute ‘replace’

Expected outcome:

Foreman and Proxy versions:
foreman-2.3.3
katello-3.18.2-1
Pulp 2.1.0
Pulp server version 2.21.5

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7.9

Other relevant data:

indent preformatted text by 4 spaces
    Running Prepare content for Pulp 3

================================================================================

Prepare content for Pulp 3:
Starting task.
2021-04-14 19:57:30 +0200: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.lan/foreman_tasks/tasks/a65170a7-9589-45c0-9017-101ea6ecc14a
rake aborted!
ForemanTasks::TaskError: Task a65170a7-9589-45c0-9017-101ea6ecc14a: Katello::Errors::Pulp3Error: ‘NoneType’ object has no attribute ‘replace’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in` <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1

Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”

iballou · April 21, 2021, 9:18pm

Hi @senetm,

Do you see the ‘NoneType’ object has no attribute ‘replace’ error in /var/log/messages? If so, can we see the entire backtrace?

Also, can you show us any backtraces from the failed Foreman task? The Dynflow console should show some.

senetm · April 22, 2021, 12:17pm

Hi,
here are the 2 files:

iballou · April 22, 2021, 4:46pm

Hi @senetm,

Thanks for the files. It looks like you are hitting a Pulp 3 bug, so I’ve created an issue: Issue #8613: ‘NoneType’ object has no attribute ‘replace’ when migrating Debian content - Migration Plugin - Pulp

Do you mind sharing your Pulp 3 versions here and/or on the Pulp 3 bug? You just paste the result of:

pip3 list | grep pulp

From the file where the error originated, it looks like Debian content has something to do with it. Do you have Debian repositories? If so, it might be good to know which ones so we can try to reproduce the problem.

Thanks!

senetm · April 23, 2021, 7:10am

Hi,
thanks for your help.

Output from pip3 list | grep pulp:

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
pulp-2to3-migration (0.11.0)
pulp-certguard (1.0.3)
pulp-container (2.1.0)
pulp-deb (2.7.0)
pulp-file (1.3.0)
pulp-rpm (3.10.0)
pulpcore (3.7.5)

We have Ubuntu 18.04 and Ubuntu 20.04.

iballou · April 23, 2021, 1:43pm

Thanks for the info @senetm.

Also, don’t worry about the failed content prepare affecting your system. You can try it multiple times. Just make sure that all of the content migration related commands work for you before you upgrade to Katello 4.0 in the future.

Justin_Sherrill · April 27, 2021, 8:54pm

I’ve been testing out official debian support and ran into a similar error. My guess is that your components field is blank for a debian repository? you may want to check that and try adding a component to the list to limit what is synced. You could also add a component, migrate, and then remove it after migration/switchover.

I hit a similar error the architecture field

Justin_Sherrill · April 27, 2021, 8:55pm

Oh and you probably want to do the same with the architecture option under your debian repo details as well

senetm · April 28, 2021, 2:39pm

Now I am getting:

Running Prepare content for Pulp 3

Prepare content for Pulp 3:
Starting task.
2021-04-28 16:25:46 +0200: Pre-migrating Pulp 2 ERRATUM content (detail info) 26413/26472Migration failed, You will want to investigate: https://foreman.com/foreman_tasks/tasks/8e922043-98a1-4726-9bd1-b25b7861cdf5
rake aborted!
ForemanTasks::TaskError: Task 8e922043-98a1-4726-9bd1-b25b7861cdf5: Katello::Errors::Pulp3Error: duplicate key value violates unique constraint “core_remote_name_key”
DETAIL: Key (name)=(5f15602283d6e4509ee59ecd-31624b81-ab69-49a4-a101-dc06ca079e80) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in` <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1

Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”

Justin_Sherrill · April 28, 2021, 2:41pm

can you check ‘journalctl -u pulpcore-worker@*’ and look for a traceback with that error ’ duplicate key value violates unique constraint “core_remote_name_key”’ ?

senetm · April 28, 2021, 2:49pm

Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: django.db.utils.IntegrityError: duplicate key value violates unique constraint “core_remote_name_key”
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: DETAIL: Key (name)=(5f15602283d6e4509ee59ecd-31624b81-ab69-49a4-a101-dc06ca079e80) already exists.
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: Traceback (most recent call last):
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 554, in update_or_create
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: obj = self.select_for_update().get(**kwargs)
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/django/db/models/query.py”, line 408, in get
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: self.model._meta.object_name
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: pulp_deb.app.models.remote.AptRemote.DoesNotExist: AptRemote matching query does not exist.
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: During handling of the above exception, another exception occurred:
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: Traceback (most recent call last):
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/django/db/backends/utils.py”, line 84, in _execute
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: return self.cursor.execute(sql, params)
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint “core_remote_name_key”
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: DETAIL: Key (name)=(5f15602283d6e4509ee59ecd-31624b81-ab69-49a4-a101-dc06ca079e80) already exists.
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: The above exception was the direct cause of the following exception:
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: Traceback (most recent call last):
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/rq/worker.py”, line 936, in perform_job
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: rv = job.perform()
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 684, in perform
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: self._result = self._execute()
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/rq/job.py”, line 690, in _execute
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: return self.func(*self.args, **self.kwargs)
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: File “/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py”, line 80, in migrate_from_pulp2
Apr 28 16:25:42 foreman.com pulpcore-worker-2[1165]: migrate_importers(plan)

senetm · April 29, 2021, 12:15pm

The problem is that when we try to sync deb repository we get:

Justin_Sherrill · April 29, 2021, 3:16pm

You might verify that mongodb is running properly? maybe try restarting all the services.

As for your migration traceback, i chatted with the pulp folks and they found a bug and filed a fix: Fixed Debian importer migration on a re-run. by ipanova · Pull Request #359 · pulp/pulp-2to3-migration · GitHub

alipphart · April 30, 2021, 8:31am

Hi,

may I take over from @senetm …
I tried the patch with the same result

foreman-maintain content prepare

Running Prepare content for Pulp 3

Prepare content for Pulp 3:

Starting task.
2021-04-30 09:07:04 +0200: Initial Migration steps complete.Migration failed, You will want to investigate: https://foreman.com/foreman_tasks/tasks/c8617a75-288b-48ba-afce-1204ab7c9419
rake aborted!
ForemanTasks::TaskError: Task c8617a75-288b-48ba-afce-1204ab7c9419: Katello::Errors::Pulp3Error: duplicate key value violates unique constraint “core_remote_name_key”
DETAIL: Key (name)=(5f15602283d6e4509ee59ecd-31624b81-ab69-49a4-a101-dc06ca079e80) already exists.
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
[FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1

Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

[content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“content-prepare”

senetm · May 4, 2021, 10:35am

Hi,
i have try it again after changing the file and i am getting an other error now: