[ContentMigration] Katello::Errors::Pulp3Error

Problem:
After a lot of problems with my last attempt to get to pulp3 & Katello 4.0 I reverted my servers to a snapshot before, making sure it’s all updated to the latest 3.18.2 and 2.3.3.

I just ran prepare the first time on this server and got an error I have never seen before:

# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Starting task.
2021-04-27 22:02:45 +0200: Migrating rpm content to Pulp 3 erratum 3004/55940
Migration failed, You will want to investigate: https://foreman.dkrz.de/foreman_tasks/tasks/dc5bd233-15e3-404a-a2e2-f5475ca0ae84
rake aborted!
ForemanTasks::TaskError: Task dc5bd233-15e3-404a-a2e2-f5475ca0ae84: Katello::Errors::Pulp3Error: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=25dc05be-79d5-41da-9e69-67959db7cb51>"
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

As far as I can see in the pulpcore database, the distribution tree 25dc05be is CentOS 8 Stream.

pulpcore=# select * from rpm_distributiontree where content_ptr_id = '25dc05be-79d5-41da-9e69-67959db7cb51';
            content_ptr_id            | header_version | release_name  | release_short | release_version | release_is_layered | base_product_name | base_product_short | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+---------------+---------------+-----------------+--------------------+-------------------+--------------------+----------------------+--------+-----------------+-----------+-----------+---------+------------
 25dc05be-79d5-41da-9e69-67959db7cb51 | 1.2            | CentOS Stream | CentOS-Stream | 8               | f                  |                   |                    |                      | x86_64 |               0 |           |           |         |           

Funny enough, in my previous attempts I didn’t get this error. Maybe it’s related to the latest updates to pulpcore like python3-pulp-2to3-migration-0.11.0-1.el7.noarch which wasn’t installed at the time of the snapshot, yet.

Expected outcome:
Successful migration preparation.

Foreman and Proxy versions:
Foreman 2.3.3, Katello 3.18.2

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

Looking further in the database I think I also find the artifact which is supposed to be missing:

pulpcore=# select * from core_contentartifact where content_id = '25dc05be-79d5-41da-9e69-67959db7cb51';
               pulp_id                |         pulp_created          |       pulp_last_updated       | relative_path |             artifact_id              |              content_id              
--------------------------------------+-------------------------------+-------------------------------+---------------+--------------------------------------+--------------------------------------
 1104d2dd-fcc9-4e2a-9cde-8bea98a92f64 | 2021-04-27 22:02:33.746114+02 | 2021-04-27 22:02:33.746122+02 | .treeinfo     | d63f247b-9af9-4641-a05d-2f402ce5444a | 25dc05be-79d5-41da-9e69-67959db7cb51
(1 row)

Looking into core_artifact I find this:

pulpcore=# select * from core_artifact where pulp_id = 'd63f247b-9af9-4641-a05d-2f402ce5444a';
               pulp_id                |         pulp_created         |       pulp_last_updated       |                                    file                                    | size |               md5                |                   sha1                   |                          sha224        
                  |                              sha256                              |                                              sha384                                              |                                                              sha512                                                   
           
--------------------------------------+------------------------------+-------------------------------+----------------------------------------------------------------------------+------+----------------------------------+------------------------------------------+----------------------------------------
------------------+------------------------------------------------------------------+--------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------
-----------
 d63f247b-9af9-4641-a05d-2f402ce5444a | 2021-04-27 22:02:29.71136+02 | 2021-04-27 22:02:29.711375+02 | artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6 |  732 | 1f8d229b70e682bfc405ab60d5a1387e | 9313f2edc4821647d592e961b9b15581f7a9a12e | 8a7df771503e9bc456c1a95145e54f5f4f0418b
7a25ba5ae40b75755 | f03aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6 | 1b923849925f22769ca10a17d28c5b50557a42a59e28f6454b0ec016fd8126130630446aadfeb145a8ea710ee122e4a1 | 724ae01f4a21a29d3de2e29f55bb331fccc46e0c9da166cc4ab0e0cb4a90a33810e45aa94c06ee80d8ab36dd083a08cb9990c5431f87f03f6c04bd
8ea9709b05
(1 row)

Looking into the file /var/lib/pulp/media/artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6

I find it’s the file for the HighAvailability variant.

...
family = CentOS Stream
name = CentOS Stream 8
packagedir = Packages
platforms = x86_64
repository = .
timestamp = 1610136273
variant = HighAvailability
variants = HighAvailability
version = 8
...

So it’s seems to be all there.

What I did notice, though, was that exactly that file is not hard-linked:

-rw-rw-r--. 1 apache pulp 732 Jan 14 08:21 3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6

Everything else in that directory has exactly 2 hard links. Example:

[root@foreman f0]# ls -la 3[ab]*
-rw-rw-r--. 2 apache pulp 19112 May 12  2020 3a3843b53fc9eba490760caefdd79679b1043b67bfccc784e5d326395fc887
-rw-rw-r--. 1 apache pulp   732 Jan 14 08:21 3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6
-rw-rw-r--. 2 apache pulp 43048 Mar 17 08:39 3bd179cf0f002a94ba1385a9d2c46e0091ed3fcf8f10d694a51d0ff24788cf

O.K. If I look through the artifact directory I find a few more:

[root@foreman artifact]# find /var/lib/pulp/media/artifact -links 1 -ls
872899381 72976 -rw-rw-r--   1 apache   pulp     74726072 Jan 14 08:22 /var/lib/pulp/media/artifact/be/b553ba1155c0b306f37c89b4aed9b44793621d24ce1f00477543f68922eda8
872899380 9420 -rw-rw-r--   1 apache   pulp      9645192 Jan 14 08:22 /var/lib/pulp/media/artifact/dc/a943b26280cb8413ec08e99a0bb0cbe105c8165dac4e83e27bd9db27cb7341
806798021 10284 -rw-rw-r--   1 apache   pulp     10530816 Jan 14 08:22 /var/lib/pulp/media/artifact/44/d72a0a7c1420d76e4bd1b8ce9a7fb177ccb403eb7aedfec849ebb6edba7bf4
77580824    4 -rw-rw-r--   1 apache   pulp          732 Jan 14 08:21 /var/lib/pulp/media/artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6
738302237    4 -rw-rw-r--   1 apache   pulp          689 Jan 14 08:22 /var/lib/pulp/media/artifact/98/eed7837314fa47a136fe0e4579a359cb793850be2f92827375618c7ce8f258
806798023 720896 -rw-rw-r--   1 apache   pulp     738197504 Jan 14 08:22 /var/lib/pulp/media/artifact/2e/bba3435a15e9a9b4367c1275a24675ace1ad130d57e3fbaa9323d411f148a9
806798022 616996 -rw-rw-r--   1 apache   pulp     631803904 Jan 14 08:22 /var/lib/pulp/media/artifact/2e/af671535c70d18f184a80eb4f16efc1bd1c168a58e242e1f3c7ea27b0e65c7
872899356    4 -rw-rw-r--   1 apache   pulp         1454 Jan 14 08:22 /var/lib/pulp/media/artifact/d5/18deff99c8c71c3e6964839e4a96028383bd601151b8be958f457cacc25a47
269037350    4 -rw-rw-r--   1 apache   pulp          682 Jan 14 08:26 /var/lib/pulp/media/artifact/80/fb11576d48e218eb6808509f51f7c09cbf83a1326b81c3ffde3d9062de04ad

All from Jan 14 08:22. I guess, I did a prepare run at that time and it left a few things broken behind?

The four smaller files are the four .treeinfo files from the CentOS 8 Stream repos Base, HighAvailablity, AppStream, PowerTools. The other four seem to be CentOS-Stream-8-x86_64-dvd and a kernel, squashfs and a boot sector.

So it looks as if there are some old remnants from a CentOS 8 Stream in the pulp3 directories which are now confusing the migration?

Looking in the pulp2 directories I can find, for instance, the .treeinfo for the PowerTools in /var/lib/pulp/content/units/distribution/c7/9812563e2e65828ae874e92f7fac82d6852663d5d81866bf4d390049dfe6ab/.treeinfo which is hard linked to a different artifact /var/lib/pulp/media/artifact/40/262674b6844371b79d0f07f78c0dcbc43cfa06d8042ae55d3bb8bc4b81db4b

How can I clean this up?

O.K. Running a

foreman-rake katello:delete_orphaned_content RAILS_ENV=production

cleaned it up (taken from the katello cron job), i.e. removed all pulp3 artifacts in the file system and the postgres database. I guess as the migration never finished, everything was considered orphaned… It also cleared up the files with hard link count = 1.

I have noticed that the last time it ran as cron job the pulp3 cleanup ran for 32 hours and never finished. I guess I have rebooted at the time and didn’t even notice that there was a job running. So there must have been something wrong there, too.

Either way, after everything was clear out I ran prepare again. However, after 2.5 hours I get the same error again:

No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=614bba5c-ede6-492e-9e89-ac1905b54584>"

Tracing the .treeinfo file again like I did before it’s the same file again for the HighAvailability variant (even though it’s the current version now).

So basically, I am back to square one.

@iballou

Do you have any insights to this one?

Looking through /var/log/messages, these are the messages five seconds before the traceback:

Apr 29 14:35:52 foreman pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [551c2278] Sync complete.
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[9a2d5cab-a68b-417b-83f2-756a3b69769a]
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [551c2278] Task pulp.server.managers.repo.sync.sync[551c2278-6617-4f8f-8db4-2d5b32f313ea] succeeded in 158.058656971s: <pulp.server.async.tasks.TaskResult object at 0x7facf339eb10>
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [a8071a6a] Task pulp.server.async.tasks._release_resource[a8071a6a-bb76-40ad-89f9-9020de70bbe2] succeeded in 0.00174587298534s: None
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[b07efeb5-8f28-4875-b350-0d8d66d7cb7f]
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [9a2d5cab] Task pulp.server.async.tasks._queue_reserved_task[9a2d5cab-a68b-417b-83f2-756a3b69769a] succeeded in 0.0175154710014s: None
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[29f40a9c-3aef-46a4-bb3b-cc92a333cfe7]
Apr 29 14:35:52 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp/abstract_async_task_group.rb:81: warning: Runcible::Response#respond_to?(:to_hash) uses the deprecated method signature, which takes one parameter
Apr 29 14:35:52 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/response.rb:10: warning: respond_to? is defined here
Apr 29 14:35:54 foreman pulp: pulp.plugins.pulp_rpm.plugins.distributors.yum.publish:WARNING: [b07efeb5] No distribution files found for unit distribution: ks-CentOS Linux-AppStream-8-x86_64-CentOS Linux-AppStream-8-x86_64
Apr 29 14:35:56 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp/abstract_async_task_group.rb:81: warning: Runcible::Response#respond_to?(:to_hash) uses the deprecated method signature, which takes one parameter
Apr 29 14:35:56 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/response.rb:10: warning: respond_to? is defined here
Apr 29 14:35:57 foreman pulpcore-worker-4: /usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1427: RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2021-04-28 10:36:16) while time zone support is active.
Apr 29 14:35:57 foreman pulpcore-worker-4: RuntimeWarning)
Apr 29 14:35:57 foreman pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 29 14:35:57 foreman pulpcore-worker-4: rv = job.perform()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 29 14:35:57 foreman pulpcore-worker-4: self._result = self._execute()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 29 14:35:57 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Apr 29 14:35:57 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Apr 29 14:35:57 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Apr 29 14:35:57 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 29 14:35:57 foreman pulpcore-worker-4: return future.result()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Apr 29 14:35:57 foreman pulpcore-worker-4: await pipeline
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 29 14:35:57 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 29 14:35:57 foreman pulpcore-worker-4: await self.run()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Apr 29 14:35:57 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 301, in _needed_remote_artifacts
Apr 29 14:35:57 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Apr 29 14:35:57 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=614bba5c-ede6-492e-9e89-ac1905b54584>"

I don’t know where this RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2021-04-28 10:36:16) while time zone support is active. message comes from.

@gvde @caseybea Here’s something that might help:

foreman-rake katello:pulp3_migration_reset

Doesn’t work either:

# foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.
Content Migration Reset failed, You will want to investigate: https://foreman.dkrz.de/foreman_tasks/tasks/83f35292-04e9-4823-957e-f85ee2b59f55
rake aborted!
ForemanTasks::TaskError: Task 83f35292-04e9-4823-957e-f85ee2b59f55: ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column "migrated_pulp3_href" of relation "katello_debs" does not exist
LINE 1: UPDATE "katello_debs" SET "migrated_pulp3_href" = $1
                                  ^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)

@gvde ah that bug, that’s slated for 3.18.3. Are you using debian content? If not, you can disable it and get it out of the way with foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false.

If you are using debian content I can get you a patch.

By the way, the Pulp team got back to be and it looks like this bug is being investigated: Issue #8566: Content Migration to Pulp 3 with Katello fails (similar to #8377) - Migration Plugin - Pulp

I realized from the bug I linked that this might not have anything to do with old data from previous migrations. I think it’s still worth trying the reset, but we might need to keep a close eye on that bug. It would be helpful if you both (@gvde and @caseybea) could comment on that bug saying you’re also hitting it.

Well, I did a orphan cleanup before and it removed all artifacts and also cleaned out the postgres database. After that, I still get the same error, but the uuid of that distribution tree changed. I think it more related to the multiple .treeinfo for the different CentOS 8 variants. But the weird thing is I cannot find anything having a timestamp of 2021-04-28 10:36:16 in either the mongodb nor the postgres database.

I am disabling deb content and run the migration reset later.

OK, to provide info as requested for my situation (same error), I start with this error:

Apr 29 15:36:25 katello pulpcore-worker-8: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=26685499-74b8-4337-adc6-5bb2d5325ae3>"

I then dig into the database and see:

pulpcore=# select * from core_contentartifact where content_id = '26685499-74b8-4337-adc6-5bb2d5325ae3';
               pulp_id                |         pulp_created         |       pulp_last_updated       | relative_path |             artifact_id              |              content_id
--------------------------------------+------------------------------+-------------------------------+---------------+--------------------------------------+--------------------------------------
 eb89b508-ba0b-4019-bb94-ab0cefbdd077 | 2021-04-29 15:36:13.54309-05 | 2021-04-29 15:36:13.543096-05 | .treeinfo     | 10e9a0f9-cf92-48ec-b3cc-e809a8d770bc | 26685499-74b8-4337-adc6-5bb2d5325ae3
(1 row)

and with that information, I look deeper:

pulpcore=# select * from core_artifact where pulp_id = '10e9a0f9-cf92-48ec-b3cc-e809a8d770bc';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                                    file                                    | size |               md5                |
      sha1                   |                          sha224                          |                              sha256                              |                                              sha384
                   |                                                              sha512
--------------------------------------+-------------------------------+-------------------------------+----------------------------------------------------------------------------+------+----------------------------------+-------------
-----------------------------+----------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------------------------------------------
-------------------+----------------------------------------------------------------------------------------------------------------------------------
 10e9a0f9-cf92-48ec-b3cc-e809a8d770bc | 2021-04-29 15:36:08.609693-05 | 2021-04-29 15:36:08.609704-05 | artifact/1b/ca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba |  679 | f7dd9a25a7ad64790d869b141e398eb8 | 709db1eb00fc
a3edab2aa48470e212311b604cd5 | e1ed4e3c4cb342a72a0c6479a80c42e209b0fb145f532afd3cf02ce4 | 1bca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba | 38658766a6656e35f519fcaa61c05ca089c5c2f40073d204fe71633dfde608cb92a04845e0a3e5
9c4cedfa3f0f2f7ef3 | 9308c3655af7f48cde75dbb96037f26b198704206dfbd6dd699137e5e5a1b9a83bb9e572c12d395b5a0c8f151033ffb962859ddbb862b9ec6b91310722c05d0b
(1 row)

Ad looking at the file, I now get the OS info-- this is from Centos8:

[root@katello log]# more /var/lib/pulp/media/artifact/1b/ca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = CentOS Linux
name = CentOS Linux 8
packagedir = Packages
platforms = x86_64
repository = .
timestamp = 1605735570
variant = PowerTools
variants = PowerTools
version = 8

[header]
type = productmd.treeinfo
version = 1.2

[release]
name = CentOS Linux
short = CentOS
version = 8

[tree]
arch = x86_64
build_timestamp = 1605735570
platforms = x86_64
variants = PowerTools

[variant-PowerTools]
id = PowerTools
name = PowerTools
packages = Packages
repository = .
type = variant
uid = PowerTools

@iballou I just commented on that pulp3 bug I am having the exact issue.

@gvde did as well :slight_smile:

Migration reset does not help:

# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Starting task.
2021-04-30 19:21:03 +0200: Migrating rpm content to Pulp 3 erratum 2647/45797Migration failed, You will want to investigate: https://foreman.example.com/foreman_tasks/tasks/6b4493c0-6847-4e7e-bfac-b65d7a093792
rake aborted!
ForemanTasks::TaskError: Task 6b4493c0-6847-4e7e-bfac-b65d7a093792: Katello::Errors::Pulp3Error: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=9193b893-4989-4b49-9532-f3afbb750424>"
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

Alright, thanks for the info @caseybea and @gvde. Now that the bug is getting more attention it shouldn’t be too long before it’s resolved. Developers might ask some more questions on the Pulp bug.