[ContentMigration] Katello::Errors::Pulp3Error

Support
1

Problem:
After a lot of problems with my last attempt to get to pulp3 & Katello 4.0 I reverted my servers to a snapshot before, making sure it’s all updated to the latest 3.18.2 and 2.3.3.

I just ran prepare the first time on this server and got an error I have never seen before:

# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Starting task.
2021-04-27 22:02:45 +0200: Migrating rpm content to Pulp 3 erratum 3004/55940
Migration failed, You will want to investigate: https://foreman.dkrz.de/foreman_tasks/tasks/dc5bd233-15e3-404a-a2e2-f5475ca0ae84
rake aborted!
ForemanTasks::TaskError: Task dc5bd233-15e3-404a-a2e2-f5475ca0ae84: Katello::Errors::Pulp3Error: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=25dc05be-79d5-41da-9e69-67959db7cb51>"
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

As far as I can see in the pulpcore database, the distribution tree 25dc05be is CentOS 8 Stream.

pulpcore=# select * from rpm_distributiontree where content_ptr_id = '25dc05be-79d5-41da-9e69-67959db7cb51';
            content_ptr_id            | header_version | release_name  | release_short | release_version | release_is_layered | base_product_name | base_product_short | base_product_version |  arch  | build_timestamp | instimage | mainimage | discnum | totaldiscs 
--------------------------------------+----------------+---------------+---------------+-----------------+--------------------+-------------------+--------------------+----------------------+--------+-----------------+-----------+-----------+---------+------------
 25dc05be-79d5-41da-9e69-67959db7cb51 | 1.2            | CentOS Stream | CentOS-Stream | 8               | f                  |                   |                    |                      | x86_64 |               0 |           |           |         |

Funny enough, in my previous attempts I didn’t get this error. Maybe it’s related to the latest updates to pulpcore like python3-pulp-2to3-migration-0.11.0-1.el7.noarch which wasn’t installed at the time of the snapshot, yet.

Expected outcome:
Successful migration preparation.

Foreman and Proxy versions:
Foreman 2.3.3, Katello 3.18.2

Distribution and version:
CentOS Linux release 7.9.2009 (Core)

2

Looking further in the database I think I also find the artifact which is supposed to be missing:

pulpcore=# select * from core_contentartifact where content_id = '25dc05be-79d5-41da-9e69-67959db7cb51';
               pulp_id                |         pulp_created          |       pulp_last_updated       | relative_path |             artifact_id              |              content_id              
--------------------------------------+-------------------------------+-------------------------------+---------------+--------------------------------------+--------------------------------------
 1104d2dd-fcc9-4e2a-9cde-8bea98a92f64 | 2021-04-27 22:02:33.746114+02 | 2021-04-27 22:02:33.746122+02 | .treeinfo     | d63f247b-9af9-4641-a05d-2f402ce5444a | 25dc05be-79d5-41da-9e69-67959db7cb51
(1 row)

Looking into core_artifact I find this:

pulpcore=# select * from core_artifact where pulp_id = 'd63f247b-9af9-4641-a05d-2f402ce5444a';
               pulp_id                |         pulp_created         |       pulp_last_updated       |                                    file                                    | size |               md5                |                   sha1                   |                          sha224        
                  |                              sha256                              |                                              sha384                                              |                                                              sha512                                                   
           
--------------------------------------+------------------------------+-------------------------------+----------------------------------------------------------------------------+------+----------------------------------+------------------------------------------+----------------------------------------
------------------+------------------------------------------------------------------+--------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------
-----------
 d63f247b-9af9-4641-a05d-2f402ce5444a | 2021-04-27 22:02:29.71136+02 | 2021-04-27 22:02:29.711375+02 | artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6 |  732 | 1f8d229b70e682bfc405ab60d5a1387e | 9313f2edc4821647d592e961b9b15581f7a9a12e | 8a7df771503e9bc456c1a95145e54f5f4f0418b
7a25ba5ae40b75755 | f03aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6 | 1b923849925f22769ca10a17d28c5b50557a42a59e28f6454b0ec016fd8126130630446aadfeb145a8ea710ee122e4a1 | 724ae01f4a21a29d3de2e29f55bb331fccc46e0c9da166cc4ab0e0cb4a90a33810e45aa94c06ee80d8ab36dd083a08cb9990c5431f87f03f6c04bd
8ea9709b05
(1 row)

Looking into the file /var/lib/pulp/media/artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6

I find it’s the file for the HighAvailability variant.

...
family = CentOS Stream
name = CentOS Stream 8
packagedir = Packages
platforms = x86_64
repository = .
timestamp = 1610136273
variant = HighAvailability
variants = HighAvailability
version = 8
...

So it’s seems to be all there.

What I did notice, though, was that exactly that file is not hard-linked:

-rw-rw-r--. 1 apache pulp 732 Jan 14 08:21 3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6

Everything else in that directory has exactly 2 hard links. Example:

[root@foreman f0]# ls -la 3[ab]*
-rw-rw-r--. 2 apache pulp 19112 May 12  2020 3a3843b53fc9eba490760caefdd79679b1043b67bfccc784e5d326395fc887
-rw-rw-r--. 1 apache pulp   732 Jan 14 08:21 3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6
-rw-rw-r--. 2 apache pulp 43048 Mar 17 08:39 3bd179cf0f002a94ba1385a9d2c46e0091ed3fcf8f10d694a51d0ff24788cf

O.K. If I look through the artifact directory I find a few more:

[root@foreman artifact]# find /var/lib/pulp/media/artifact -links 1 -ls
872899381 72976 -rw-rw-r--   1 apache   pulp     74726072 Jan 14 08:22 /var/lib/pulp/media/artifact/be/b553ba1155c0b306f37c89b4aed9b44793621d24ce1f00477543f68922eda8
872899380 9420 -rw-rw-r--   1 apache   pulp      9645192 Jan 14 08:22 /var/lib/pulp/media/artifact/dc/a943b26280cb8413ec08e99a0bb0cbe105c8165dac4e83e27bd9db27cb7341
806798021 10284 -rw-rw-r--   1 apache   pulp     10530816 Jan 14 08:22 /var/lib/pulp/media/artifact/44/d72a0a7c1420d76e4bd1b8ce9a7fb177ccb403eb7aedfec849ebb6edba7bf4
77580824    4 -rw-rw-r--   1 apache   pulp          732 Jan 14 08:21 /var/lib/pulp/media/artifact/f0/3aa0a6f3d4b0e9d20b6d758cd283bbc87cb37f467f9dd2ab30f7f1a4551fc6
738302237    4 -rw-rw-r--   1 apache   pulp          689 Jan 14 08:22 /var/lib/pulp/media/artifact/98/eed7837314fa47a136fe0e4579a359cb793850be2f92827375618c7ce8f258
806798023 720896 -rw-rw-r--   1 apache   pulp     738197504 Jan 14 08:22 /var/lib/pulp/media/artifact/2e/bba3435a15e9a9b4367c1275a24675ace1ad130d57e3fbaa9323d411f148a9
806798022 616996 -rw-rw-r--   1 apache   pulp     631803904 Jan 14 08:22 /var/lib/pulp/media/artifact/2e/af671535c70d18f184a80eb4f16efc1bd1c168a58e242e1f3c7ea27b0e65c7
872899356    4 -rw-rw-r--   1 apache   pulp         1454 Jan 14 08:22 /var/lib/pulp/media/artifact/d5/18deff99c8c71c3e6964839e4a96028383bd601151b8be958f457cacc25a47
269037350    4 -rw-rw-r--   1 apache   pulp          682 Jan 14 08:26 /var/lib/pulp/media/artifact/80/fb11576d48e218eb6808509f51f7c09cbf83a1326b81c3ffde3d9062de04ad

All from Jan 14 08:22. I guess, I did a prepare run at that time and it left a few things broken behind?

The four smaller files are the four .treeinfo files from the CentOS 8 Stream repos Base, HighAvailablity, AppStream, PowerTools. The other four seem to be CentOS-Stream-8-x86_64-dvd and a kernel, squashfs and a boot sector.

So it looks as if there are some old remnants from a CentOS 8 Stream in the pulp3 directories which are now confusing the migration?

Looking in the pulp2 directories I can find, for instance, the .treeinfo for the PowerTools in /var/lib/pulp/content/units/distribution/c7/9812563e2e65828ae874e92f7fac82d6852663d5d81866bf4d390049dfe6ab/.treeinfo which is hard linked to a different artifact /var/lib/pulp/media/artifact/40/262674b6844371b79d0f07f78c0dcbc43cfa06d8042ae55d3bb8bc4b81db4b

How can I clean this up?

3

O.K. Running a

foreman-rake katello:delete_orphaned_content RAILS_ENV=production

cleaned it up (taken from the katello cron job), i.e. removed all pulp3 artifacts in the file system and the postgres database. I guess as the migration never finished, everything was considered orphaned… It also cleared up the files with hard link count = 1.

I have noticed that the last time it ran as cron job the pulp3 cleanup ran for 32 hours and never finished. I guess I have rebooted at the time and didn’t even notice that there was a job running. So there must have been something wrong there, too.

Either way, after everything was clear out I ran prepare again. However, after 2.5 hours I get the same error again:

No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=614bba5c-ede6-492e-9e89-ac1905b54584>"

Tracing the .treeinfo file again like I did before it’s the same file again for the HighAvailability variant (even though it’s the current version now).

So basically, I am back to square one.

4

@iballou

Do you have any insights to this one?

5

Looking through /var/log/messages, these are the messages five seconds before the traceback:

Apr 29 14:35:52 foreman pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [551c2278] Sync complete.
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[9a2d5cab-a68b-417b-83f2-756a3b69769a]
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [551c2278] Task pulp.server.managers.repo.sync.sync[551c2278-6617-4f8f-8db4-2d5b32f313ea] succeeded in 158.058656971s: <pulp.server.async.tasks.TaskResult object at 0x7facf339eb10>
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [a8071a6a] Task pulp.server.async.tasks._release_resource[a8071a6a-bb76-40ad-89f9-9020de70bbe2] succeeded in 0.00174587298534s: None
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[b07efeb5-8f28-4875-b350-0d8d66d7cb7f]
Apr 29 14:35:52 foreman pulp: celery.app.trace:INFO: [9a2d5cab] Task pulp.server.async.tasks._queue_reserved_task[9a2d5cab-a68b-417b-83f2-756a3b69769a] succeeded in 0.0175154710014s: None
Apr 29 14:35:52 foreman pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[29f40a9c-3aef-46a4-bb3b-cc92a333cfe7]
Apr 29 14:35:52 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp/abstract_async_task_group.rb:81: warning: Runcible::Response#respond_to?(:to_hash) uses the deprecated method signature, which takes one parameter
Apr 29 14:35:52 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/response.rb:10: warning: respond_to? is defined here
Apr 29 14:35:54 foreman pulp: pulp.plugins.pulp_rpm.plugins.distributors.yum.publish:WARNING: [b07efeb5] No distribution files found for unit distribution: ks-CentOS Linux-AppStream-8-x86_64-CentOS Linux-AppStream-8-x86_64
Apr 29 14:35:56 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/app/lib/actions/pulp/abstract_async_task_group.rb:81: warning: Runcible::Response#respond_to?(:to_hash) uses the deprecated method signature, which takes one parameter
Apr 29 14:35:56 foreman dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/response.rb:10: warning: respond_to? is defined here
Apr 29 14:35:57 foreman pulpcore-worker-4: /usr/lib/python3.6/site-packages/django/db/models/fields/__init__.py:1427: RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2021-04-28 10:36:16) while time zone support is active.
Apr 29 14:35:57 foreman pulpcore-worker-4: RuntimeWarning)
Apr 29 14:35:57 foreman pulpcore-worker-4: pulp: rq.worker:ERROR: Traceback (most recent call last):
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
Apr 29 14:35:57 foreman pulpcore-worker-4: rv = job.perform()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
Apr 29 14:35:57 foreman pulpcore-worker-4: self._result = self._execute()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
Apr 29 14:35:57 foreman pulpcore-worker-4: return self.func(*self.args, **self.kwargs)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py", line 81, in migrate_from_pulp2
Apr 29 14:35:57 foreman pulpcore-worker-4: migrate_content(plan, skip_corrupted=skip_corrupted)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 47, in migrate_content
Apr 29 14:35:57 foreman pulpcore-worker-4: plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/migrator.py", line 145, in migrate_content_to_pulp3
Apr 29 14:35:57 foreman pulpcore-worker-4: loop.run_until_complete(dm.create())
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete
Apr 29 14:35:57 foreman pulpcore-worker-4: return future.result()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py", line 89, in create
Apr 29 14:35:57 foreman pulpcore-worker-4: await pipeline
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline
Apr 29 14:35:57 foreman pulpcore-worker-4: await asyncio.gather(*futures)
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in __call__
Apr 29 14:35:57 foreman pulpcore-worker-4: await self.run()
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 244, in run
Apr 29 14:35:57 foreman pulpcore-worker-4: RemoteArtifact.objects.bulk_get_or_create(self._needed_remote_artifacts(batch))
Apr 29 14:35:57 foreman pulpcore-worker-4: File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/artifact_stages.py", line 301, in _needed_remote_artifacts
Apr 29 14:35:57 foreman pulpcore-worker-4: msg.format(rp=content_artifact.relative_path, c=d_content.content)
Apr 29 14:35:57 foreman pulpcore-worker-4: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=614bba5c-ede6-492e-9e89-ac1905b54584>"

I don’t know where this RuntimeWarning: DateTimeField Pulp2RepoContent.pulp2_created received a naive datetime (2021-04-28 10:36:16) while time zone support is active. message comes from.

6

@gvde @caseybea Here’s something that might help:

foreman-rake katello:pulp3_migration_reset
7

Doesn’t work either:

# foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.
Content Migration Reset failed, You will want to investigate: https://foreman.dkrz.de/foreman_tasks/tasks/83f35292-04e9-4823-957e-f85ee2b59f55
rake aborted!
ForemanTasks::TaskError: Task 83f35292-04e9-4823-957e-f85ee2b59f55: ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column "migrated_pulp3_href" of relation "katello_debs" does not exist
LINE 1: UPDATE "katello_debs" SET "migrated_pulp3_href" = $1
                                  ^
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)
8

@gvde ah that bug, that’s slated for 3.18.3. Are you using debian content? If not, you can disable it and get it out of the way with foreman-installer --katello-use-pulp-2-for-deb=false --katello-enable-deb=false.

If you are using debian content I can get you a patch.

By the way, the Pulp team got back to be and it looks like this bug is being investigated: Issue #8566: Content Migration to Pulp 3 with Katello fails (similar to #8377) - Migration Plugin - Pulp

9

I realized from the bug I linked that this might not have anything to do with old data from previous migrations. I think it’s still worth trying the reset, but we might need to keep a close eye on that bug. It would be helpful if you both (@gvde and @caseybea) could comment on that bug saying you’re also hitting it.

10

Well, I did a orphan cleanup before and it removed all artifacts and also cleaned out the postgres database. After that, I still get the same error, but the uuid of that distribution tree changed. I think it more related to the multiple .treeinfo for the different CentOS 8 variants. But the weird thing is I cannot find anything having a timestamp of 2021-04-28 10:36:16 in either the mongodb nor the postgres database.

I am disabling deb content and run the migration reset later.

11

OK, to provide info as requested for my situation (same error), I start with this error:

Apr 29 15:36:25 katello pulpcore-worker-8: ValueError: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=26685499-74b8-4337-adc6-5bb2d5325ae3>"

I then dig into the database and see:

pulpcore=# select * from core_contentartifact where content_id = '26685499-74b8-4337-adc6-5bb2d5325ae3';
               pulp_id                |         pulp_created         |       pulp_last_updated       | relative_path |             artifact_id              |              content_id
--------------------------------------+------------------------------+-------------------------------+---------------+--------------------------------------+--------------------------------------
 eb89b508-ba0b-4019-bb94-ab0cefbdd077 | 2021-04-29 15:36:13.54309-05 | 2021-04-29 15:36:13.543096-05 | .treeinfo     | 10e9a0f9-cf92-48ec-b3cc-e809a8d770bc | 26685499-74b8-4337-adc6-5bb2d5325ae3
(1 row)

and with that information, I look deeper:

pulpcore=# select * from core_artifact where pulp_id = '10e9a0f9-cf92-48ec-b3cc-e809a8d770bc';
               pulp_id                |         pulp_created          |       pulp_last_updated       |                                    file                                    | size |               md5                |
      sha1                   |                          sha224                          |                              sha256                              |                                              sha384
                   |                                                              sha512
--------------------------------------+-------------------------------+-------------------------------+----------------------------------------------------------------------------+------+----------------------------------+-------------
-----------------------------+----------------------------------------------------------+------------------------------------------------------------------+-------------------------------------------------------------------------------
-------------------+----------------------------------------------------------------------------------------------------------------------------------
 10e9a0f9-cf92-48ec-b3cc-e809a8d770bc | 2021-04-29 15:36:08.609693-05 | 2021-04-29 15:36:08.609704-05 | artifact/1b/ca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba |  679 | f7dd9a25a7ad64790d869b141e398eb8 | 709db1eb00fc
a3edab2aa48470e212311b604cd5 | e1ed4e3c4cb342a72a0c6479a80c42e209b0fb145f532afd3cf02ce4 | 1bca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba | 38658766a6656e35f519fcaa61c05ca089c5c2f40073d204fe71633dfde608cb92a04845e0a3e5
9c4cedfa3f0f2f7ef3 | 9308c3655af7f48cde75dbb96037f26b198704206dfbd6dd699137e5e5a1b9a83bb9e572c12d395b5a0c8f151033ffb962859ddbb862b9ec6b91310722c05d0b
(1 row)

Ad looking at the file, I now get the OS info-- this is from Centos8:

[root@katello log]# more /var/lib/pulp/media/artifact/1b/ca0dfb154119313656ef088332125aadf3c35c0803952e8a4bc10cd2789cba

[general]
; WARNING.0 = This section provides compatibility with pre-productmd treeinfos.
; WARNING.1 = Read productmd documentation for details about new format.
arch = x86_64
family = CentOS Linux
name = CentOS Linux 8
packagedir = Packages
platforms = x86_64
repository = .
timestamp = 1605735570
variant = PowerTools
variants = PowerTools
version = 8

[header]
type = productmd.treeinfo
version = 1.2

[release]
name = CentOS Linux
short = CentOS
version = 8

[tree]
arch = x86_64
build_timestamp = 1605735570
platforms = x86_64
variants = PowerTools

[variant-PowerTools]
id = PowerTools
name = PowerTools
packages = Packages
repository = .
type = variant
uid = PowerTools
12

@iballou I just commented on that pulp3 bug I am having the exact issue.

@gvde did as well :slight_smile:

13

Migration reset does not help:

# foreman-maintain content prepare
Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3: 
Starting task.
2021-04-30 19:21:03 +0200: Migrating rpm content to Pulp 3 erratum 2647/45797Migration failed, You will want to investigate: https://foreman.example.com/foreman_tasks/tasks/6b4493c0-6847-4e7e-bfac-b65d7a093792
rake aborted!
ForemanTasks::TaskError: Task 6b4493c0-6847-4e7e-bfac-b65d7a093792: Katello::Errors::Pulp3Error: No declared artifact with relative path ".treeinfo" for content "<DistributionTree: pk=9193b893-4989-4b49-9532-f3afbb750424>"
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.
14

Alright, thanks for the info @caseybea and @gvde. Now that the bug is getting more attention it shouldn’t be too long before it’s resolved. Developers might ask some more questions on the Pulp bug.

15

It seems there is no progress on this issue. I would try to troubleshoot myself if I knew more python and if it wasn’t in a task which takes hours until the error appears, so each with each additional debug log in the code I have to wait to see it.

Is there a debugger or something how I could set a breakpoint in the code? I hoped I might get an idea what’s going wrong if I was able to check what’s inside those artifacts arrays…

16

Debugging Pulp isn’t something I’ve tried yet myself. A remote debugger might do the job, like remote-pdb perhaps?

I’ll try to get some updates on the bug.

17

For anyone still watching this issue, there is a patch on the Pulp 3 issue that should help: Issue #8566: Content Migration to Pulp 3 with Katello fails (similar to #8377) - Migration Plugin - Pulp

1 Like
18

Just ran into this again: The patch doesn’t work for AlmaLinux because with AlmaLinux the build_timestamp isn’t unique between variants either:

AlmaLinux/8.4/AppStream/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/AppStream/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/kickstart/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/BaseOS/x86_64/os/.treeinfo:build_timestamp = 1622014553
AlmaLinux/8.4/HighAvailability/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/HighAvailability/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/PowerTools/x86_64/os/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/kickstart/.treeinfo:build_timestamp = 1622014558
AlmaLinux/8.4/extras/x86_64/os/.treeinfo:build_timestamp = 1622014558
19

Looks like they changed build_timestamp in their repos. You can check it

20

Yes. They have increased the timestamps for the os repos. Looks like a manual change to me. I hope it won’t happen again for 8.5… The kickstart repos are still at the old values.