[ContentMigration] PulpcoreClient::ApiError: the server returns an error

Problem:
foreman-maintain content migration-stats:

Retrieve Pulp 2 to Pulp 3 migration statistics:
============Migration Summary================
Migrated/Total RPMs: 0/153057
Migrated/Total errata: 0/112427
Migrated/Total repositories: 0/255

foreman-maintain content prepare:

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-03-30 16:16:17 +0000: Content migration starting. Migration failed, You will want to investigate: https://foreman.isbank.is/foreman_tasks/tasks/eabe3507-91a5-451d-8c3f-9f0138870648
rake aborted!
ForemanTasks::TaskError: Task eabe3507-91a5-451d-8c3f-9f0138870648: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 30 Mar 2021 16:16:18 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Foreman and Proxy versions:
Katello 3.18.2 - just updated

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7

Other relevant data:

did you already start a migration previously? could it still be running? If you check Monitor > tasks and search for ‘label ~ Migration’ you should see all the instances, check if one is still running.

I have similiar situation… Tasks queue is clear. I have restarted services. Nothing helps.
Somewhere are locks, but I can not find them.

foreman ~ # foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.  
Content Migration Reset failed, You will want to investigate: 
ForemanTasks::TaskError: Task 305326fc-21e3-43b5-9dc0-d2b832fd144d: 
Pulp2to3MigrationClient::ApiError: Error message: the server returns an error 
HTTP status code: 400
Response headers: {"date"=>"Thu, 01 Apr 2021 07:46:45 GMT", "server"=>"gunicorn/20.0.4", 
"content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame- 
options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 ", " 
"connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello- 
3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)

If i remember correctly, I ran the first iteration of the migration in 3.16 (or whenever it was introduced into the upgrade process), but no current migration task is running.

Now there’s a different error:

ForemanTasks::TaskError: Task df844f21-7bc5-40ce-96e3-73a02054d273: PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {"date"=>"Wed, 07 Apr 2021 12:37:50 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: {"detail":"You do not have permission to perform this action."}

And the foreman task shows that its happening on Actions::Pulp3::ImportMigration

I actually tried the migration the very first time today. The first run ended with:

Response body: {"detail":"You do not have permission to perform this action."}

Now any run ends with

Response body: ["Only one migration plan can run or be reset at a time"]

No task is active though…

I had a host of other problems that i seem to have finally fixed, among them was a disk issue. Not that a disk was full, but that qdrouterd default behaviour is to stop functioning properly if disk usage is >90%, after fixing that my Foreman instance appears to finally be back to stable and this being the last thing that doesn’t work.
I tested running foreman-maintain content switchover and watching logs, i found this error:

apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/

Hopefully someone can tell us whats going on and how to fix it :slight_smile:

Just for completeness sake, here’s the entire log output from pulpcore-api for the run.

apr 08 14:24:12 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:12 +0000] “GET /pulp/api/v3/status HTTP/1.1” 301 0 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status HTTP/1.1” 301 0 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:14 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:14 +0000] “POST /pulp/api/v3/migration-plans/ HTTP/1.1” 201 53897 “-” “OpenAPI-Generator/0.7.0/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “POST /pulp/api/v3/migration-plans/0c3ce164-b1fb-4cd9-945e-47ab6f95fa3c/run/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/0.7.0/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “GET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “GET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”

Found in another post ( [ContentMigration] foreman-maintain content prepare failed - #5 by touchardv ).
Ran command
sudo su - postgres -c "psql -d pulpcore -c 'UPDATE auth_user SET is_superuser = true WHERE id =1;'"
and retried, that seems to have worked, so that seems to be the permission error at least.

Tagging @Justin_Sherrill to make sure he sees the solution, if others have the same issue. :smiley:

@ekohl suggested using the pulpcore shell to make the change in this topic: Pulp3 content migration - pulpcore admin is_superuser

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell
from django.contrib.auth import get_user_model
user_model = get_user_model()
user = user_model.objects.get(username='admin')
user.is_superuser = True
user.save()
1 Like

This is probably the more elegant solution, instead of going directly into the database yourself - marking this as the better solution to the permission error.

I was so close…
Now i’m hitting another issue it seems:

OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn’t compiled with zchunk support

Found a but report for this issue: Issue #8400: Migration fails if there is .zck metadata in pulp2 - Migration Plugin - Pulp (plan.io)

Guess i’ll have to wait until that is resolved to finish my migration - but at least we figured out the permission error :slight_smile:

I had the exact same errors as you did birkirf, both the preceding issue and the last zchunk one. That issue was resolved for me once we upgraded python3-pulp-2to3-migration to 0.11.0-1.el7 from 0.10

I spoke too soon, another run, that went on for hours this time - stopped with the same kind of error.

Hitting exactly the same issue (.zck metadata migration failure). Any known workarounds by now? Bug seems to be 2 months old.

1 Like

Unfortunately i’ve not found any workarounds, but i have seen some threads that imply that finding the offending repos and removing them, finish upgrading and recreating the repos would work, but in my Production system i’m simply not going to do that, so haven’t tried it.
I can wait on 3.18.x until they find a fix for a few more months if thats what it takes.

Hi,
Juste as reference:

Anyone heard anything regarding the fix for zck metadata? Ticket was finished 2 months ago now, but problem still exists on latest 3.18.3 installation - Server is fully updated as of 2 minutes ago, but still getting the error

Here’s the full error from journal

júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp: rq.worker:ERROR: Traceback (most recent call last):
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: rv = job.perform()
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: self._result = self._execute()
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: return self.func(*self.args, **self.kwargs)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: migrated_repo.pulp3_repository_version
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp2dist, repo_version)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: metadata_signing_service=metadata_signing_service,
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: record.fill(checksum_type)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn't compiled with zchunk support
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: Traceback (most recent call last):
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: rv = job.perform()
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: self._result = self._execute()
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: return self.func(*self.args, **self.kwargs)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: migrated_repo.pulp3_repository_version
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp2dist, repo_version)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: metadata_signing_service=metadata_signing_service,
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: record.fill(checksum_type)
júl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn't compiled with zchunk support

You seem to have a different createrepo_c installed. Please check:

# which createrepo_c
/bin/createrepo_c
# rpm -qf /bin/createrepo_c
createrepo_c-0.17.1-1.el7.x86_64
# yum list createrepo_c-0.17.1-1.el7.x86_64
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
              : tracer_upload
Loading mirror speeds from cached hostfile
Installed Packages
createrepo_c.x86_64                                0.17.1-1.el7                                @ORG_katello_pulpcore_3_7_el7_x86_64
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager