[ContentMigration] PulpcoreClient::ApiError: the server returns an error

Problem:
foreman-maintain content migration-stats:

Retrieve Pulp 2 to Pulp 3 migration statistics:
============Migration Summary================
Migrated/Total RPMs: 0/153057
Migrated/Total errata: 0/112427
Migrated/Total repositories: 0/255

foreman-maintain content prepare:

Running Prepare content for Pulp 3
================================================================================
Prepare content for Pulp 3:
Starting task.
2021-03-30 16:16:17 +0000: Content migration starting. Migration failed, You will want to investigate: https://foreman.isbank.is/foreman_tasks/tasks/eabe3507-91a5-451d-8c3f-9f0138870648
rake aborted!
ForemanTasks::TaskError: Task eabe3507-91a5-451d-8c3f-9f0138870648: Pulp2to3MigrationClient::ApiError: Error message: the server returns an error
HTTP status code: 400
Response headers: {"date"=>"Tue, 30 Mar 2021 16:16:18 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.2.1/lib/katello/tasks/pulp3_migration.rake:33:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration
(See full trace by running task with --trace)
                                                                      [FAIL]
Failed executing foreman-rake katello:pulp3_migration, exit status 1
--------------------------------------------------------------------------------
Scenario [Prepare content for Pulp 3] failed.

The following steps ended up in failing state:

  [content-prepare]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="content-prepare"

Foreman and Proxy versions:
Katello 3.18.2 - just updated

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7

Other relevant data:

did you already start a migration previously? could it still be running? If you check Monitor > tasks and search for ‘label ~ Migration’ you should see all the instances, check if one is still running.

I have similiar situation… Tasks queue is clear. I have restarted services. Nothing helps.
Somewhere are locks, but I can not find them.

foreman ~ # foreman-rake katello:pulp3_migration_reset
enabled
Starting Content Migration Reset.  
Content Migration Reset failed, You will want to investigate: 
ForemanTasks::TaskError: Task 305326fc-21e3-43b5-9dc0-d2b832fd144d: 
Pulp2to3MigrationClient::ApiError: Error message: the server returns an error 
HTTP status code: 400
Response headers: {"date"=>"Thu, 01 Apr 2021 07:46:45 GMT", "server"=>"gunicorn/20.0.4", 
"content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"POST, OPTIONS", "x-frame- 
options"=>"SAMEORIGIN", "content-length"=>"57", "via"=>"1.1 ", " 
"connection"=>"close"}
Response body: ["Only one migration plan can run or be reset at a time"]
/opt/theforeman/tfm/root/usr/share/gems/gems/katello- 
3.18.2.1/lib/katello/tasks/pulp3_migration_reset.rake:18:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:pulp3_migration_reset
(See full trace by running task with --trace)

If i remember correctly, I ran the first iteration of the migration in 3.16 (or whenever it was introduced into the upgrade process), but no current migration task is running.

Now there’s a different error:

ForemanTasks::TaskError: Task df844f21-7bc5-40ce-96e3-73a02054d273: PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {"date"=>"Wed, 07 Apr 2021 12:37:50 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: {"detail":"You do not have permission to perform this action."}

And the foreman task shows that its happening on Actions::Pulp3::ImportMigration

I actually tried the migration the very first time today. The first run ended with:

Response body: {"detail":"You do not have permission to perform this action."}

Now any run ends with

Response body: ["Only one migration plan can run or be reset at a time"]

No task is active though…

I had a host of other problems that i seem to have finally fixed, among them was a disk issue. Not that a disk was full, but that qdrouterd default behaviour is to stop functioning properly if disk usage is >90%, after fixing that my Foreman instance appears to finally be back to stable and this being the last thing that doesn’t work.
I tested running foreman-maintain content switchover and watching logs, i found this error:

apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/

Hopefully someone can tell us whats going on and how to fix it :slight_smile:

Just for completeness sake, here’s the entire log output from pulpcore-api for the run.

apr 08 14:24:12 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:12 +0000] “GET /pulp/api/v3/status HTTP/1.1” 301 0 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “Ruby”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status HTTP/1.1” 301 0 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] “GET /pulp/api/v3/status/ HTTP/1.1” 200 2085 “-” “rest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157”
apr 08 14:24:14 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:14 +0000] “POST /pulp/api/v3/migration-plans/ HTTP/1.1” 201 53897 “-” “OpenAPI-Generator/0.7.0/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “POST /pulp/api/v3/migration-plans/0c3ce164-b1fb-4cd9-945e-47ab6f95fa3c/run/ HTTP/1.1” 202 67 “-” “OpenAPI-Generator/0.7.0/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “GET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] “GET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1” 403 63 “-” “OpenAPI-Generator/3.7.1/ruby”

Found in another post ( [ContentMigration] foreman-maintain content prepare failed - #5 by touchardv ).
Ran command
sudo su - postgres -c "psql -d pulpcore -c 'UPDATE auth_user SET is_superuser = true WHERE id =1;'"
and retried, that seems to have worked, so that seems to be the permission error at least.

Tagging @Justin_Sherrill to make sure he sees the solution, if others have the same issue. :smiley:

@ekohl suggested using the pulpcore shell to make the change in this topic: Pulp3 content migration - pulpcore admin is_superuser

# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell
from django.contrib.auth import get_user_model
user_model = get_user_model()
user = user_model.objects.get(username='admin')
user.is_superuser = True
user.save()

This is probably the more elegant solution, instead of going directly into the database yourself - marking this as the better solution to the permission error.

I was so close…
Now i’m hitting another issue it seems:

OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn’t compiled with zchunk support

Found a but report for this issue: Issue #8400: Migration fails if there is .zck metadata in pulp2 - Migration Plugin - Pulp (plan.io)

Guess i’ll have to wait until that is resolved to finish my migration - but at least we figured out the permission error :slight_smile:

I had the exact same errors as you did birkirf, both the preceding issue and the last zchunk one. That issue was resolved for me once we upgraded python3-pulp-2to3-migration to 0.11.0-1.el7 from 0.10

I spoke too soon, another run, that went on for hours this time - stopped with the same kind of error.