If i remember correctly, I ran the first iteration of the migration in 3.16 (or whenever it was introduced into the upgrade process), but no current migration task is running.
Now thereâs a different error:
ForemanTasks::TaskError: Task df844f21-7bc5-40ce-96e3-73a02054d273: PulpcoreClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {"date"=>"Wed, 07 Apr 2021 12:37:50 GMT", "server"=>"gunicorn/20.0.4", "content-type"=>"application/json", "vary"=>"Accept,Cookie", "allow"=>"GET, PATCH, DELETE, HEAD, OPTIONS", "x-frame-options"=>"SAMEORIGIN", "content-length"=>"63", "via"=>"1.1 foreman.isbank.is", "connection"=>"close"}
Response body: {"detail":"You do not have permission to perform this action."}
And the foreman task shows that its happening on Actions::Pulp3::ImportMigration
I actually tried the migration the very first time today. The first run ended with:
Response body: {"detail":"You do not have permission to perform this action."}
Now any run ends with
Response body: ["Only one migration plan can run or be reset at a time"]
No task is active thoughâŠ
I had a host of other problems that i seem to have finally fixed, among them was a disk issue. Not that a disk was full, but that qdrouterd default behaviour is to stop functioning properly if disk usage is >90%, after fixing that my Foreman instance appears to finally be back to stable and this being the last thing that doesnât work.
I tested running foreman-maintain content switchover and watching logs, i found this error:
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
Hopefully someone can tell us whats going on and how to fix it 
Just for completeness sake, hereâs the entire log output from pulpcore-api for the run.
apr 08 14:24:12 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:12 +0000] âGET /pulp/api/v3/status HTTP/1.1â 301 0 â-â ârest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157â
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status/ HTTP/1.1â 200 2085 â-â ârest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157â
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status/ HTTP/1.1â 200 2085 â-â âRubyâ
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status/ HTTP/1.1â 200 2085 â-â âRubyâ
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status/ HTTP/1.1â 200 2085 â-â âRubyâ
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status HTTP/1.1â 301 0 â-â ârest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157â
apr 08 14:24:13 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:13 +0000] âGET /pulp/api/v3/status/ HTTP/1.1â 200 2085 â-â ârest-client/2.0.2 (linux-gnu x86_64) ruby/2.5.5p157â
apr 08 14:24:14 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:14 +0000] âPOST /pulp/api/v3/migration-plans/ HTTP/1.1â 201 53897 â-â âOpenAPI-Generator/0.7.0/rubyâ
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] âPOST /pulp/api/v3/migration-plans/0c3ce164-b1fb-4cd9-945e-47ab6f95fa3c/run/ HTTP/1.1â 202 67 â-â âOpenAPI-Generator/0.7.0/rubyâ
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] âGET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1â 403 63 â-â âOpenAPI-Generator/3.7.1/rubyâ
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: pulp: django.request:WARNING: Forbidden: /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/
apr 08 14:24:15 foreman.isbank.is pulpcore-api[1293]: - - [08/Apr/2021:14:24:15 +0000] âGET /pulp/api/v3/tasks/a81d5bec-e57e-4ced-9a4b-28cd4efd96b5/ HTTP/1.1â 403 63 â-â âOpenAPI-Generator/3.7.1/rubyâ
Found in another post ( [ContentMigration] foreman-maintain content prepare failed - #5 by touchardv ).
Ran command
sudo su - postgres -c "psql -d pulpcore -c 'UPDATE auth_user SET is_superuser = true WHERE id =1;'"
and retried, that seems to have worked, so that seems to be the permission error at least.
Tagging @Justin_Sherrill to make sure he sees the solution, if others have the same issue. 
@ekohl suggested using the pulpcore shell to make the change in this topic: Pulp3 content migration - pulpcore admin is_superuser
# PULP_SETTINGS=/etc/pulp/settings.py pulpcore-manager shell
from django.contrib.auth import get_user_model
user_model = get_user_model()
user = user_model.objects.get(username='admin')
user.is_superuser = True
user.save()
1 Like
This is probably the more elegant solution, instead of going directly into the database yourself - marking this as the better solution to the permission error.
I was so closeâŠ
Now iâm hitting another issue it seems:
OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasnât compiled with zchunk support
Found a but report for this issue: Issue #8400: Migration fails if there is .zck metadata in pulp2 - Migration Plugin - Pulp (plan.io)
Guess iâll have to wait until that is resolved to finish my migration - but at least we figured out the permission error
I had the exact same errors as you did birkirf, both the preceding issue and the last zchunk one. That issue was resolved for me once we upgraded python3-pulp-2to3-migration to 0.11.0-1.el7 from 0.10
I spoke too soon, another run, that went on for hours this time - stopped with the same kind of error.
Hitting exactly the same issue (.zck metadata migration failure). Any known workarounds by now? Bug seems to be 2 months old.
1 Like
Unfortunately iâve not found any workarounds, but i have seen some threads that imply that finding the offending repos and removing them, finish upgrading and recreating the repos would work, but in my Production system iâm simply not going to do that, so havenât tried it.
I can wait on 3.18.x until they find a fix for a few more months if thats what it takes.
Hi,
Juste as reference:
Anyone heard anything regarding the fix for zck metadata? Ticket was finished 2 months ago now, but problem still exists on latest 3.18.3 installation - Server is fully updated as of 2 minutes ago, but still getting the error
Hereâs the full error from journal
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp: rq.worker:ERROR: Traceback (most recent call last):
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: rv = job.perform()
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: self._result = self._execute()
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: return self.func(*self.args, **self.kwargs)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: migrated_repo.pulp3_repository_version
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp2dist, repo_version)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: metadata_signing_service=metadata_signing_service,
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: record.fill(checksum_type)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn't compiled with zchunk support
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: Traceback (most recent call last):
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: rv = job.perform()
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: self._result = self._execute()
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: return self.func(*self.args, **self.kwargs)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 231, in complex_repo_migration
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: migrated_repo.pulp3_repository_version
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py", line 478, in migrate_repo_distributor
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: pulp2dist, repo_version)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/rpm/repository.py", line 78, in migrate_to_pulp3
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: publish(repo_version.pk, checksum_types=checksum_types, sqlite_metadata=sqlite)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 320, in publish
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: metadata_signing_service=metadata_signing_service,
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/publishing.py", line 551, in create_repomd_xml
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: record.fill(checksum_type)
jĂșl 13 11:46:55 foreman.isbank.is pulpcore-worker-6[1255]: OSError: Error while computing stat of compressed content of updateinfo.xml.zck:Cannot open a file updateinfo.xml.zck: createrepo_c wasn't compiled with zchunk support
You seem to have a different createrepo_c installed. Please check:
# which createrepo_c
/bin/createrepo_c
# rpm -qf /bin/createrepo_c
createrepo_c-0.17.1-1.el7.x86_64
# yum list createrepo_c-0.17.1-1.el7.x86_64
Loaded plugins: enabled_repos_upload, fastestmirror, package_upload, product-id, search-disabled-repos, subscription-manager,
: tracer_upload
Loading mirror speeds from cached hostfile
Installed Packages
createrepo_c.x86_64 0.17.1-1.el7 @ORG_katello_pulpcore_3_7_el7_x86_64
Uploading Enabled Repositories Report
Loaded plugins: fastestmirror, product-id, subscription-manager
got the exact same output as you.
createrepo_c version 0.17.1-1.el7
however - Iâve just found that i have tfm-rubygem-pulp_2to3_migration_client-0.11.0-1.el7.noarch installed, while at a quick glance, it looks like the zck fix is was pushed to milestone 0.11.1, not sure if i somehow have an out-of-date package there or if it was backported
And I think I forgot to run the migration-reset before retrying⊠Coming back to this after holiday break and doing things right the first time is hardâŠ
Will report back if reset and rerun gets me through this. Thanks for all the help thus far.
Problem appears to have been my holiday fog forgetting to run the migration-reset.
After running that and rerunning the prepare step it ended with some corrupt rpms - after fixing that and rerunning it finally finished.
Currently running switchover, then itâs just upgrade time!
1 Like