@langesmalle I get 404s as well if I try only /pulp/repos/Default_Organization/Library/custom/. Looks like you have to navigate to the repository itself to not get the 404, i.e. /pulp/repos/Default_Organization/Library/custom/Product/Repository/
If you really can’t see that content, can I see the output of:
sudo foreman-installer --full-help | grep "\-\-foreman-proxy-content-proxy-pulp-yum-to-pulpcore"
@iballou ,
Thnx for your feedback.
My first test was with ‘the product/repo’ included in the url which resulted in a 404.
Only a wget without ‘product/repo’ went well. And for a wget for ‘Library/custom’ I could see that the content for the custom file did not contained the new product/repo, neither the new repo I added into an existing product.
wget http://my-foreman-server/pulp/repos/DIDM/Library/custom/Postgres_11/Postgres_11_EL8/
–2021-03-24 08:29:25-- http://my-foreman-server/pulp/repos/DIDM/Library/custom/Postgres_11/Postgres_11_EL8/
Resolving my-foreman-server (my-foreman-server)… 10.16.x.y
Connecting to my-foreman-server (my-foreman-server)|10.16.x.y|:80… connected.
HTTP request sent, awaiting response… 404 Not Found
So regarding the requested output, it has the state ‘false’:
[root@my-foreman-server ~]# foreman-installer --full-help | grep “–foreman-proxy-content-proxy-pulp-yum-to-pulpcore”
–foreman-proxy-content-proxy-pulp-yum-to-pulpcore Proxy /pulp/yum to Pulpcore at /pulp/content (current: false)
Is this required since the migration to pulp3, because before the migration it worked well with pulp2?
Before the migration I created a new product for centos8 and repos as well added new repos to existing products. And those were immediately available.
As far as I can remember, this was not mentioned in the documentation that it had to be activated.
So in case you confirm I need to activate this proxy, how should I proceed in order to make these new product and repo’s available? Do I need to remove them en create them again, or is there an easy way of workaround to do that?
Thnx in advance for your help.
Hi @langesmalle,
You didn’t miss any docs, it’s something we need to add to foreman maintain. I’ll be doing that soon.
To fix this, you simply need to run the following to update Apache:
foreman-installer --foreman-proxy-content-proxy-pulp-yum-to-pulpcore true --foreman-proxy-content-proxy-pulp-isos-to-pulpcore true --foreman-proxy-content-proxy-pulp-deb-to-pulpcore true
After that, all of your content should be hosted.
This is a one-time step that should be run after the switchover. You won’t need to recreate any products or repos. For anyone else reading this, foreman maintain should be doing this for you during the switchover command. That is the fix that will be taken care of soon.
@iballou ,
Thnx for you confirmation that i did not missed any action from the docs.
The provided forman-installer command is currently busy.
Will keep you informed about the result or when i discover other issues.
@iballou ,
Yeah, I can confirm that after applying this command, the repo’s are available on the hosts, a wget works as expected.
However, I just notices that the new product and/or repos I added after the pulp3 migration are not visible in the folders:
This was the case before the pulp3 migration.
Is this normal?
Glad the installer fixed it, apologies for the mix-up there.
That’s normal. You won’t be seeing hosted content on the filesystem. I just double checked, and with new Pulp 3 only installs, that /var/www/pub/ folder doesn’t even exist. You’re seeing leftover files from Pulp 2.
@iballou ,
OK thnx for this feedback.
Will this mean that the folder /var/www/pub (pub is the symlink) is no more needed?
Is it safe to remove it, and is this some thing that also needs to be implemented during the foreman-maintain content switchover process?
@langesmalle /var/www/pub/ can be deleted.
Pulp 2 cleanup is going to happen during the upgrade to 4.0 (foreman_maintain/remove.rb at master · theforeman/foreman_maintain · GitHub). At the moment I don’t think that symlink is cleaned up though, I’ll add an issue to make that happen.
@iballou ,
It looks like an never ending story here.
Today I had to sync the TimeScaleDB-el7 and TimeScaleDB-el8 repo’s which where created in pulp2.
Both syncs failed with: " Package matching query does not exist".
Details for el7:
Details:
Dynflow-console:
Subtask:
Dynflow-console:
Errors:
{“pulp_tasks”=>
[{“pulp_href”=>“/pulp/api/v3/tasks/65bfb43c-2384-4f47-bcb4-7918ef940274/”,
“pulp_created”=>“2021-03-25T10:11:33.055+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“started_at”=>“2021-03-25T10:11:33.192+00:00”,
“finished_at”=>“2021-03-25T10:11:50.411+00:00”,
“error”=>
{“traceback”=>
" File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job\n" +
" rv = job.perform()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform\n" +
" self._result = self._execute()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in _execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize\n" +
" dv.create()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File "/usr/lib64/python3.6/asyncio/base_events.py", line 484, in run_until_complete\n" +
" return future.result()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 225, in create_pipeline\n" +
" await asyncio.gather(*futures)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py", line 43, in call\n" +
" await self.run()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/content_stages.py", line 105, in run\n" +
" d_content.content.q()\n" +
" File "/usr/lib/python3.6/site-packages/django/db/models/manager.py", line 82, in manager_method\n" +
" return getattr(self.get_queryset(), name)(*args, **kwargs)\n" +
" File "/usr/lib/python3.6/site-packages/django/db/models/query.py", line 408, in get\n" +
" self.model._meta.object_name\n",
“description”=>“Package matching query does not exist.”},
“worker”=>“/pulp/api/v3/workers/dc3a7676-2030-4f46-9109-8c01f586cb3e/”,
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“completed”,
“done”=>4},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“completed”,
“done”=>0},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“canceled”,
“done”=>0},
{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“completed”,
“total”=>199,
“done”=>199}],
“created_resources”=>,
“reserved_resources_record”=>
[“/pulp/api/v3/repositories/rpm/rpm/62d212e7-4908-4734-866c-5cfeb38455e1/”,
“/pulp/api/v3/remotes/rpm/rpm/a1b6c066-0ccc-4bac-8caf-f06227e52321/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>15, “failed”=>1}}
===
Also for our old Epel7 repo which existed in pulp 2 the sync ends in an error:
With following details:
Dynflow-console:
Detailed errors:
{“pulp_tasks”=>
[{“pulp_href”=>“/pulp/api/v3/tasks/44eaf6ff-c5ee-4a55-aad0-dc2c6262a095/”,
“pulp_created”=>“2021-03-25T09:59:52.646+00:00”,
“state”=>“failed”,
“name”=>“pulp_rpm.app.tasks.synchronizing.synchronize”,
“started_at”=>“2021-03-25T09:59:52.781+00:00”,
“finished_at”=>“2021-03-25T10:01:39.545+00:00”,
“error”=>
{“traceback”=>
" File "/usr/lib/python3.6/site-packages/rq/worker.py", line 936, in perform_job\n" +
" rv = job.perform()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 684, in perform\n" +
" self._result = self._execute()\n" +
" File "/usr/lib/python3.6/site-packages/rq/job.py", line 690, in execute\n" +
" return self.func(*self.args, **self.kwargs)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/tasks/synchronizing.py", line 266, in synchronize\n" +
" dv.create()\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/declarative_version.py", line 148, in create\n" +
" loop.run_until_complete(pipeline)\n" +
" File "/usr/lib/python3.6/site-packages/pulpcore/app/models/repository.py", line 795, in exit\n" +
" repository.finalize_new_version(self)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/models/repository.py", line 238, in finalize_new_version\n" +
" resolve_advisories(new_version, previous_version)\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py", line 82, in resolve_advisories\n" +
" previous_advisory, added_advisory\n" +
" File "/usr/lib/python3.6/site-packages/pulp_rpm/app/advisory.py", line 157, in resolve_advisory_conflict\n" +
" raise AdvisoryConflict(('Incoming and existing advisories have the same id and '\n",
“description”=>
“Incoming and existing advisories have the same id and timestamp but different and intersecting package lists. At least one of them is wrong. Advisory id: FEDORA-EPEL-2019-bf69cd4c68”},
“worker”=>“/pulp/api/v3/workers/dc3a7676-2030-4f46-9109-8c01f586cb3e/”,
“child_tasks”=>,
“progress_reports”=>
[{“message”=>“Parsed Packages”,
“code”=>“parsing.packages”,
“state”=>“completed”,
“total”=>13565,
“done”=>13565},
{“message”=>“Parsed Advisories”,
“code”=>“parsing.advisories”,
“state”=>“completed”,
“total”=>4645,
“done”=>4645},
{“message”=>“Downloading Metadata Files”,
“code”=>“downloading.metadata”,
“state”=>“completed”,
“done”=>5},
{“message”=>“Downloading Artifacts”,
“code”=>“downloading.artifacts”,
“state”=>“completed”,
“done”=>0},
{“message”=>“Associating Content”,
“code”=>“associating.content”,
“state”=>“completed”,
“done”=>4954},
{“message”=>“Parsed Comps”,
“code”=>“parsing.comps”,
“state”=>“completed”,
“total”=>41,
“done”=>41}],
“created_resources”=>,
“reserved_resources_record”=>
[“/pulp/api/v3/repositories/rpm/rpm/0232c308-144b-4ab7-9083-d4c5dbfd56d4/”,
“/pulp/api/v3/remotes/rpm/rpm/63df72dd-e71f-48fc-a77e-592c7e03698f/”]}],
“create_version”=>true,
“task_groups”=>,
“poll_attempts”=>{“total”=>27, “failed”=>1}}
I hope you can help me out with these new issues as well.
Thnx in advance.
@iballou ,
As a test I created/synced for the product epel7 a new repo test_epel_7, and all went fine till the end.
As a result I saw a difference between the old and new test-repo , where the number of packages was higher for the old one.
I activated “Mirror on Sync” for it, started the sync again, and all went fine till the end.
For the TimeScaleDB repo’s I’m still investigating this further and will keep you informed if i have found a solution.
In case you need more info regarding the TimeScaleDB, please let me know.
@iballou ,
So far I still did not found a solution for the TimeScaleDB repo’s.
But in Issue #2003: Pulp will not sync RPM files hosted using "packagecloud". - Pulp I saw there was in the past already an issue with syncing repos from packagecloud.io, so it looks like with pulp3 the problem is there again.
Following Upstream URL’s I would like to sync in the repo’s:
Thnx for investigating this in detail.
Hey @langesmalle,
Thanks for the info on those TimeScaleDB repos. I’m going to try syncing them myself and get some info. It might end up being a bug for Pulp 3.
@langesmalle are you able to delete and re-create the TimeScaleDB repos? Mine synced fine after fresh creation. I’m wondering if it had to do with the migration.
If you’re unable to delete them due to content view versions or something, we can look into working around it.
Also, Katello 3.18.2 was released! I recommend upgrading when possible since it has some good bug fixes.
Just wanted to add that I was able to reproduce the “package matching query” problem after switching over. I’ll see if upgrading to 3.18.2 helps.
@iballou ,
Just removed the content views and product for TimeScaleDB_EL7 and recreated them.
But once I sync the repo it still ends up with the same warning for the subtask with: “Package matching query does not exist.”
Please let me know if version 3.18.2 would do the thing, otherwise a fix whcih I can apply in one of the sources.
Will the Katello upgrade to 3.18.2 also result in the upgrade of Foreman to 2.4 or will it stay at 2.3.3?
And of course, I hope that 3.18.2 will not result in new problems with my current products/repos which are now fine except the ones for TimeScaleDB.
Thnx in advance for your feedback.
@langesmalle I didn’t see any difference upgrading to 3.18.2 post-migration, so feel free to stay on 3.18.1 since you’re mostly settled. The upgrade to 3.18.2 would keep foreman at 2.3.3.
I’m going to ask the Pulp team about the issue.
@langesmalle I’ve filed a bug about the problem: Issue #8452: Package matching query does not exist when syncing TimeScaleDB repo after migration - Migration Plugin - Pulp
In the meantime, I have something new to try. Delete the packagecloud.io repositories, run foreman-rake katello:delete_orphaned_content, then re-create the repositories and re-sync them one-by-one. It would be good to know if syncing them one-by-one helps since the repos are potentially related. There have been Pulp issues in the past when repositories share RPMs.
@iballou ,
As said in the irc-channel, despite the extra work (removing content view versions, repos, re-create,…), the suggested workaround solved the problem.
The repo’s are back available and I’m able to re-sync them again without any problems.
Again, many thanks for your help.
Thanks a lot. That was the one and only that worked for me.
Thanks for the fix ! I followed the instructions and worked perfectly. All my repositories are synced again.
