Hi, Confirmed. I followed your steps and I recovered all the space wasted on the Proxy. I’m still stuck with the issue that the Optimised sync does not work and I have to do a manual Complete sync.
Okay, glad the smart proxy is back to “normal”. Now back to the issue at hand.
Are there any errors in /var/log/messages from Pulp?
If you publish a content view version in a new content view and promote it to the proper environment does is your new content still missing on the smart proxy?
Since I would recommend that you upgrade to Foreman 2.3.2/Katello 3.18.1 anyway, perhaps we can try that in the meantime if it works for you. I would recommend upgrading because Katello 3.18 is the last release to support Pulp 2. This means it is going to get the most attention from the developers since those who cannot upgrade to Pulp 3 for whatever reason might stick around on it longer.
There is a decent chance that the issue will go away upon upgrade since we’ve changed some of our smart proxy code between 3.17 and 3.18, but let me know if you would prefer not to upgrade
I recall that I upgraded pulp2 to pulp3 previously but it seems that pulp2 is still in use.
These are of interest:
Jan 25 09:06:04 forman-server pulpcore-resource-manager: pulp: pulp_2to3_migration.app.plugin:INFO: Plugin pulp_deb is not installed in pulp3 therefore it will not be migrated from pulp2
and:
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: There are 0 pulpcore-resource-manager processes running. Pulp will not operate correctly without at least one pulpcore-resource-mananger process running.
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '1802@foreman-server.domain' has gone missing, removing from list of workers
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1802@foreman-server.domain is missing. Canceling the tasks in its queue.
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '1800@foreman-server.domain' has gone missing, removing from list of workers
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1800@foreman-server.domain is missing. Canceling the tasks in its queue.
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: Worker '1792@foreman-server.domain' has gone missing, removing from list of workers
Jan 25 09:29:24 foreman-server pulpcore-resource-manager: pulp: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 1792@foreman-server.domain is missing. Canceling the tasks in its queue.
I’m guessing you ran the Pulp 3 migration during Katello 3.15 when we only upgraded Docker and File content. Don’t worry about upgrading to Pulp 3 for Yum content yet, going to 3.18 won’t do that automatically. You’d have to enable it yourself if you wanted to.
Since your on Pulp 2 for yum, you shouldn’t need to worry about these errors. They’re all Pulp 3 related.
Do you see the missing workers error often?
I’m not sure what the missing worker implies. What is a high or low value? For the issue at had, where I have t o do complete syncs to get the packages available to the proxies is there a command I can run that will fix the missing dynflow jobs?
The missing workers implies something is wrong with Pulp 3 in the background. It could just be that the services need a restart, unless you’ve already done that. I’d be more concerned if this worker issue is popping up as an error when doing a specific action.
I’d recommend that you upgrade to Foreman 2.3/Katello 3.18 to see if it fixes the issue. I’m not sure yet about what is causing the issue.
Thanks. I have upgraded to 2.3 / 3.18 recently which made no difference to the issue.
Just to be sure, both your foreman server and your remote smart proxy have pulp-server-2.21.5-1.el7.noarch and foreman-proxy-2.3.2-1.el7.noarch ?
Also have you double checked that none of your settings have changed under Administer -> Settings -> Content? I’ve heard before of people who’ve had the settings change unexpectedly after an upgrade. It would be worth checking
Is this issue you’re seeing happening to all repositories, or only certain ones? Or perhaps only certain lifecycle environments or content views? I’m curious if there’s a pattern for the failures.
Here’s a simple test I did in my 3.18 environment that succeeded. It sounds like it should fail in your environment:
- Set my environment to have yum content on Pulp 2, and docker + file content on Pulp 3
- Sync this test repo we use on-demand
- Create a lifecycle environment called “env”
- Create a content view with the yum repo from above and publish a new version
- Promote the new version 1 to “env” (so now it’s “env” and “Library”)
- Set my smart proxy to use both lifecycle environments and sync
- Register my content host to the smart proxy and assign it to the content view and “env”
- Check that I can install rpms from the repo above on my content host
- Filter out all rpms from the content view and publish a new version
- Promote the new version 2 to “env”
- Check that the smart proxy synced automatically
- Check that no rpms from the repo I synced are available to the content host
- For completeness, I lastly un-did the filters, republished and promoted, then checked that I can install rpms on the content host again
Does a simple case like the above fail for you?
Hi,
Long of it. I upgraded the proxy to 2.3.3 and ran the upgrade and now everything sync on the normal optomized sync. Problem solved.
Thanks for your time on this.
1 Like
Hi @daniejstriata ,
I’m really glad to hear your problem was resolved with an upgrade. It’s a good tip that the proxy upgrade fixed the issue in case anyone else hits this.
If you have it, do you mind just posting what your old proxy version was? No worries if not.
I was on 2.3.2 and 2.3.1 but I think I forgot to execute the upgrade processes on the proxy after upgrading the packages.
1 Like
Gotcha, thanks for the info.