I have 2 Foreman servers. One is a Proxy getting its repos from the main foreman. When I publish new versions of Content Views they are successfully synced across to the Foreman Proxy but I do not have access to the new packages on hosts assigned to the proxy. I have to do a Complete Sync each time before the Proxy serves the updates packages.
I want to find out why the Quick sync is not working as expected but don’t see any errors in production log and no error on frontend as the GUI states the quick sync was successfully completed.
Foreman and Proxy versions:
2.2.2
Foreman and Proxy plugin versions:
Distribution and version:
Centos 7
Other relevant data:
I cannot see any errors in the production.log and the quick sync completes without errors.
Hi @daniejstriata,![Screenshot from 2021-01-08 12-03-32|516x500]
Can you show us the Dynflow console plans for your quick sync and your complete sync? You will find it by clicking the “Dynflow console” button on each of the “Synchronize smart proxy” tasks.
The plan in the Dynflow console will look like this:
Would you mind showing me the “Services” tab for your smart proxy that isn’t the remote one as well? That would be the smart proxy that says “Pulp” instead of “Pulp node” on the “Services” tab. I can tell that your remote proxy doesn’t have Pulp 3 (which is expected) but I’m still not 100% sure about your main Katello server’s smart proxy.
Was this Katello server a fresh install for Foreman 2.2.2 or did you upgrade from a previous version?
I’m guessing you did the migration to Pulp 3 back when it was just for docker and file content. That’s fine that you’re on Pulp 2 still for yum content, it’ll just have to be migrated again before you update to Katello 4.0 in the future.
Is it just yum content that you’re syncing to your smart proxy? I’m going to see if I can reproduce the issue in my environment.
If you expand the single Actions::Pulp::Consumer::SyncCapsule action in your optimized sync task’s Dynflow console, there should be at least one line that contains - pulp:repository:<repo id>. Can I see what that says?
I think you should see a Actions::Pulp::Consumer::SyncCapsule for each repo that needs to be synced. Are you expecting more than 1 repo to be synced to your smart proxy?
Oh also just to be sure, when you said quick sync, did you mean optimized sync?
There should be a “Synchronize smart proxy” task that pops up right after you promote your content view, I’d be curious to see the - pulp:repository:<repo id> for that task specifically (if it’s not the same one that you started showing me above).
I was expecting a smart proxy sync task to run automatically after you promote a new content view version to one of the lifecycle environments that are attached to your smart proxy. Since you’re manually syncing your smart proxy after content view publishing, it leads me to think that isn’t happening for some reason.
So we’re on the same page, what lifecycle environments are you syncing with your smart proxy? It would be good to know if Library is among them.
In your newest Dynflow screen shot, I see many SyncCapsule actions (should be one per repo synced), which makes me think this smart proxy sync you did actually updated your smart proxy. Did this work? Was it an optimized sync or complete? Your original screen shot earlier only had one SyncCapsule task, so I’d like to know the difference.
Can you to expand one of the Dynflow SyncCapsule actions for a smart proxy sync task that didn’t update packages on your smart proxy and show all of the text inside?
Open the foreman console with sudo foreman-rake console
Show me the output of:
::Katello::Repository.find(<ID HERE>).pulp_id
I’m trying to find out if the missing packages are happening because there is a missing SyncCapsule action or if there is a correct SyncCapsule action that is silently doing nothing. By giving me the Pulp ID of a repository that failed to sync, I can check your logs to see if the repository is mentioned in the SyncCapsule action.
The images I sent you are for the Automatic sync. When I do complete syncs the data is available but the entries are much longer from Dynflow. If you want I can share the complete sync’s dynflow task with you.
Okay, I didn’t find that pulp_id in the logs for the automatic sync that you showed me. You’re sure that the repository with ID 1367 was in your content view before you published the new content view version?
I’m not yet seeing in the code how a repository could go missing like that if it was in your content view. I also wasn’t able to reproduce the problem in the quick tests I tried, hm.
Has this issue only been occurring since you upgraded to Katello 3.17?
Yes. It’s been a product I setup for a long time now. I did not have this issue on version 2.0 (I always use the Katello version matching the Foreman version). I may have noticed it happen a couple of times on 2.1 but since 2.2 it is consistently not auto syncing.
Another thing I noticed (not sure related) about syncing the repos I found is that I cannot use any of a new version’s packages. If I try to update/install a package on the proxy first the yum would tell me it knows about the new package but it fails to download. When I download the package in question first on a host managed by the main foreman instance it seems to make it available for the proxy’s host too as once that is done I can go to a host managed by the proxy and then the package will download from the repo hosted by the proxy. All my repos are set to: Mirror on Sync - Yes but previously this also was not an issue.
For the content view that we’ve been discussing (the one from this post), can you find the ID from the url and show me the following in the console?
::Katello::ContentView.find(<content view ID here>).repositories.pluck(:pulp_id)
I’d like to see how many of the repositories on your content view match up to the log you sent.
I have to think more about this behavior, but I don’t suppose it has to do with the remote smart proxy being set to download “on demand” instead of “immediate”? That would be a bug and this would be a workaround, but it might be worth trying if your smart proxy has the disk space available.
