I’m guessing you did the migration to Pulp 3 back when it was just for docker and file content. That’s fine that you’re on Pulp 2 still for yum content, it’ll just have to be migrated again before you update to Katello 4.0 in the future.
Is it just yum content that you’re syncing to your smart proxy? I’m going to see if I can reproduce the issue in my environment.
If you expand the single Actions::Pulp::Consumer::SyncCapsule action in your optimized sync task’s Dynflow console, there should be at least one line that contains - pulp:repository:<repo id>. Can I see what that says?
I think you should see a Actions::Pulp::Consumer::SyncCapsule for each repo that needs to be synced. Are you expecting more than 1 repo to be synced to your smart proxy?
Oh also just to be sure, when you said quick sync, did you mean optimized sync?
There should be a “Synchronize smart proxy” task that pops up right after you promote your content view, I’d be curious to see the - pulp:repository:<repo id> for that task specifically (if it’s not the same one that you started showing me above).
I was expecting a smart proxy sync task to run automatically after you promote a new content view version to one of the lifecycle environments that are attached to your smart proxy. Since you’re manually syncing your smart proxy after content view publishing, it leads me to think that isn’t happening for some reason.
So we’re on the same page, what lifecycle environments are you syncing with your smart proxy? It would be good to know if Library is among them.
In your newest Dynflow screen shot, I see many SyncCapsule actions (should be one per repo synced), which makes me think this smart proxy sync you did actually updated your smart proxy. Did this work? Was it an optimized sync or complete? Your original screen shot earlier only had one SyncCapsule task, so I’d like to know the difference.
Can you to expand one of the Dynflow SyncCapsule actions for a smart proxy sync task that didn’t update packages on your smart proxy and show all of the text inside?
Open the foreman console with sudo foreman-rake console
Show me the output of:
::Katello::Repository.find(<ID HERE>).pulp_id
I’m trying to find out if the missing packages are happening because there is a missing SyncCapsule action or if there is a correct SyncCapsule action that is silently doing nothing. By giving me the Pulp ID of a repository that failed to sync, I can check your logs to see if the repository is mentioned in the SyncCapsule action.
The images I sent you are for the Automatic sync. When I do complete syncs the data is available but the entries are much longer from Dynflow. If you want I can share the complete sync’s dynflow task with you.
Okay, I didn’t find that pulp_id in the logs for the automatic sync that you showed me. You’re sure that the repository with ID 1367 was in your content view before you published the new content view version?
I’m not yet seeing in the code how a repository could go missing like that if it was in your content view. I also wasn’t able to reproduce the problem in the quick tests I tried, hm.
Has this issue only been occurring since you upgraded to Katello 3.17?
Yes. It’s been a product I setup for a long time now. I did not have this issue on version 2.0 (I always use the Katello version matching the Foreman version). I may have noticed it happen a couple of times on 2.1 but since 2.2 it is consistently not auto syncing.
Another thing I noticed (not sure related) about syncing the repos I found is that I cannot use any of a new version’s packages. If I try to update/install a package on the proxy first the yum would tell me it knows about the new package but it fails to download. When I download the package in question first on a host managed by the main foreman instance it seems to make it available for the proxy’s host too as once that is done I can go to a host managed by the proxy and then the package will download from the repo hosted by the proxy. All my repos are set to: Mirror on Sync - Yes but previously this also was not an issue.
For the content view that we’ve been discussing (the one from this post), can you find the ID from the url and show me the following in the console?
::Katello::ContentView.find(<content view ID here>).repositories.pluck(:pulp_id)
I’d like to see how many of the repositories on your content view match up to the log you sent.
I have to think more about this behavior, but I don’t suppose it has to do with the remote smart proxy being set to download “on demand” instead of “immediate”? That would be a bug and this would be a workaround, but it might be worth trying if your smart proxy has the disk space available.
I must’ve misunderstood about the space issue you asked/ warned me about. I ran out. I grew the volume and ran out again. Let me see when the Complete sync finish tonight if the proxy’s pulp store still works.
BTW. Why is there the option to set the sync of a smart proxy’s Download policy to be different than the product’s download policy? I’m trying to think of a scenario where I’d want to do that but can’t.
Ah yeah I was a little worried about space running out since you’re syncing Library and seem to have a lot of repos. Hopefully that doesn’t cause too much trouble.
When a repository is set to on demand, the main Katello server will only download packages from the upstream repository when they are requested (or you do a verify checksum sync). When a smart proxy with Pulp is set to on demand, it’s a similar situation, but now the upstream repository is Katello. Packages will be downloaded from the main Katello server only when a client requests a download from the smart proxy.
Will the main foreman download the package when requested but does not have it in its own repo? Then the proxy should seamlessly download the package when a host request it. I think that is what is broken.
I’ll have to roll back on this idea. I don’t have enough space to sync all the repos in full. What I don’t understand is my main Foreman server’s pulp storage is also now running out of space although I did not set the main one to immediately download.