Can you give us your Katello version? Some of the smart proxy code changed between 3.17 and 3.18. I’d like to be sure which version you’re on.
I’m on katello-3.17.1-1.el7.noarch for both the servers. Should I upgrade to Foreman 2.3 and see if problem goes away?
I’m not yet if upgrading would fix your issue.
If you expand the single Actions::Pulp::Consumer::SyncCapsule action in your optimized sync task’s Dynflow console, there should be at least one line that contains - pulp:repository:<repo id>. Can I see what that says?
I think you should see a Actions::Pulp::Consumer::SyncCapsule for each repo that needs to be synced. Are you expecting more than 1 repo to be synced to your smart proxy?
Oh also just to be sure, when you said quick sync, did you mean optimized sync?
There should be a “Synchronize smart proxy” task that pops up right after you promote your content view, I’d be curious to see the - pulp:repository:<repo id> for that task specifically (if it’s not the same one that you started showing me above).
Hi,
Just to make sure I’m understanding you. Here is the steps as I see it.
- I create a new version of my CV but have not promoted any Lifecycle Environments. It is just built Library.
- When it completed to create a task was kicked off called
Synchronize smart proxy 'xxx' - I waited for this task to complete.
- I clicked on Dynflow console and this is what I see ( I don’t see what you’re asking for)
Perhaps related to: Bug: Smart Proxies do not sync (Katello 3.15 through 3.18 RC2) - #6 by Thulium-Drake
Check your logs for any python stacktraces.
1 Like
Thank you for this! It is very possible.
I have a different log from what is in the reported bug though
I was expecting a smart proxy sync task to run automatically after you promote a new content view version to one of the lifecycle environments that are attached to your smart proxy. Since you’re manually syncing your smart proxy after content view publishing, it leads me to think that isn’t happening for some reason.
So we’re on the same page, what lifecycle environments are you syncing with your smart proxy? It would be good to know if Library is among them.
In your newest Dynflow screen shot, I see many SyncCapsule actions (should be one per repo synced), which makes me think this smart proxy sync you did actually updated your smart proxy. Did this work? Was it an optimized sync or complete? Your original screen shot earlier only had one SyncCapsule task, so I’d like to know the difference.
Can you to expand one of the Dynflow SyncCapsule actions for a smart proxy sync task that didn’t update packages on your smart proxy and show all of the text inside?
1 Like
Hi,
The screenshot is from the automatically created sync as soon as I created the new version for the CV.
I’m expecting all the lifecycle environments to sync across and expect it to happen automatically.
I don’t under stand your question about “smart proxy sync you did actually updated your smart proxy”
I believe I attached what you asked for:
Dynflow Console.tar.gz (11.6 KB)
I wasn’t sure if the smart proxy sync you just showed me worked or not. Were packages on your smart proxy updated?
If yes, they were updated, then I need to see the same logs for a smart proxy sync that didn’t work.
If no, the smart proxy sync failed to sync new packages, then I’d like to see the following:
- Determine which repository did not get an update but should’ve
- Browse to the repository and find the numeric ID that’s at the end of the url for the repo (url should look something like https://katello.com/products/1/repositories/5)
- Open the foreman console with
sudo foreman-rake console - Show me the output of:
::Katello::Repository.find(<ID HERE>).pulp_id
I’m trying to find out if the missing packages are happening because there is a missing SyncCapsule action or if there is a correct SyncCapsule action that is silently doing nothing. By giving me the Pulp ID of a repository that failed to sync, I can check your logs to see if the repository is mentioned in the SyncCapsule action.
Hi,
I have many products but I chose one specifically because I know it was missing from hosts served by the proxy. Its ID is 1367.
irb(main):002:0> ::Katello::Repository.find(1367).pulp_id
=> "a06330c6-1a54-4733-9d6b-483a16670232"
The images I sent you are for the Automatic sync. When I do complete syncs the data is available but the entries are much longer from Dynflow. If you want I can share the complete sync’s dynflow task with you.
Okay, I didn’t find that pulp_id in the logs for the automatic sync that you showed me. You’re sure that the repository with ID 1367 was in your content view before you published the new content view version?
I’m not yet seeing in the code how a repository could go missing like that if it was in your content view. I also wasn’t able to reproduce the problem in the quick tests I tried, hm.
Has this issue only been occurring since you upgraded to Katello 3.17?
Yes. It’s been a product I setup for a long time now. I did not have this issue on version 2.0 (I always use the Katello version matching the Foreman version). I may have noticed it happen a couple of times on 2.1 but since 2.2 it is consistently not auto syncing.
Another thing I noticed (not sure related) about syncing the repos I found is that I cannot use any of a new version’s packages. If I try to update/install a package on the proxy first the yum would tell me it knows about the new package but it fails to download. When I download the package in question first on a host managed by the main foreman instance it seems to make it available for the proxy’s host too as once that is done I can go to a host managed by the proxy and then the package will download from the repo hosted by the proxy. All my repos are set to: Mirror on Sync - Yes but previously this also was not an issue.
For the content view that we’ve been discussing (the one from this post), can you find the ID from the url and show me the following in the console?
::Katello::ContentView.find(<content view ID here>).repositories.pluck(:pulp_id)
I’d like to see how many of the repositories on your content view match up to the log you sent.
I have to think more about this behavior, but I don’t suppose it has to do with the remote smart proxy being set to download “on demand” instead of “immediate”? That would be a bug and this would be a workaround, but it might be worth trying if your smart proxy has the disk space available.
Here you go:
=> [“d7284714-6935-430a-a32d-db23249a3382”, “b7a33532-cec2-4935-bd0d-2c9cc161c1fc”, “2757be0e-fdd6-4d6c-a3e3-80588019feaa”, “99a61391-8f23-4c3e-ad71-2463ae67d5d5”, “36fdfc31-e6a3-4b23-8063-d47cf5a5a2f3”, “49142233-adf7-4652-8198-5b44276eb9dc”, “554d45e7-7d0b-451c-bbc4-50c458a15187”, “fadf3065-549a-498f-83cb-7e8bf466a011”, “d335c6b6-c626-4157-8f63-badcdac1888e”, “f77c09c7-2adb-47ba-88dc-ee34ca74bccf”, “c289f413-c5a4-485b-921d-5954e8b10b8c”, “fcfe72f2-1c29-4058-ab3f-890179b1983f”, “fa9f1139-1664-4392-b80e-1874a16f5272”, “a06330c6-1a54-4733-9d6b-483a16670232”, “ef1ac883-0488-4c30-9bba-e7ef424b72e2”, “dabc7c61-7ee3-465c-9525-b8bb196f2206”, “8200d0fc-fbb0-4d5a-968e-1e9206efe6b7”, “2f5382e3-5744-4eaa-97b3-dea445e1b890”, “bbfeb105-4197-4b77-b718-92196baeeef9”, “db9ceb66-e2df-457c-825e-2bb13deb6d8d”, “dd2a15ac-eb6b-47ec-a97f-ce7f9b686f83”, “3f9055fc-d836-4473-81aa-096bcf915fdb”, “a00797c8-8343-4f1c-aebe-afba08c9ed41”, “5389daff-52fa-4076-a14b-a646d6fc74ad”, “153c3435-2641-42ec-8fc6-19a1ef817cf0”, “2af3ec5a-216a-4072-9ef1-a45c78442a1c”]
All my Product repos use the On Demand Download Policy.
Thanks.
What about the Download Policy on your smart proxy itself? Same thing? You can see it from the edit menu on the proxy.
I must’ve misunderstood about the space issue you asked/ warned me about. I ran out. I grew the volume and ran out again. Let me see when the Complete sync finish tonight if the proxy’s pulp store still works.
BTW. Why is there the option to set the sync of a smart proxy’s Download policy to be different than the product’s download policy? I’m trying to think of a scenario where I’d want to do that but can’t.
Ah yeah I was a little worried about space running out since you’re syncing Library and seem to have a lot of repos. Hopefully that doesn’t cause too much trouble.
When a repository is set to on demand, the main Katello server will only download packages from the upstream repository when they are requested (or you do a verify checksum sync). When a smart proxy with Pulp is set to on demand, it’s a similar situation, but now the upstream repository is Katello. Packages will be downloaded from the main Katello server only when a client requests a download from the smart proxy.
Will the main foreman download the package when requested but does not have it in its own repo? Then the proxy should seamlessly download the package when a host request it. I think that is what is broken.
I’ll have to roll back on this idea. I don’t have enough space to sync all the repos in full. What I don’t understand is my main Foreman server’s pulp storage is also now running out of space although I did not set the main one to immediately download.
If the package isn’t listed in the repository’s package list in the Foreman UI, it won’t be fetched. I’m guessing this would be a case where the upstream repository has a new or updated package? You’d have to re-sync the repository first in Foreman, then sync the smart proxy. (Let me know if I’m misinterpreting what you mean by “…does not have it in its own repo”). If your package is in the repo’s package list in the UI (the package is indexed), but Foreman hasn’t downloaded it yet, then yeah Foreman will download the package to serve locally.
Since the smart proxy is set to immediate, it will ask to download all packages in the lifecycle environments. It’s asking to download all packages from Foreman, so Foreman will then go download all packages from the upstream repository. Let’s scratch that idea then.