Smart proxy sync not complete although showing that it successfully completed

daniejstriata · January 12, 2021, 9:23pm

Here you go:

=> [“d7284714-6935-430a-a32d-db23249a3382”, “b7a33532-cec2-4935-bd0d-2c9cc161c1fc”, “2757be0e-fdd6-4d6c-a3e3-80588019feaa”, “99a61391-8f23-4c3e-ad71-2463ae67d5d5”, “36fdfc31-e6a3-4b23-8063-d47cf5a5a2f3”, “49142233-adf7-4652-8198-5b44276eb9dc”, “554d45e7-7d0b-451c-bbc4-50c458a15187”, “fadf3065-549a-498f-83cb-7e8bf466a011”, “d335c6b6-c626-4157-8f63-badcdac1888e”, “f77c09c7-2adb-47ba-88dc-ee34ca74bccf”, “c289f413-c5a4-485b-921d-5954e8b10b8c”, “fcfe72f2-1c29-4058-ab3f-890179b1983f”, “fa9f1139-1664-4392-b80e-1874a16f5272”, “a06330c6-1a54-4733-9d6b-483a16670232”, “ef1ac883-0488-4c30-9bba-e7ef424b72e2”, “dabc7c61-7ee3-465c-9525-b8bb196f2206”, “8200d0fc-fbb0-4d5a-968e-1e9206efe6b7”, “2f5382e3-5744-4eaa-97b3-dea445e1b890”, “bbfeb105-4197-4b77-b718-92196baeeef9”, “db9ceb66-e2df-457c-825e-2bb13deb6d8d”, “dd2a15ac-eb6b-47ec-a97f-ce7f9b686f83”, “3f9055fc-d836-4473-81aa-096bcf915fdb”, “a00797c8-8343-4f1c-aebe-afba08c9ed41”, “5389daff-52fa-4076-a14b-a646d6fc74ad”, “153c3435-2641-42ec-8fc6-19a1ef817cf0”, “2af3ec5a-216a-4072-9ef1-a45c78442a1c”]

All my Product repos use the On Demand Download Policy.

iballou · January 12, 2021, 9:33pm

Thanks.

What about the Download Policy on your smart proxy itself? Same thing? You can see it from the edit menu on the proxy.

daniejstriata · January 13, 2021, 2:16pm

I must’ve misunderstood about the space issue you asked/ warned me about. I ran out. I grew the volume and ran out again. Let me see when the Complete sync finish tonight if the proxy’s pulp store still works.

BTW. Why is there the option to set the sync of a smart proxy’s Download policy to be different than the product’s download policy? I’m trying to think of a scenario where I’d want to do that but can’t.

iballou · January 13, 2021, 2:51pm

@daniejstriata,

Ah yeah I was a little worried about space running out since you’re syncing Library and seem to have a lot of repos. Hopefully that doesn’t cause too much trouble.

When a repository is set to on demand, the main Katello server will only download packages from the upstream repository when they are requested (or you do a verify checksum sync). When a smart proxy with Pulp is set to on demand, it’s a similar situation, but now the upstream repository is Katello. Packages will be downloaded from the main Katello server only when a client requests a download from the smart proxy.

daniejstriata · January 13, 2021, 2:55pm

Will the main foreman download the package when requested but does not have it in its own repo? Then the proxy should seamlessly download the package when a host request it. I think that is what is broken.

I’ll have to roll back on this idea. I don’t have enough space to sync all the repos in full. What I don’t understand is my main Foreman server’s pulp storage is also now running out of space although I did not set the main one to immediately download.

iballou · January 13, 2021, 3:08pm

If the package isn’t listed in the repository’s package list in the Foreman UI, it won’t be fetched. I’m guessing this would be a case where the upstream repository has a new or updated package? You’d have to re-sync the repository first in Foreman, then sync the smart proxy. (Let me know if I’m misinterpreting what you mean by “…does not have it in its own repo”). If your package is in the repo’s package list in the UI (the package is indexed), but Foreman hasn’t downloaded it yet, then yeah Foreman will download the package to serve locally.

Since the smart proxy is set to immediate, it will ask to download all packages in the lifecycle environments. It’s asking to download all packages from Foreman, so Foreman will then go download all packages from the upstream repository. Let’s scratch that idea then.

daniejstriata · January 13, 2021, 3:58pm

Out of interest. How would one calculate the space required?

Product size * Life cycle count for each product or is it only the product size of each product and the version for each CV?

iballou · January 13, 2021, 7:15pm

The content files are all reused so it would just be the size of all of the content in all of the repositories. No need to multiply by content view versions or LCEs or anything like that.

daniejstriata · January 13, 2021, 8:06pm

I dug myself a deeper hole. I Force stopped the sync that was persisting and taking up the space on the pulp volumes. Now the proxy sync status stays at 100% and when I start a new sync. Any sync. I get a failure.

There was an issue with the backend service pulp_auth: 500 Internal Server ErrorThere was an issue with the backend service pulp_auth: 500 Internal Server Error

I’ve restarted both the master and proxy. Is there a way I can reset the pulp volume on the proxy?

daniejstriata · January 13, 2021, 9:00pm

returned to a previous snapshot. Rerunning a Complete sync and hoping it will complete successfully.

iballou · January 13, 2021, 9:11pm

Keep me updated on that. If it doesn’t work we should be able to reset your Pulp (or perhaps resolve the failure directly).

daniejstriata · January 14, 2021, 5:00am

After the Complete sync I’m still using over 120Gb for pulp on the proxy and 9Gb on the main foreman. Do you have any ideas how to reset the proxy’s pulp store?

daniejstriata · January 14, 2021, 2:16pm

I’m back to where I was except that my proxy is now 130Gb Pulp store and the main foreman pulp on 9Gb. A complete Sync did not bring down usage on the proxy.

iballou · January 14, 2021, 2:45pm

Okay, I’ve got something to try.

Disassociate all lifecycle environments from your overflowing smart proxy
- Double check that your smart proxy is set to on demand
On your Foreman server, run:

sudo foreman-rake katello:delete_orphaned_content

Once the task is finished, check if your smart proxy now has free space. If it doesn’t, we’ll have to try something else (and ignore step 4).
Add your lifecycle environments back to your smart proxy and sync again.

daniejstriata · January 20, 2021, 9:52am

What do you mean with “Disassociate all lifecycle environments from your overflowing smart proxy”? Not sure how I can do that.

iballou · January 20, 2021, 3:01pm

Here’s what I meant:

Hit Edit in the UI for your overflowing Smart Proxy.
Go over to the Lifecycle Environments tab.
Remove all Lifecycle Environments from the Selected items box.
Save your Smart Proxy.

Then proceed with steps 2-4.

daniejstriata · January 20, 2021, 8:28pm

Thanks for clarifying. I had completely the wrong thing in mind.

I see to tasks running to clear orphans.

The one failed. Seems to be a pulp3 task.

The other one is still running with no error output.

iballou · January 20, 2021, 9:12pm

Can you double check if all of your Pulp 3 services are running? Here they are:

pulpcore-api
pulpcore-resource-manager
pulpcore-content
pulpcore-worker@<id>

Also, for that task that failed, can I see the dynflow console? Looks like there is one “Remove orphans” for your main Katello and one for your smart proxy. The actions in the Dynflow console could probably tell us which one failed.

One related thing to double check is that

::SmartProxy.find(<your remote smart proxy id>).pulp3_enabled?

in the foreman console returns false.

daniejstriata · January 21, 2021, 1:59pm

Thanks. Here is the running services on main forman:

● pulpcore-worker@1.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:51 UTC; 1s ago
● pulpcore-worker@2.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:46 UTC; 6s ago
● pulpcore-content.service - Pulp Content App
   Active: active (running) since Wed 2021-01-13 20:33:20 UTC; 1 weeks 0 days ago
● pulpcore-api.service - Pulp WSGI Server
   Active: active (running) since Wed 2021-01-13 20:33:20 UTC; 1 weeks 0 days ago
● pulpcore-resource-manager.service - Pulp Resource Manager
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-21 13:40:52 UTC; 1s ago
● pulpcore-worker@4.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:47 UTC; 5s ago
● pulpcore-worker@3.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:51 UTC; 2s ago

And on the Proxy I have none of them running.

The dynflow console:

Output for SmartProxy.find is:
rb(main):004:0> ::SmartProxy.find(2).pulp3_enabled?
=> false

iballou · January 21, 2021, 2:29pm

Okay, so it looks like the orphaned cleanup for your smart proxy wasn’t the one that failed since Pulp 3 is definitely not enabled for it. If that other remove orphans task passed successfully, do you see reduced disk usage on your smart proxy?

If not, can I see the expanded dynflow console for the failing remove orphans? Feel free to just download the page and send it over like earlier. You definitely have all the Pulpcore services running (well one is “activating” but maybe it was just restarted?), so I’m not sure why the task failed.