Smart proxy sync not complete although showing that it successfully completed

Here you go:

=> [“d7284714-6935-430a-a32d-db23249a3382”, “b7a33532-cec2-4935-bd0d-2c9cc161c1fc”, “2757be0e-fdd6-4d6c-a3e3-80588019feaa”, “99a61391-8f23-4c3e-ad71-2463ae67d5d5”, “36fdfc31-e6a3-4b23-8063-d47cf5a5a2f3”, “49142233-adf7-4652-8198-5b44276eb9dc”, “554d45e7-7d0b-451c-bbc4-50c458a15187”, “fadf3065-549a-498f-83cb-7e8bf466a011”, “d335c6b6-c626-4157-8f63-badcdac1888e”, “f77c09c7-2adb-47ba-88dc-ee34ca74bccf”, “c289f413-c5a4-485b-921d-5954e8b10b8c”, “fcfe72f2-1c29-4058-ab3f-890179b1983f”, “fa9f1139-1664-4392-b80e-1874a16f5272”, “a06330c6-1a54-4733-9d6b-483a16670232”, “ef1ac883-0488-4c30-9bba-e7ef424b72e2”, “dabc7c61-7ee3-465c-9525-b8bb196f2206”, “8200d0fc-fbb0-4d5a-968e-1e9206efe6b7”, “2f5382e3-5744-4eaa-97b3-dea445e1b890”, “bbfeb105-4197-4b77-b718-92196baeeef9”, “db9ceb66-e2df-457c-825e-2bb13deb6d8d”, “dd2a15ac-eb6b-47ec-a97f-ce7f9b686f83”, “3f9055fc-d836-4473-81aa-096bcf915fdb”, “a00797c8-8343-4f1c-aebe-afba08c9ed41”, “5389daff-52fa-4076-a14b-a646d6fc74ad”, “153c3435-2641-42ec-8fc6-19a1ef817cf0”, “2af3ec5a-216a-4072-9ef1-a45c78442a1c”]

All my Product repos use the On Demand Download Policy.


What about the Download Policy on your smart proxy itself? Same thing? You can see it from the edit menu on the proxy.

I must’ve misunderstood about the space issue you asked/ warned me about. I ran out. I grew the volume and ran out again. Let me see when the Complete sync finish tonight if the proxy’s pulp store still works.

BTW. Why is there the option to set the sync of a smart proxy’s Download policy to be different than the product’s download policy? I’m trying to think of a scenario where I’d want to do that but can’t.


Ah yeah I was a little worried about space running out since you’re syncing Library and seem to have a lot of repos. Hopefully that doesn’t cause too much trouble.

When a repository is set to on demand, the main Katello server will only download packages from the upstream repository when they are requested (or you do a verify checksum sync). When a smart proxy with Pulp is set to on demand, it’s a similar situation, but now the upstream repository is Katello. Packages will be downloaded from the main Katello server only when a client requests a download from the smart proxy.

Will the main foreman download the package when requested but does not have it in its own repo? Then the proxy should seamlessly download the package when a host request it. I think that is what is broken.

I’ll have to roll back on this idea. I don’t have enough space to sync all the repos in full. What I don’t understand is my main Foreman server’s pulp storage is also now running out of space although I did not set the main one to immediately download.

If the package isn’t listed in the repository’s package list in the Foreman UI, it won’t be fetched. I’m guessing this would be a case where the upstream repository has a new or updated package? You’d have to re-sync the repository first in Foreman, then sync the smart proxy. (Let me know if I’m misinterpreting what you mean by “…does not have it in its own repo”). If your package is in the repo’s package list in the UI (the package is indexed), but Foreman hasn’t downloaded it yet, then yeah Foreman will download the package to serve locally.

Since the smart proxy is set to immediate, it will ask to download all packages in the lifecycle environments. It’s asking to download all packages from Foreman, so Foreman will then go download all packages from the upstream repository. Let’s scratch that idea then.

Out of interest. How would one calculate the space required?

Product size * Life cycle count for each product or is it only the product size of each product and the version for each CV?

The content files are all reused so it would just be the size of all of the content in all of the repositories. No need to multiply by content view versions or LCEs or anything like that.

I dug myself a deeper hole. I Force stopped the sync that was persisting and taking up the space on the pulp volumes. Now the proxy sync status stays at 100% and when I start a new sync. Any sync. I get a failure.

There was an issue with the backend service pulp_auth: 500 Internal Server ErrorThere was an issue with the backend service pulp_auth: 500 Internal Server Error

I’ve restarted both the master and proxy. Is there a way I can reset the pulp volume on the proxy?

returned to a previous snapshot. Rerunning a Complete sync and hoping it will complete successfully.

Keep me updated on that. If it doesn’t work we should be able to reset your Pulp (or perhaps resolve the failure directly).

1 Like

After the Complete sync I’m still using over 120Gb for pulp on the proxy and 9Gb on the main foreman. Do you have any ideas how to reset the proxy’s pulp store?

I’m back to where I was except that my proxy is now 130Gb Pulp store and the main foreman pulp on 9Gb. A complete Sync did not bring down usage on the proxy.

Okay, I’ve got something to try.

  1. Disassociate all lifecycle environments from your overflowing smart proxy
    • Double check that your smart proxy is set to on demand
  2. On your Foreman server, run:
sudo foreman-rake katello:delete_orphaned_content
  1. Once the task is finished, check if your smart proxy now has free space. If it doesn’t, we’ll have to try something else (and ignore step 4).
  2. Add your lifecycle environments back to your smart proxy and sync again.

What do you mean with “Disassociate all lifecycle environments from your overflowing smart proxy”? Not sure how I can do that.

Here’s what I meant:

  1. Hit Edit in the UI for your overflowing Smart Proxy.
  2. Go over to the Lifecycle Environments tab.
  3. Remove all Lifecycle Environments from the Selected items box.
  4. Save your Smart Proxy.

Then proceed with steps 2-4.

1 Like

Thanks for clarifying. I had completely the wrong thing in mind.

I see to tasks running to clear orphans.

The one failed. Seems to be a pulp3 task.

The other one is still running with no error output.

Can you double check if all of your Pulp 3 services are running? Here they are:


Also, for that task that failed, can I see the dynflow console? Looks like there is one “Remove orphans” for your main Katello and one for your smart proxy. The actions in the Dynflow console could probably tell us which one failed.

One related thing to double check is that

::SmartProxy.find(<your remote smart proxy id>).pulp3_enabled?

in the foreman console returns false.

Thanks. Here is the running services on main forman:

● pulpcore-worker@1.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:51 UTC; 1s ago
● pulpcore-worker@2.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:46 UTC; 6s ago
● pulpcore-content.service - Pulp Content App
   Active: active (running) since Wed 2021-01-13 20:33:20 UTC; 1 weeks 0 days ago
● pulpcore-api.service - Pulp WSGI Server
   Active: active (running) since Wed 2021-01-13 20:33:20 UTC; 1 weeks 0 days ago
● pulpcore-resource-manager.service - Pulp Resource Manager
   Active: activating (auto-restart) (Result: exit-code) since Thu 2021-01-21 13:40:52 UTC; 1s ago
● pulpcore-worker@4.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:47 UTC; 5s ago
● pulpcore-worker@3.service - Pulp RQ Worker
   Active: active (running) since Thu 2021-01-21 13:40:51 UTC; 2s ago

And on the Proxy I have none of them running.

The dynflow console:

Output for SmartProxy.find is:
rb(main):004:0> ::SmartProxy.find(2).pulp3_enabled?
=> false

Okay, so it looks like the orphaned cleanup for your smart proxy wasn’t the one that failed since Pulp 3 is definitely not enabled for it. If that other remove orphans task passed successfully, do you see reduced disk usage on your smart proxy?

If not, can I see the expanded dynflow console for the failing remove orphans? Feel free to just download the page and send it over like earlier. You definitely have all the Pulpcore services running (well one is “activating” but maybe it was just restarted?), so I’m not sure why the task failed.