Smart Proxy 3.5 errors out 404 with yum package install

Problem:
I have installed a new smart-proxy on version 3.5. Our foreman is currently running on 3.5 as well. I am able to register new proxy to foreman successfully. I am able to register new hosts to the new smart proxy successfully. But any host registered to the new smart-proxy cannot do yum updates. this is the :
ERROR: Status code: 404 for https://my-smart-proxy-url/pulp/content/Default_Organization/Library/RHEL8/content/dist/rhel8/8/x86_64/baseos/os/repodata/repomd.xml (IP: xxx.xxx.xx.xxx)
Error: Failed to download metadata for repo β€˜rhel-8-for-x86_64-baseos-rpms’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

Expected outcome:
Yum updates complete successfully

Foreman and Proxy versions:
3.5 with Container_Gateway, Logs, Pulpcore, Registration, and Templates

Foreman and Proxy plugin versions:
foreman-tasks 7.1.1
foreman-openscap 5.2.2
foreman_puppet 5.0.0
foreman_remote_executions 8.2.0
Distribution and version:
RHEL 8.9

Other relevant data:

Just a wild guess, but did you sync content to the smart-proxy?
With external content proxies, you need to sync content to them before clients can download packages. If you have not, under Infrastructure β†’ Smart-Proxies, select your new smart-proxy and hit the Syncronize button. For an initial sync, it should not matter which option you pick.
If you have already synced and it still does not work, I would guess something in the sync went wrong. Check the sync task if it reports any errors.

I have sysnced several times. No failures on sync process from the UI. I don’t see any errors in the smart proxy logs at /var/logs/foreman-proxy/proxy.log. I only see GET 200 messages. Nothing in the UI logs either. is there a feature we are missing?

The features look correct to me.
Do you have the correct lifecycle environment(s) assigned to the proxy? Infrastructure β†’ Smart Proxies β†’ Select your Proxy β†’ Edit β†’ Lifecycle environments.
Also, on the smart-proxy page, in the content tab, do you see the lifecycle environments you expect (Library in this case) and the associated content views? Is there a check-mark on all of them?

Yes the correct lifecycle is assigned. I see a check mark for all assigned as well. Is there a way to verify on the server side(cli) that the content is available on the smart proxy server? Also are there any specific changes required in rhsm.conf file needed?

I have no experience with it, but you could try analyzing with the pulp cli. Please not the information for katello users in the configuration docs.
The cli does not seem to be installed by default, but it is packaged in the Katello repos as python38-pulp-cli.

Thank you so much for all the help. We added Library to the lifecycle and reran the complete sync. We saw the size of /var/lib/pulp go from 7GB to 19GB.
We also had to set the subscription release version for each registered host to run yum updates on RHEL 8 servers.I think this part is more redhat related than foreman.

1 Like