CentOS7 base repo incomplete on proxy

Hi,
Configured CentOS 7 and 8 products on our Foreman 2.2.1 server (scenario Katello) - and the usual repos (base, extras, appstream, epel, puppet … a.s.o.).
Kickstart works fine from synced content for both OS versions - next we deployed a proxy in a remote location. Went pretty smooth but when we try to kickstart from synced content on the proxy, it fails for CentOS 7. Found out that the content in the base repo on the proxy is incomplete. The pulp repository index only shows Packages and Repodata whereas the server also has LiveOS, Images, config.repo and .treeinfo.
Tried about everything just can’t find way to make the sync to copy the missing parts.
We even deleted and re-created the base_x86_64 repo hoping that would fix things but no luck.
No error/warnings seen during product/repo creation and proxy setup so I’m completely in the woods here - anyone an idea?
(Note: found out that one other repo also has a missing Packages directory on the pulp node - all others seem to be ok. 2 problematic ones on a total of 28 repositories)

I think this issue was fixed by Bug #31306: pxe files are not properly prefetched on smart proxy sync - Katello - Foreman and that’ll be coming out soon-ish in Katello 3.17.2

Would it be possible to post the outline of that ticket to the forum please?

@Polle basically kickstart related files were not getting synced to the smart proxy when they were supposed to. And at provisioning time the host would receive a 404 when trying to fetch them. The change ensures that the kickstart files are copied when syncing to the smart proxy in the first place

Well, I just upgraded to 3.18 and now things seem to be even worse, this is becoming a real nightmare …
On my central server, all packages have gone missing, I have like Libary->Staging->Production
If I e.g. browse to:
https:///pulp/repos//Library/CentOS_7_view/custom/CentOS_7/base_x86_64/
I get

  • .treeinfo
  • LiveOS
  • Packages
  • config.repo
  • images
  • repodata
    If I go down Packages, and click one of the letters/numbers it show a list of the packages starting with that letter or number but clicking any of those package names
    results in:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request *[GET /pulp/repos//Library/CentOS_7_view/custom/CentOS_7/base_x86_64/Packages/a/abrt-desktop-2.1.11-60.el7.centos.x86_64.rpm
Reason: Error reading from remote server

Another thing I noticed is that /var/lib/pulp contains a huge number of files with a name that starts with tmp - that doesn’t look right to me. Also, /var/lib/pulp/content/units/ is empty (not sure if that is correct or not - it’s not empty on the smart proxies)

Anyone ready to safe my life? (wouldn’t that be a great way to start the year :slight_smile: )

Can’t safe your life … sorry :slight_smile: But I can tell you I am chasing the same issues and the dev team from Katello is working on a different discussion to solve it.
https://community.theforeman.org/t/smart-proxy-synchronize-always-fails-at-the-same-repository/21371

Hi,

Thanks for pointing me to that thread - I quickly read it but I don’t see the relation with my findings, would you mind to comment a little?

Ahh, sorry I was pulling the wrong thread. It was an issue I encountered during upgrade of Katello to 3.18, when most of the repos stopped working and a lot of tmp files are piling up inside of the /var/lib/pulp directory.

This is the correct thread: https://community.theforeman.org/t/updating-foreman-2-2-1-to-2-3-1-and-katello-3-17-1-to-3-18-issue/21738/24?u=rbremer

I also commented on that repair issue - smells like some counter overflow but couldn’t put my finger on it (yet)