Problem:
/var/lib/pulp is jumping up 14gb on every sync/content-view publish after upgrade to katello 3.15, compared to holding steady at 150gb total before upgrade, it’s now going up so fast, that it’s filled the 250gb disk it was on, which was just increased by another 100gb while we try to find a solution to this.
Our process is to sync, publish, delete views older than 30 days once per day, which hasn’t changed since long before updating.
Expected outcome:
Size remains relatively the same.
Foreman and Proxy versions:
Foreman 2.0
Katello 3.15
Foreman and Proxy plugin versions:
Foreman 2.0
Katello 3.15
Distribution and version:
CentOS 7 - only missing this weekends update to puppetserver.
Other relevant data:
Not sure if it’s related to this problem, but another problem i’ve been having with Foreman since the upgrade is Package Profile update tasks have been failing with error:
ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERROR: duplicate key value violates unique constraint “katello_host_installed_packages_h_id_ip_id”
Will create a different thread for that if deemed unrelated.
It appears to be behaving as though repos are set to Immediate sync, instead of their actual setting of on-demand, if tthat helps at all.
I’ve gone through every repository and made sure they’re all set to on-demand, so they really should not be using up so much space on each sync
I didn’t check on the exact details, but I know there was debate about this. For repositories which don’t delete content (like Foreman’s RPM repos) on-demand is great. However, for repositories that do delete content (EPEL I think) you do get issues when the metadata in Pulp still refers to content that was deleted. Those were arguments in the discussion.
That’s all fine and dandy, but prior to katello 3.15 the disk-usage growth wasn’t an issue, after the update it’s eating GB’s for breakfast, lunch and dinner - Was there some change in katello 3.15 that affected this or am i hitting some bug?
After looking and testing in smaller parts, it looks like it has more to do with publishing the content-views rather than the repository sync itself
There isn’t a change I’m aware of that would cause this kind of behavior. I think it would be best to monitor what is actually changing in /var/lib/pulp to narrow down what may be causing this.
You can use du -s /var/lib/pulp/* to monitor general changes to the top level directories before and after a CV publish and drill down deeper into the directories. I would recommend running something like:
du -s /var/lib/pulp/* && du -s /var/lib/pulp/content/* && du -s /var/lib/pulp/content/units/*
and piping to a file before publishing a content view and then doing the same after a publish and diffing the files. Then you can do the same after the publish. You can continue with deeper directories, trying to get down to the actual file changes.
There is also a tool called inotifywait that might be helpful here. You can install with yum install inotify-tools and pipe inotifywait -m -r /var/lib/pulp/ to a file during a content view publish. Likely you will have to alter the max watches
Hopefully this info can help us narrow things down, you can upload any created output files here
Hi again, after running the du commands it appears that it’s only the published content that’s taking up all this space.
At the moment i’m about to hit disk capacity again, so my only solution right now is to just keep expanding the disk until it reaches all 30 versions that i keep, which will likely end up being 2-3 times larger than it was before upgrading to katello 3.15.
Narrowed it down further to see that it’s really just our Oracle Linux content views that are taking up all the published space, while centos 7 and 8 are taking 200mb-ish ( + 80~ish per lifecyle ), Oracle Linux repos are taking from 4-ish GB to 6.5 GB…
