I think the best place for this would be in the Installer since it’s what maintains the overall state of the server’s configuration. To fix the issue a user would only need to re-run the foreman-installer. That would be better than a rake task since no documentation would be needed. Although, it makes me wonder why the other file permission steps from the Pulp 2 - Pulp 3 migration weren’t in the installer.
Has anyone tested updating the permissions to see if that does fix the issue?
@ekohl or @wbclark I’m curious what you would think about having the installer ensure that all permissions under /var/lib/pulp/media are owned by pulp. It would probably mean we’d just need another addition here? puppet-pulpcore/config.pp at master · theforeman/puppet-pulpcore · GitHub
This fixed the problem for me, many thanks!
I have a PR out that should fix this: Fixes #34631 - Recurse into artifacts folder to ensure owner is proper by ianballou · Pull Request #250 · theforeman/puppet-pulpcore · GitHub
I would not expect this to ever happen in a pure Pulp 3 world. It certainly is a problem something that happens in 2->3, but once you’re on Pulp 3 (and fixed the permissions once) it should never show up again. That’s why I’m very hesitant to add this to the installer (due to the significant cost on large installations).
I was sure we implemented this, but somehow I can’t find it so we missed this?
I’m having a hard time getting Pulp to create hard links during the migration process. Perhaps most people are migrating without hard links? It doesn’t make sense that more people aren’t hitting this issue. If artifacts have apache:pulp permissions, Katello will have failing sync tasks.
I suppose we don’t need to dig too deep into the cause if the fix is a one-time deal.
I thought it was a problem with pulpcore 3.16 trying to chmod on files, which didn’t happen with 3.14…
I did try flipping some artifacts to have apache:pulp ownership on Pulpcore 3.14 and I got sync issues. Maybe somehow the artifacts with bad ownership were ignored on 3.14, but that’s just a guess. Were those artifacts hard linked by any chance? Perhaps by now the hardlinks have been overwritten, but I’m not sure how that stuff works in Pulpcore.
My production server is still on 4.2 and pulpcore 3.14 and has no sync issues.
I have upgraded a clone of that server and it has this issue after the upgrade. So it has to be something between 4.2 and 4.3…
@iballou To be honest, I was - and still am - affected by a migration issue, which could be traced back to this. Some number of files were never migrated over to Pulp 3 and I get tons of 500 errors when using those affected repos. I opened some issues in here but we never got a resolve. So maybe that is all related in a way.
Hi @rbremer, if you’re still having Pulp troubles relating to older posts of yours, feel free to create a new post or ping “katello” on the older ones. For example, I’m not sure if this is still causing you troubles, but I wonder if it would be solved by increasing the sync timeouts in the settings (that caused a ton of people issues before): Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)
(I also don’t mind when people ping me directly on threads, sometimes I’m a bit busy and forget to browse the forums).
@iballou Thank you Ian! I will ping Katello on that one. Its an issue which bugs me for months now and I think it should be easily fixable, but I can’t figure out how to do it properly.
@ekohl Are you thinking of foreman-maintain prep-6-upgrade? I think this is only properly part of the satellite scenario of foreman-maintain… I forget the exact commands it executes, but it involves preparing the Pulp 2 files on disk for hard linking, which should be used before the content prepare and content switchover commands are run.