I am on 4.3 now, so migration was a long time ago. This error, though, is new. It does not happen with 4.2. So I guess it must be related with pulpcore 3.16 or something around that, that it tries to do a chmod on a file…
I have changed owners and the following sync just finished without problems.
Hm, good point. I am def surprised to see this being a problem “suddenly” - I would have expected us to hit this A Long Time Ago. “Just upgraded from core/3.15 to core/3.16” is def a big hint, though.
pulpcore 3.14 to 3.16 to be exact.
@dralley Is this now a pulpcore or a katello problem? How do we get this fixed? Should foreman-installer do a recursive chown on the pulp directory? Or is this something pulpcore should do as part of the migrations? Or how? I guess, a recursive chown can take a long time depending on the number of files in the directories and the speed of the disk, thus it’s probably not an option to do this during each upgrade…
We’ve been discussing that, the best place to put it is most probably foreman-maintain content switchover. So, only once, right at the end when Pulp 2 is being shut off.
It could also be done as a separate manual step, but it’s better to avoid making the migration process any more complicated than it is already
That’s useful for those you are on 3.18 running pulp2, but doesn’t help those you are running pulp3 and are on 4.0+. This chmod problem for instance only occurs on 4.3/pulpcore 3.16.
At a minimum, katello upgrade instructions needed to be updated, e.g. to include a simple find to see if there is a potential issue:
# find /var/lib/pulp/media/ \! -user pulp -ls
and then instruct the user to do the chown.
But I kind of don’t like these upgrade instructions where users have to fix things, that got broken somehow, when it could be easily scripted…
I think the best place for this would be in the Installer since it’s what maintains the overall state of the server’s configuration. To fix the issue a user would only need to re-run the foreman-installer. That would be better than a rake task since no documentation would be needed. Although, it makes me wonder why the other file permission steps from the Pulp 2 - Pulp 3 migration weren’t in the installer.
Has anyone tested updating the permissions to see if that does fix the issue?
@ekohl or @wbclark I’m curious what you would think about having the installer ensure that all permissions under /var/lib/pulp/media are owned by pulp. It would probably mean we’d just need another addition here? puppet-pulpcore/config.pp at master · theforeman/puppet-pulpcore · GitHub
This fixed the problem for me, many thanks!
I have a PR out that should fix this: Fixes #34631 - Recurse into artifacts folder to ensure owner is proper by ianballou · Pull Request #250 · theforeman/puppet-pulpcore · GitHub
I would not expect this to ever happen in a pure Pulp 3 world. It certainly is a problem something that happens in 2->3, but once you’re on Pulp 3 (and fixed the permissions once) it should never show up again. That’s why I’m very hesitant to add this to the installer (due to the significant cost on large installations).
I was sure we implemented this, but somehow I can’t find it so we missed this?
I’m having a hard time getting Pulp to create hard links during the migration process. Perhaps most people are migrating without hard links? It doesn’t make sense that more people aren’t hitting this issue. If artifacts have apache:pulp permissions, Katello will have failing sync tasks.
I suppose we don’t need to dig too deep into the cause if the fix is a one-time deal.
I thought it was a problem with pulpcore 3.16 trying to chmod on files, which didn’t happen with 3.14…
I did try flipping some artifacts to have apache:pulp ownership on Pulpcore 3.14 and I got sync issues. Maybe somehow the artifacts with bad ownership were ignored on 3.14, but that’s just a guess. Were those artifacts hard linked by any chance? Perhaps by now the hardlinks have been overwritten, but I’m not sure how that stuff works in Pulpcore.
My production server is still on 4.2 and pulpcore 3.14 and has no sync issues.
I have upgraded a clone of that server and it has this issue after the upgrade. So it has to be something between 4.2 and 4.3…
@iballou To be honest, I was - and still am - affected by a migration issue, which could be traced back to this. Some number of files were never migrated over to Pulp 3 and I get tons of 500 errors when using those affected repos. I opened some issues in here but we never got a resolve. So maybe that is all related in a way.
Hi @rbremer, if you’re still having Pulp troubles relating to older posts of yours, feel free to create a new post or ping “katello” on the older ones. For example, I’m not sure if this is still causing you troubles, but I wonder if it would be solved by increasing the sync timeouts in the settings (that caused a ton of people issues before): Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)
(I also don’t mind when people ping me directly on threads, sometimes I’m a bit busy and forget to browse the forums).
@iballou Thank you Ian! I will ping Katello on that one. Its an issue which bugs me for months now and I think it should be easily fixable, but I can’t figure out how to do it properly.
@ekohl Are you thinking of foreman-maintain prep-6-upgrade? I think this is only properly part of the satellite scenario of foreman-maintain… I forget the exact commands it executes, but it involves preparing the Pulp 2 files on disk for hard linking, which should be used before the content prepare and content switchover commands are run.