I am on 4.3 now, so migration was a long time ago. This error, though, is new. It does not happen with 4.2. So I guess it must be related with pulpcore 3.16 or something around that, that it tries to do a chmod on a file…
I have changed owners and the following sync just finished without problems.
@dralley Is this now a pulpcore or a katello problem? How do we get this fixed? Should foreman-installer do a recursive chown on the pulp directory? Or is this something pulpcore should do as part of the migrations? Or how? I guess, a recursive chown can take a long time depending on the number of files in the directories and the speed of the disk, thus it’s probably not an option to do this during each upgrade…
I think the best place for this would be in the Installer since it’s what maintains the overall state of the server’s configuration. To fix the issue a user would only need to re-run the foreman-installer. That would be better than a rake task since no documentation would be needed. Although, it makes me wonder why the other file permission steps from the Pulp 2 - Pulp 3 migration weren’t in the installer.
Has anyone tested updating the permissions to see if that does fix the issue?
I would not expect this to ever happen in a pure Pulp 3 world. It certainly is a problem something that happens in 2->3, but once you’re on Pulp 3 (and fixed the permissions once) it should never show up again. That’s why I’m very hesitant to add this to the installer (due to the significant cost on large installations).
I was sure we implemented this, but somehow I can’t find it so we missed this?
I’m having a hard time getting Pulp to create hard links during the migration process. Perhaps most people are migrating without hard links? It doesn’t make sense that more people aren’t hitting this issue. If artifacts have apache:pulp permissions, Katello will have failing sync tasks.
I suppose we don’t need to dig too deep into the cause if the fix is a one-time deal.
I did try flipping some artifacts to have apache:pulp ownership on Pulpcore 3.14 and I got sync issues. Maybe somehow the artifacts with bad ownership were ignored on 3.14, but that’s just a guess. Were those artifacts hard linked by any chance? Perhaps by now the hardlinks have been overwritten, but I’m not sure how that stuff works in Pulpcore.
@iballou To be honest, I was - and still am - affected by a migration issue, which could be traced back to this. Some number of files were never migrated over to Pulp 3 and I get tons of 500 errors when using those affected repos. I opened some issues in here but we never got a resolve. So maybe that is all related in a way.
Hi @rbremer, if you’re still having Pulp troubles relating to older posts of yours, feel free to create a new post or ping “katello” on the older ones. For example, I’m not sure if this is still causing you troubles, but I wonder if it would be solved by increasing the sync timeouts in the settings (that caused a ton of people issues before): Still fighting with the repair of repositories after migrating to Katello 3.18 (now on 4.1.2)
(I also don’t mind when people ping me directly on threads, sometimes I’m a bit busy and forget to browse the forums).
@ekohl Are you thinking of foreman-maintain prep-6-upgrade? I think this is only properly part of the satellite scenario of foreman-maintain… I forget the exact commands it executes, but it involves preparing the Pulp 2 files on disk for hard linking, which should be used before the content prepare and content switchover commands are run.