@gvde This is a migrated system, yes? I’m pretty sure these permissions are supposed to be uniform, and probably the files owned by apache came from Pulp 2… could be a migration bug, which would explain why I can’t reproduce.
Yes. Started with Katello 3.15 I think, i.e. with pulp2 and migrated eventually to pulp3.
That could be. The migration steps in the 3.18 docs only set up uniform group and group permissions on everything in the pulp2 content directory.
The artifact causing the problem has owner apache, i.e. if the pulpcore worker running as user pulp tries to change permissions on that file it’ll fail with ‘Operation not permitted’.
Looking at the dates of the files listed it seems everything should be owner/group pulp.pulp and permission 0600?
On my fresh install (never 2to3 migrated), the permissions look like this:
# ls -al /var/lib/pulp/media/artifact/00/
total 10372
drwxr-xr-x. 2 pulp pulp 4096 Mar 10 12:04 .
drwxr-xr-x. 257 pulp pulp 8192 Mar 10 12:04 ..
-rw-------. 1 pulp pulp 13040 Mar 10 12:03 655717ec2494a953b1c3f477bc86b080760402b72a926057076144aa8b9c41
-rw-------. 1 pulp pulp 1381996 Mar 10 11:53 9767e4fcb269500c8b2ebe0d2cf7f818b9774858019cbed6c325d7c3e73596
-rw-------. 1 pulp pulp 686508 Mar 10 11:54 b9e63e287f45300d4a4f59b6b88e25918443c932ae3e5845d5761ae193c530
-rw-------. 1 pulp pulp 8515480 Mar 10 12:04 d0de456134668f41bd9ea308a076bc0a6a805180445af8a37209d433f41efe
So yes, I would say you are right.
Our system was migrated from pulp2 to pulp3 and the majority of files are owned by apache rather than pulp and have the wrong permissions as well
# ls -la /var/lib/pulp/media/artifact/00
total 1203204
drwxr-xr-x 2 pulp pulp 32768 Feb 21 10:39 .
drwxr-xr-x 258 pulp pulp 4096 Jun 22 2021 ..
-rw-rw-r-- 1 apache pulp 42540 Mar 27 2020 0149f21cbf7579915c09ea2f659c357395c0da4b9a286c4de09df558098f7b
-rw------- 1 pulp pulp 125729 Oct 26 08:18 0225aca00d646eb11e18344d3b8d97eb59eaa30fadef186d2bc11524570c1c
-rw-rw-r-- 1 apache pulp 71284 Oct 21 2020 047c0995e145e37da62cc328a1566c494d3ef7298450494f23f9f1dfcaa543
-rw-rw-r-- 1 apache pulp 343320 Nov 18 2020 063f470db329bdbf0defd2b9622d72db94830b9c0a108b7646f6acb3316563
-rw-rw-r-- 1 apache pulp 364064 Mar 27 2020 0787002b6a9fa53a6818a92bbd82d75c0deec343540dc08a66ba630fa1e8e8
-rw-rw-r-- 1 apache pulp 34432 Mar 27 2020 085cd8333fafa109179e0d672bb352b10dedd4eef3876fc0835833bafc8de0
-rw-rw-r-- 1 apache pulp 5653596 Mar 27 2020 096c7131af7b7f1e7120bfb01651356a15d94f13e38494ccfa07e86af8fef6
-rw-rw-r-- 1 apache pulp 13720 Mar 27 2020 0a434f3dc587fcad9a5e566391daedd4f78b38f48b37c59103d0dd51f7610b
-rw-rw-r-- 1 apache pulp 46980 Dec 18 2020 0c783ad67a44fbc26df8ad81ecd49cdb80c72bac2e0468445fe1454d75a522
-rw-rw-r-- 1 apache pulp 72616 Mar 27 2020 0c82d22234b0c7817153d85ba7a4a66e2b335173ab0fff97178f18a2684dca
-rw-rw-r-- 1 apache pulp 76008 Oct 21 2020 0cb20f305398259ef2c1bef9919d6cd7abfae7492f3a75acf25c2ea26d266e
-rw-rw-r-- 1 apache pulp 95224 Nov 12 2020 0ea54e38c1c408fcf9548fe7d830eb475e867dc698d6f2f5aa00850c4bd67b
-rw-rw-r-- 1 apache pulp 98312 Apr 27 2020 0fd1719737652c9766ebfbb9d8acd63b4c854bb62f4738f3f8ee23eab188f7
-rw-rw-r-- 1 apache pulp 57156 Mar 27 2020 0ff36975b410e23a092f6ab0b0b159dab993325d9a7b45565424c6333d326b
-rw-rw-r-- 1 apache pulp 952356 Mar 27 2020 119100dc629a7357cf5847e231dc0523e24152e40ae7b9e85660d6e45bb650
-rw-rw-r-- 1 apache pulp 2631124 Mar 27 2020 136ed06ddba50e2a66e0241fcfe9085634c5dda843984e2dc9ed87b3928472
-rw-rw-r-- 1 apache pulp 56464 Mar 27 2020 164893c28d09401a9fd295082b4ee1e3d4201cede1b65c17fd08fe9322304c
-rw-rw-r-- 1 apache pulp 7995824 Oct 21 2020 167451bfc93d622557ee62240a7f90d35e11e21fac2e953f83a9325f3bca28
-rw-r--r-- 1 pulp pulp 182840 Aug 4 2021 16a44310db36215e198957df42b0bdf3fc6098673e2347e4bb789e61b06e6e
-rw-rw-r-- 1 apache pulp 65414 Feb 22 2021 1abafa991740446acf0ba79fa50c401b5555a081ce793f72c904f21e8ddddf
-rw-rw-r-- 1 apache pulp 740492 Oct 21 2020 1db24815b09f65a0c60cceb4f436cdf71dc9e2742e0af358b378ce4b628780
-rw-rw-r-- 1 apache pulp 289120 May 6 2020 21110620b0dce48b05ab0359cc22991cd701f058a1b8099e650c6272081873
-rw-rw-r-- 1 apache pulp 23508 Apr 27 2020 2126fe84a7207f9ee184f2acd2618da10c2be4c49ab3b0248bfe2eddb90199
-rw-rw-r-- 1 apache pulp 33788 Oct 21 2020 213fd7804aad49be9b2f9e0d949cc3a2c93a32b16ab958c4bf216eb78549c1
-rw-rw-r-- 1 apache pulp 115758573 Apr 23 2020 21cb8ca0978f52a8f0ff2d759eb695001a8632cdd8cd4b782de167d66603c3
-rw-rw-r-- 1 apache pulp 168484 Mar 27 2020 23356012efcb7eadd3be350868f8113ba41062ec20181d44a37d233e0e0001
-rw-rw-r-- 1 apache pulp 2353520 Oct 21 2020 243e95aec094be1112a35536ad5d570d5d59423e2ad1de9cb5beeef0eba075
-rw-rw-r-- 1 apache pulp 12756 Mar 27 2020 269ac5b12e96f874e9cee7f5090a80b97bdce1564a7a01819ae9f516559ab9
-rw-rw-r-- 1 apache pulp 13730 Feb 27 2021 2a53b5c093bf8f171866dbd27aa667bf2407a400706ebd5365bc9b268d16ab
-rw-rw-r-- 1 apache pulp 81540 Apr 29 2020 2b73881adce7c1097c6eacae3126bbc5a8291bcae3287bfd5c6ebd54d3df8a
-rw-rw-r-- 1 apache pulp 30012 Mar 27 2020 2c83c87fbc85eea1404fbacb3a4afcea547a55933348b5d5d015dded765a9b
-rw-rw-r-- 1 apache pulp 58720 Mar 27 2020 2e52327d7f3d35a74e802763af005e6f3e7c7296601da6c99feaa8234aff16
-rw-r--r-- 1 pulp pulp 99708 Feb 21 10:39 2ea2460b9e864c85bf1ed8a108e4203f562116643f19001914afc7b07ff4d5
So a recursive chown on the artifact folder should straighten out the problem with the owner which would allow pulpcore-workers to do chmod on files…
# chown -hR pulp.pulp /var/lib/pulp/media/artifact/
The permissions on files (600, 644, 664) shouldn’t be an issue here…
pulp-2to3-migration tries to make hardlinks for artifacts if it can, in order to not double the storage needed when migrating. That’s how you end up with pulp2 perms/users on pulp3 content post-migration, is my guess - your pulp2 and pulp3 storage live on the same volume. We prob need a post-migration step to do the suggested chown; you don’t want to do it until you’re ready to “turn off” pulp2, so as not to impact pulp2 Doing It’s Thing while you’re still preparing Pulp3.
1 Like
I am on 4.3 now, so migration was a long time ago. This error, though, is new. It does not happen with 4.2. So I guess it must be related with pulpcore 3.16 or something around that, that it tries to do a chmod on a file…
I have changed owners and the following sync just finished without problems.
Hm, good point. I am def surprised to see this being a problem “suddenly” - I would have expected us to hit this A Long Time Ago. “Just upgraded from core/3.15 to core/3.16” is def a big hint, though.
pulpcore 3.14 to 3.16 to be exact.
@dralley Is this now a pulpcore or a katello problem? How do we get this fixed? Should foreman-installer do a recursive chown on the pulp directory? Or is this something pulpcore should do as part of the migrations? Or how? I guess, a recursive chown can take a long time depending on the number of files in the directories and the speed of the disk, thus it’s probably not an option to do this during each upgrade…
We’ve been discussing that, the best place to put it is most probably foreman-maintain content switchover. So, only once, right at the end when Pulp 2 is being shut off.
It could also be done as a separate manual step, but it’s better to avoid making the migration process any more complicated than it is already
That’s useful for those you are on 3.18 running pulp2, but doesn’t help those you are running pulp3 and are on 4.0+. This chmod problem for instance only occurs on 4.3/pulpcore 3.16.
At a minimum, katello upgrade instructions needed to be updated, e.g. to include a simple find to see if there is a potential issue:
# find /var/lib/pulp/media/ \! -user pulp -ls
and then instruct the user to do the chown.
But I kind of don’t like these upgrade instructions where users have to fix things, that got broken somehow, when it could be easily scripted…
I think the best place for this would be in the Installer since it’s what maintains the overall state of the server’s configuration. To fix the issue a user would only need to re-run the foreman-installer. That would be better than a rake task since no documentation would be needed. Although, it makes me wonder why the other file permission steps from the Pulp 2 - Pulp 3 migration weren’t in the installer.
Has anyone tested updating the permissions to see if that does fix the issue?
@ekohl or @wbclark I’m curious what you would think about having the installer ensure that all permissions under /var/lib/pulp/media are owned by pulp. It would probably mean we’d just need another addition here? puppet-pulpcore/config.pp at master · theforeman/puppet-pulpcore · GitHub
This fixed the problem for me, many thanks!
1 Like
I have a PR out that should fix this: Fixes #34631 - Recurse into artifacts folder to ensure owner is proper by ianballou · Pull Request #250 · theforeman/puppet-pulpcore · GitHub
I would not expect this to ever happen in a pure Pulp 3 world. It certainly is a problem something that happens in 2->3, but once you’re on Pulp 3 (and fixed the permissions once) it should never show up again. That’s why I’m very hesitant to add this to the installer (due to the significant cost on large installations).
I was sure we implemented this, but somehow I can’t find it so we missed this?
I’m having a hard time getting Pulp to create hard links during the migration process. Perhaps most people are migrating without hard links? It doesn’t make sense that more people aren’t hitting this issue. If artifacts have apache:pulp permissions, Katello will have failing sync tasks.
I suppose we don’t need to dig too deep into the cause if the fix is a one-time deal.
I thought it was a problem with pulpcore 3.16 trying to chmod on files, which didn’t happen with 3.14…