Update from 3.17.0 to 3.17.1 fails with error

I am trying to update my katello 3.17.0 server to 3.17.1. After I update the package I run foreman-installer which fails with errors.

[ INFO 2020-12-15T18:13:02 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Starting to evaluate the resource (1184 of 2352)
[DEBUG 2020-12-15T18:13:02 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' won't be executed because of failed check 'refreshonly'
[DEBUG 2020-12-15T18:13:02 main]  Exec[pulpcore-manager collectstatic --noinput](provider=posix): Executing 'pulpcore-manager collectstatic --noinput'
[DEBUG 2020-12-15T18:13:02 main]  Executing with uid=pulp: 'pulpcore-manager collectstatic --noinput'
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: pulp: pulp_2to3_migration.app.plugin:INFO: Plugin pulp_deb is not installed in pulp3 therefore it will not be migrated from pul
p2
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: Traceback (most recent call last):
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/bin/pulpcore-manager", line 11, in <module>
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     load_entry_point('pulpcore==3.7.3', 'console_scripts', 'pulpcore-manager')()
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/pulpcore/app/manage.py", line 11, in manage
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     execute_from_command_line(sys.argv)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     utility.execute()
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/__init__.py", line 375, in execute
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.fetch_command(subcommand).run_from_argv(self.argv)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 323, in run_from_argv
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.execute(*args, **cmd_options)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/management/base.py", line 364, in execute
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     output = self.handle(*args, **options)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 188, in handle
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     collected = self.collect()
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 114, in collect
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     handler(path, prefixed_path, storage)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 342, in copy_file
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     if not self.delete_file(path, prefixed_path, source_storage):
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py", line 292, in delete_file
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     self.storage.delete(prefixed_path)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:   File "/usr/lib/python3.6/site-packages/django/core/files/storage.py", line 303, in delete
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns:     os.remove(name)
[ WARN 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]/returns: PermissionError: [Errno 13] Permission denied: '/var/lib/pulp/assets/admin/css/base.css'
[ERROR 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Failed to call refresh: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ERROR 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: 'pulpcore-manager collectstatic --noinput' returned 1 instead of one of [0]
[ INFO 2020-12-15T18:13:04 main]  /Stage[main]/Pulpcore::Static/Pulpcore::Admin[collectstatic --noinput]/Exec[pulpcore-manager collectstatic --noinput]: Evaluated in 2.01 seconds

If I understand the messages correctly, it’s running the command as user ‘pulp’. However, user ‘pulp’ cannot write file /var/lib/pulp/assets/admin/css/base.css as those files and directores belong ‘root’.

$ ls -al /var/lib/pulp/assets/
total 0
drwxr-xr-x.  6 pulp pulp  78 Aug 12 14:31 .
drwxrwxr-x. 13 pulp pulp 234 Nov 10 16:08 ..
drwxr-xr-x.  6 root root  51 May 12  2020 admin
drwxr-xr-x.  5 root root 193 May 12  2020 drf-yasg
drwxr-xr-x.  2 root root  49 Aug 12 14:31 import_export
drwxr-xr-x.  7 root root  63 May 12  2020 rest_framework
$ ls -al /var/lib/pulp/assets/admin/css/
total 100
drwxr-xr-x. 3 root root   237 Aug 12 14:31 .
drwxr-xr-x. 6 root root    51 May 12  2020 ..
-rw-r--r--. 1 root root  8440 May 12  2020 autocomplete.css
-rw-r--r--. 1 root root 16378 Aug 12 14:31 base.css
-rw-r--r--. 1 root root  6170 Aug 12 14:31 changelists.css
-rw-r--r--. 1 root root   412 Aug 12 14:31 dashboard.css
-rw-r--r--. 1 root root   423 May 12  2020 fonts.css
-rw-r--r--. 1 root root  8518 Aug 12 14:31 forms.css
-rw-r--r--. 1 root root  1233 Aug 12 14:31 login.css
-rw-r--r--. 1 root root 17944 Aug 12 14:31 responsive.css
-rw-r--r--. 1 root root  1921 Aug 12 14:31 responsive_rtl.css
-rw-r--r--. 1 root root  3808 Aug 12 14:31 rtl.css
drwxr-xr-x. 3 root root    21 May 12  2020 vendor
-rw-r--r--. 1 root root 10340 Aug 12 14:31 widgets.css

This is exactly what’s happening. I did write a hook that should have corrected the permissions:

However, we may have picked a newer puppet-pulpcore but forgot this hook.

A safe workaround is to chown the assets directory.

So I should be safe to do a

# chown -R root.root /var/lib/pulp/assets

before the foreman-installer run?

Yes. Would you also mind filing an issue for this so we can properly track and backport it?

Issue #31515 created.

Of course the correct command is

# chown -R pulp.pulp /var/lib/pulp/assets

to set the owner and group to pulp.