The woes of the httpd update (and others)

Support
1

Problem:
Today a number of updates arrived in AlmaLinux 8 BaseOS and AppStream. Among them, httpd. I didn’t really think and simply installed the updates and rebooted for the new kernel.

Of course, after the reboot, foreman didn’t come up. httpd does not start. The httpd update restored a number of configuration files which were deleted by foreman-installer. After I have noticed I have ran foreman-installer twice and then had to start httpd manually and ran foreman-installer another time to get it to finish without errors.

The problem is: configuration files in rpms are restored if they are missing. So if foreman-installer deletes /etc/httpd/conf.d/autoindex.conf, then an update of httpd restores that file. Now obviously the default configuration of httpd distributed causes conflicts with what foreman expects.

foreman-installer should not delete configuration files which are part of rpms but “neutralize” them, either by commenting out all lines inside or by leaving behind a zero-size file instead.Only then the change will survive a simple update.

Expected outcome:
An httpd update should break foreman.

Foreman and Proxy versions:
Foreman 3.5.1, Katello 4.7.2

Distribution and version:
AlmaLinux release 8.7

Other relevant data:

I ran rpm -Va and scanned the output for “missing” files on my main foreman server:

missing   c /etc/foreman/dynflow/worker.yml
missing   c /etc/httpd/conf.d/authnz_pam.conf
missing   c /etc/httpd/conf.d/autoindex.conf
missing   c /etc/httpd/conf.d/intercept_form_submit.conf
missing   c /etc/httpd/conf.d/lookup_identity.conf
missing   c /etc/httpd/conf.d/ssl.conf
missing   c /etc/httpd/conf.d/userdir.conf
missing   c /etc/httpd/conf.d/welcome.conf
missing   c /etc/httpd/conf.modules.d/00-base.conf
missing   c /etc/httpd/conf.modules.d/00-dav.conf
missing   c /etc/httpd/conf.modules.d/00-lua.conf
missing   c /etc/httpd/conf.modules.d/00-mpm.conf
missing   c /etc/httpd/conf.modules.d/00-optional.conf
missing   c /etc/httpd/conf.modules.d/00-proxy.conf
missing   c /etc/httpd/conf.modules.d/00-ssl.conf
missing   c /etc/httpd/conf.modules.d/00-systemd.conf
missing   c /etc/httpd/conf.modules.d/01-cgi.conf
missing   c /etc/httpd/conf.modules.d/10-auth_gssapi.conf
missing   c /etc/httpd/conf.modules.d/10-h2.conf
missing   c /etc/httpd/conf.modules.d/10-proxy_h2.conf
missing   c /etc/httpd/conf.modules.d/55-authnz_pam.conf
missing   c /etc/httpd/conf.modules.d/55-intercept_form_submit.conf
missing   c /etc/httpd/conf.modules.d/55-lookup_identity.conf
missing   c /etc/systemd/system/redis.service.d/limit.conf
missing     /etc/httpd/conf.d/README
missing     /etc/httpd/conf.modules.d/README

The sources of the files are the following rpms:

foreman-3.5.1-1.el8.noarch
httpd-2.4.37-51.module_el8.7.0+3405+9516b832.1.x86_64
httpd-filesystem-2.4.37-51.module_el8.7.0+3405+9516b832.1.noarch
mod_auth_gssapi-1.6.1-9.el8.x86_64
mod_authnz_pam-1.1.0-7.el8.x86_64
mod_http2-1.15.7-5.module_el8.6.0+2872+fe0ff7aa.x86_64
mod_intercept_form_submit-1.1.0-5.el8.x86_64
mod_lookup_identity-1.0.0-4.el8.x86_64
mod_ssl-2.4.37-51.module_el8.7.0+3405+9516b832.1.x86_64
redis-6.2.7-1.module_el8.7.0+3288+a82c1b48.x86_64

So basically any update of any of those rpms causes configuration files to be recreated which may break foreman or at least cause some unexpected configuration until foreman-installer is run again.

1 Like
2

this RPM behavior is among the reasons why foreman-maintain packages update runs the installer after performing package actions

1 Like
3

I didn’t know that command.

However, during my last update, foreman-installer did not start httpd:

2023-02-23 06:57:57 [DEBUG ] [configure] /Service[httpd]: Starting to evaluate the resource (1294 of 2058)
2023-02-23 06:57:57 [DEBUG ] [configure] /Service[httpd]: Skipping because of failed dependencies
2023-02-23 06:57:57 [DEBUG ] [configure] /Service[httpd]: Resource is being skipped, unscheduling all events
2023-02-23 06:57:57 [DEBUG ] [configure] /Service[httpd]: Evaluated in 0.00 seconds

I had to start it manually and then run foreman-installer again to get through.

Possibly, the reason for that is that usually my foreman server gets the updates from itself and thus, there was one error before:

2023-02-23 06:57:41 [ERROR ] [configure] Could not prefetch package provider 'dnfmodule': Execution of '/usr/bin/dnf module list -d 0 -e 1' returned 1: Error: Failed to download metadata for repo 'ORG_epel8_epel': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

On the other hand, that should be a reason not to start httpd unless there was really something missing.

4

Yes, self-registred setups are sadly full of pitfalls and thus neither tested nor recommended :slight_smile:

5

In Spacewalk, the Spacewalk server was able to patch itself without any issues.
It seems that Foreman took a step back in this regard.

I agree with @gvde that regular dnf update should not break the application. By not deleting rpm package files, updates should become more robust.