Foreman broken after `dnf update`

This topic is currently also being discussed here: The woes of the httpd update (and others)
The problem is how foreman-installer (or rather, the backend Puppet modules) work differently to the underlying package systems.
Rerunning foreman-installer (or using the command mentioned in the linked thread) should solve your problems.

1 Like

This is the httpd config before the dnf update (restored VM snapshot):

# grep :443 /etc/httpd/conf.d/*.conf | grep -v -P '^[^:]+: *#'
/etc/httpd/conf.d/05-foreman-ssl.conf:<VirtualHost *:443>

There is only one listener on port :443, hence there is no conflict in the httpd config.

Before the dnf update, the file /etc/httpd/conf.d/ssl.conf did not exist.
It seems that dnf update added this file, presumably when updateing the httpd software package.

Thanks for this suggestion.
I have not tried it yet.
How is this done?
Can I run just foreman-installer without any arguments, and it will not change my installation?
Or do I need to add all the correct options, including katello and certificates?

Thanks
Toni

yes, this is correct, just without any arguments. It is highly recommended after every software update …

1 Like

Indeed! :slightly_smiling_face:

After reboot:

foreman-maintain service stop
foreman-installer
foreman-maintain service start

And Foreman is running again.

It has deleted the file, that the httpd package manager had added.

# ls /etc/httpd/conf.d/ssl.conf
ls: cannot access '/etc/httpd/conf.d/ssl.conf': No such file or directory

Does it make sense to create an empty file with that name, so that yum update httpd in the future will not create the same problem again?

The way it is currently handled, the issue is likely to reoccur.

1 Like

That doesn’t work. foreman-installer uses a httpd puppet module which purges the httpd configuration, i.e. any file not set up by foreman-installer will be removed. You can easily try: create an empty file and run foreman-installer. The empty file should disappear…

As pointed out in my thread The woes of the httpd update (and others) it’s not the only packages/files affected…

1 Like

Puppet modules handling of files is a bit suboptimal here, as it assumes it runs periodical and so it will fix things after a while or a reboot done after the upgrade. I do not like this normally, but with Foreman it can get annoying if you forget the installer run afterwards.

A solution would be managing the files with puppet as empty or better something like # Managed by puppet to prevent errors caused by package management. This could be done for all files probably causing errors as the puppet module is managing the configuration more in line with the Debian style. I think this would be accepted as a pull request if well explained, but it would add more complexity to an already very complex module.

1 Like

I ran into this before on RHEL8. It is because both the /etc/httpd/conf/ports.conf file and the default /etc/httpd/conf.d/ssl.conf file from mod_ssl both have Listen 443 in them and Apache no longer likes that apparently.

Comment out the Listen in the default ssl.conf file and it will fix the issue.

You can’t remove it from the ports.conf file, because that will be replaced everytime you run foreman-installer, but if you modify the ssl.conf file, the red hat rpm is smart enough to not overwrite it and just creates a .rpmnew file if the rpm version changes.

1 Like

FYI, I did some other testing.

Looks like foreman-installer removes the /etc/httpd/conf.d/ssl.conf file if it exists. That is a bad idea because the next time mod_ssl gets a patch, it will put it back.

1 Like

It doesn’t fix it. You have to run foreman-installer which will purge the conf.d directory.

As pointed out before: this doesn’t work. It’s incomplete to start with and it will be gone once you run foreman-installer.

See the linked thread above. It’s not the only file and not the only rpm…

1 Like

Your best bet is probably to enable the foreman package version lock; it exists for exactly this reason. There are a number of things besides the SSL configuration that can break with a dnf update.

foreman-maintain packages lock

will enable it once, but may later leave packages again unlocked in some scenarios.

foreman-installer --lock-package-versions

will make package locking the default.

Once you enable package locking, dnf update will tell you that X packages are locked and cannot be updated. That’s exactly what you want.

The way to update packages is through foreman-maintain, or through foreman-installer.

1 Like

This is a well know ‘issue’ with the recent apache update(s) - port 443 is implicit and the listener is started automatically. If you specify port 443 in the apache configuration it will try to start a second listener on the same port which results in an error. As indicated above, comment out the port 443 line in the apache configuration and restart the services.
Happened to me too …

Again: do not comment out anything. That’s just wrong. Run foreman-installer. It will configure httpd the way it is needed. If you don’t, it’s broken. Commenting out may help a little but other things may fail as well.

Do not mess with the configuration. Run foreman-installer.

Commening out is a bandaid for a single issue. foreman-installer is the complete solution.

1 Like

Some confusion mostly my bad: I was referring to /etc/httpd/conf.d/ssl.conf - what nixfu says is correct, a mod_ssl update will put it back. Having to run foreman-installer after each dnf update looks somewhat odd to me since those updates mostly do not update any foreman (/katello) modules.
The installer cleans it up indeed - I recently upgraded to katello 4.7.3 which does require running the installer and it did remove the /etc/httpd/conf.d/ssl.conf (with the 443 listener commented out …).
.

It’s what you have to do. ssl.conf is not the only file which may have been restored and causing conflicting configurations…

My 2ct, I deploy my OS updates with an Ansible playbook that will check whether or not the host runs Foreman/Satellite, if so, will always run foreman-installer before rebooting. IIRC this is even mentioned in the manual…

1 Like

Is there any update on resolving this conflict around ssl.conf on EL8

Running foreman-installer again seems a bit drastic to resolve the matter. it must be mentioned that if your installer had custom CLI options, they will also be needed again.

so far just moving the oftending ssl.conf out of the way, gets the server back up!

foreman needs to adjust its config file to match what el8 wants

That is incorrect. foreman-installer saves the cli options. Running foreman-installer without options runs everything with your latest configuration and thus restores any files which a package update might have restored.

There may be a lot of more issues which you don’t even see immediately. Run foreman-installer. It fixes the problems, whatever it may be, with whatever package you may have updated.

Don’t try to manually fix problems when a single foreman-installer will fix them all.

running foreman-installer again ! mine didnt, it didnt find the dir where I put modules in a custom location. I used customized options , so needed to fish out my CLI options for this to work.

On the actual BUG, I would consider this to be the following!

  1. use of the default server :: issue with foreman
  2. writing ssl.conf in the 1st place, issue with rocky linux. it should never add this to a running webserver

this update took out a number of my apache.webservers.
is is possible for the foreman to just have an exclude in the main config for this system delivered file?

I don’t what you did. Me and many others simply run foreman-installer and it (re)configures the system how it is supposed to be. If you break it, you have to solve it. If you use different locations configure them with foreman-installer and it’ll work the next time.

That’s is well known and noted a long time ago.

So you are running additional webservers on your foreman server? I don’t think that’s really supported.

If it breaks on other servers, it doesn’t really has anything to do with foreman but then general rpm installations.

It’s really confusing what you write. It’s unclear if you only run foreman or other web servers on the same server. It’s unclear if you mean effects on other web servers on other servers or not.

Generally, I can say, if you don’t break it, it’ll work. Run foreman-installer after any update on the foreman server to restore any potential changes by the rpm update. Don’t mix and break.