Foreman upgrade from 3.2 to 3.3 fails due to apache2 errors

Support
1

Problem:
While upgrading from 3.2 to 3.3 using the foreman installer failed the first time in a row of 8 updates. I think this is a good quote, but of course not wanted.

Expected outcome: No errors :slight_smile: I would like to solve the issue to get it running again, but also check if there is a bug in the update procedure. @evgeni :wink:

Foreman and Proxy versions: 3.2

Foreman and Proxy plugin versions: 3.2

Distribution and version:
Ubuntu 20.04 LTS

Other relevant data:

[11:24]root@rmvm069:~# foreman-installer --foreman-proxy-dhcp true --foreman-proxy-dns true --enable-foreman-plugin-remote-execution
2022-10-21 11:25:23 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-10-21 11:25:33 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-10-21 11:25:33 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-10-21 11:25:36 [NOTICE] [configure] Starting system configuration.
2022-10-21 11:25:52 [NOTICE] [configure] 250 configuration steps out of 1462 steps complete.
2022-10-21 11:25:53 [NOTICE] [configure] 500 configuration steps out of 1464 steps complete.
2022-10-21 11:25:53 [NOTICE] [configure] 750 configuration steps out of 1475 steps complete.
2022-10-21 11:25:54 [NOTICE] [configure] 1000 configuration steps out of 1641 steps complete.
2022-10-21 11:26:33 [ERROR ] [configure] Systemd start for apache2 failed!
2022-10-21 11:26:33 [ERROR ] [configure] journalctl log for apache2:
2022-10-21 11:26:33 [ERROR ] [configure] -- Logs begin at Fri 2022-08-05 10:55:01 UTC, end at Fri 2022-10-21 11:26:33 UTC. --
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: Starting The Apache HTTP Server...
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354833]: [Fri Oct 21 11:26:33.071162 2022] [so:warn] [pid 354833:tid 140394669222976] AH01574: module passenger_module is already loaded, skipping
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354833]: AH00534: apache2: Configuration error: More than one MPM loaded.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354825]: Action 'start' failed.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354825]: The Apache error log may have more information.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: apache2.service: Failed with result 'exit-code'.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: Failed to start The Apache HTTP Server.
2022-10-21 11:26:33 [ERROR ] [configure] /Stage[main]/Apache::Service/Service[httpd]/ensure: change from 'stopped' to 'running' failed: Systemd start for apache2 failed!
2022-10-21 11:26:33 [ERROR ] [configure] journalctl log for apache2:
2022-10-21 11:26:33 [ERROR ] [configure] -- Logs begin at Fri 2022-08-05 10:55:01 UTC, end at Fri 2022-10-21 11:26:33 UTC. --
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: Starting The Apache HTTP Server...
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354833]: [Fri Oct 21 11:26:33.071162 2022] [so:warn] [pid 354833:tid 140394669222976] AH01574: module passenger_module is already loaded, skipping
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354833]: AH00534: apache2: Configuration error: More than one MPM loaded.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354825]: Action 'start' failed.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 apachectl[354825]: The Apache error log may have more information.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: apache2.service: Control process exited, code=exited, status=1/FAILURE
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: apache2.service: Failed with result 'exit-code'.
2022-10-21 11:26:33 [ERROR ] [configure] Oct 21 11:26:33 rmvm069 systemd[1]: Failed to start The Apache HTTP Server.
2022-10-21 11:26:33 [NOTICE] [configure] 1250 configuration steps out of 1641 steps complete.
2022-10-21 11:26:34 [NOTICE] [configure] 1500 configuration steps out of 1852 steps complete.
2022-10-21 11:27:00 [NOTICE] [configure] 1750 configuration steps out of 1864 steps complete.
2022-10-21 11:27:07 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.
2

If you are just performing an update, it looks a little strange you also include other parameters to foreman-installer at the same time. I personally always use only these commands after an package update:

foreman-maintain service stop && foreman-installer
3

Foreman 3.3 was when we switched to MPM Event by default (https://github.com/theforeman/foreman-installer/pull/753), but it seems your system is still referring to Passenger and Prefork.

Can you show us a grep -E '(apache|passenger)' /etc/foreman-installer/scenarios.d/foreman-answers.yaml?

4

Oh, and a grep -RiE '(mpm_|passenger)' /etc/apache2 too, please.

5

Of course I can!

[12:15]root@rmvm069:~# grep -E '(apache|passenger)' /etc/foreman-installer/scenarios.d/foreman-answers.yaml
  apache: true
[12:15]root@rmvm069:~# grep -RiE '(mpm_|passenger)' /etc/apache2
/etc/apache2/mods-available/mpm_prefork.conf:<IfModule mpm_prefork_module>
/etc/apache2/mods-available/worker.load:LoadModule mpm_worker_module /usr/lib/apache2/modules/mod_mpm_worker.so
/etc/apache2/mods-available/zpassenger.load:LoadModule passenger_module /usr/lib/apache2/modules/mod_passenger.so
/etc/apache2/mods-available/mpm_event.load:# Conflicts: mpm_worker mpm_prefork
/etc/apache2/mods-available/mpm_event.load:LoadModule mpm_event_module /usr/lib/apache2/modules/mod_mpm_event.so
/etc/apache2/mods-available/http2.conf:# mod_http2 doesn't work with mpm_prefork
/etc/apache2/mods-available/http2.conf:<IfModule !mpm_prefork>
/etc/apache2/mods-available/mpm_event.conf:<IfModule mpm_event_module>
/etc/apache2/mods-available/mpm_worker.load:# Conflicts: mpm_event mpm_prefork
/etc/apache2/mods-available/mpm_worker.load:LoadModule mpm_worker_module /usr/lib/apache2/modules/mod_mpm_worker.so
/etc/apache2/mods-available/worker.conf:<IfModule mpm_worker_module>
/etc/apache2/mods-available/passenger.conf:# The Passenger Apache module configuration file is being
/etc/apache2/mods-available/passenger.conf:<IfModule mod_passenger.c>
/etc/apache2/mods-available/passenger.conf:  PassengerDefaultRuby "/usr/bin/ruby"
/etc/apache2/mods-available/passenger.conf:  PassengerMaxInstancesPerApp 6
/etc/apache2/mods-available/passenger.conf:  PassengerMaxPoolSize 12
/etc/apache2/mods-available/passenger.conf:  PassengerMaxPreloaderIdleTime 0
/etc/apache2/mods-available/passenger.conf:  PassengerMaxRequestQueueSize 250
/etc/apache2/mods-available/passenger.conf:  PassengerMaxRequests 10000
/etc/apache2/mods-available/passenger.conf:  PassengerMinInstances 1
/etc/apache2/mods-available/passenger.conf:  PassengerRoot "/usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini"
/etc/apache2/mods-available/passenger.conf:  PassengerStatThrottleRate 120
/etc/apache2/mods-available/mpm_prefork.load:# Conflicts: mpm_event mpm_worker
/etc/apache2/mods-available/mpm_prefork.load:LoadModule mpm_prefork_module /usr/lib/apache2/modules/mod_mpm_prefork.so
/etc/apache2/mods-available/mpm_worker.conf:<IfModule mpm_worker_module>
/etc/apache2/mods-available/passenger.load:LoadModule passenger_module /usr/lib/apache2/modules/mod_passenger.so
/etc/apache2/sites-enabled/99-foreman-alias.conf:  PassengerRuby /usr/bin/foreman-ruby
/etc/apache2/sites-enabled/99-foreman-alias.conf:  PassengerAppRoot /usr/share/foreman
/etc/apache2/sites-enabled/99-foreman-alias.conf:  PassengerMinInstances 1
/etc/apache2/sites-enabled/99-foreman-alias.conf:  PassengerStartTimeout 600
/etc/apache2/mods-enabled/worker.load:LoadModule mpm_worker_module /usr/lib/apache2/modules/mod_mpm_worker.so
/etc/apache2/mods-enabled/zpassenger.load:LoadModule passenger_module /usr/lib/apache2/modules/mod_passenger.so
/etc/apache2/mods-enabled/worker.conf:<IfModule mpm_worker_module>
/etc/apache2/mods-enabled/passenger.conf:# The Passenger Apache module configuration file is being
/etc/apache2/mods-enabled/passenger.conf:<IfModule mod_passenger.c>
/etc/apache2/mods-enabled/passenger.conf:  PassengerDefaultRuby "/usr/bin/ruby"
/etc/apache2/mods-enabled/passenger.conf:  PassengerMaxInstancesPerApp 6
/etc/apache2/mods-enabled/passenger.conf:  PassengerMaxPoolSize 12
/etc/apache2/mods-enabled/passenger.conf:  PassengerMaxPreloaderIdleTime 0
/etc/apache2/mods-enabled/passenger.conf:  PassengerMaxRequestQueueSize 250
/etc/apache2/mods-enabled/passenger.conf:  PassengerMaxRequests 10000
/etc/apache2/mods-enabled/passenger.conf:  PassengerMinInstances 1
/etc/apache2/mods-enabled/passenger.conf:  PassengerRoot "/usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini"
/etc/apache2/mods-enabled/passenger.conf:  PassengerStatThrottleRate 120
/etc/apache2/mods-enabled/passenger.load:LoadModule passenger_module /usr/lib/apache2/modules/mod_passenger.so
/etc/apache2/sites-available/99-foreman-alias.conf:  PassengerRuby /usr/bin/foreman-ruby
/etc/apache2/sites-available/99-foreman-alias.conf:  PassengerAppRoot /usr/share/foreman
/etc/apache2/sites-available/99-foreman-alias.conf:  PassengerMinInstances 1
/etc/apache2/sites-available/99-foreman-alias.conf:  PassengerStartTimeout 600
6

What is this file, that’s nothing we deploy using the installer?

Are you running the installer after each upgrade? Because the apache config doesn’t look like it should look like on a 3.3.