Katello becomes too busy to respond and display login during remote jobs.
Foreman and proxy version is 1.24.2
All Katello host running CentOS 7.7
Apache/Tomcat seems to get too busy to display webUI or receive output from remote executed jobs. I have set the concurrence level down to just 25 at a time, but still have timeouts.
Thanks for any advice.
Clay
Hi,
does it make a difference what kind of job you are executing? Say some subscription related task versus something like “echo true”?
Also, how many passenger instances do you have configured? This is usually found in /etc/httpd/conf.modules.d/passenger_extra.conf as “PassengerMaxInstancesPerApp” if you have not configured a seperate PassengerMaxInstances for Foreman. With 16 cores, I would set this to 14 or 16. As far as I remember, the default is 6 though. I don’t believe this is currently configurable through the installer, so it might get reset on an update.
Could you also share the output of passenger-status during that timeframe? Foreman “locking up” is usually related to the passenger queue getting to big, either because of calls that take very long or simply because Foreman gets “flooded”.
Regards
Hi Areyus,
Thanks for the reply. I think you are onto something. Looks like my configuration tuning has been reset sent my last update. Normally this happens when the remote job has more console output that it returns. I can run a reboot job on all nearly 600 hosts just fine, but if there is a lot of output like a patching even running YUM then it will hammer the system. I have written the patching template to be as non verbose as possible to help with the output. Below is the configuration of the passenger_extra.conf which looks to have reset back to the default 6 for PassengerMaxInstancesPerApp.
# The Passenger Apache module configuration file is being
# managed by Puppet and changes will be overwritten.
<IfModule mod_passenger.c>
PassengerMaxInstancesPerApp 6
PassengerMaxPoolSize 12
PassengerMaxPreloaderIdleTime 0
PassengerMaxRequestQueueSize 250
PassengerMaxRequests 10000
PassengerMinInstances 1
PassengerStatThrottleRate 120
</IfModule>
Thanks again.
Also, I receive this message when I run the passenger-status command.
sudo passenger-status
ERROR: Phusion Passenger doesn’t seem to be running.
Hello,
I have resolved the same issues few weeks ago. The problem is with module passenger.
If you upda
houps, finger mistake!!!
Hello,
I have resolved the same issues few weeks ago. The problem is with module passenger.
If you chek repository content
You can find mod_passenger in this repository. please checj that you have installed mod_passenger from foreman repository and kiss kool effect.
Too busy situation disappear
passenger-status works fine
Hi,
@jmma59650 it appears that my master Katello/smart-proxy has it installed, but my capsule smart-proxies do not. Does this need to install on all smart-proxies?
Installed Packages
mod_passenger.x86_64 4.0.53-4.el7 @epel
Thanks…
Hello,
What I can see is
mod_passenger.x86_64 4.0.53-4.el7 @epel from epel
The solution is to install package from Foreman repository on your master.
Try this command to identify the location and version
$> yum info mod_passenger --showduplicates
Rules
mod_passenger and foreman package should provide from same repository
Current version in foreman latest repo https://yum.theforeman.org/releases/latest/el7/x86_64/
Hi,
There are multiple versions available from EPEL and Foreman. It probably took the one from EPEL when I patched since it has the newer version number.
Installed Packages
Name : mod_passenger
Arch : x86_64
Version : 4.0.53
Release : 4.el7
Size : 1.4 M
Repo : installed
From repo : epel
Summary : Apache Module for Phusion Passenger
URL : https://www.phusionpassenger.com
License : Boost and BSD and BSD with advertising and MIT and zlib
Description : This package contains the pluggable Apache server module for Phusion Passenger®.
Available Packages
Name : mod_passenger
Arch : x86_64
Version : 4.0.18
Release : 10.12.el7
Size : 196 k
Repo : foreman/x86_64
Summary : Apache Module for Phusion Passenger
URL : http://www.modrails.com
License : Boost and BSD and BSD with advertising and MIT and zlib
Description : This package contains the pluggable Apache server module for Phusion Passenger™.
Name : mod_passenger
Arch : x86_64
Version : 4.0.53
Release : 4.el7
Size : 389 k
Repo : epel/x86_64
Summary : Apache Module for Phusion Passenger
URL : https://www.phusionpassenger.com
License : Boost and BSD and BSD with advertising and MIT and zlib
Description : This package contains the pluggable Apache server module for Phusion Passenger®.
Hi,
we have also experienced this behaviour in the past. I think it is due to foreman-installer setting that configuration, though have not found a way to configure this value through foreman-installer yet.
For the mod_passenger situation, I have no clue about that. Only thing I can add here is that 1.20 (our current production version, updating takes a lot of bureaucracy here…) as well as 1.22 (current version in testing) work fine with 4.0.53 from EPEL. But this might be a 1.24 “exclusive” problem?
Regards
Thanks for your ideas and insight. I have made a few adjustments based on your input. Perhaps in the future I will be more cautious with the upgrading of Foreman/Katello. I’ll focus more on keeping the operating system patched verse running the latest version of the management tools.
Regards,
Clay
Feature #27775: Add support for pre-defined tuning profiles for Foreman & Katello installations - Installer - Foreman might be relevant for you. The documentation has more information.
Note that we’re working on switching from Passenger to Puma in Foreman 2.1 due to issues with building Passenger in combination with SCLs (as you’ve seen here). This issue has affected us for some time already and is not 1.24 exclusive.