Web Components Stop Responding

Jason_B_Nance · January 6, 2017, 4:41pm

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as well) I run into the situation where occasionally the web interface and API stop responding. Attempts to load any page, click on any button/link/whatever results in the browser just spinning and eventually timing out. The only log message printed is:

==&gt; /var/log/httpd/foreman-ssl_error_ssl.log &lt;==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client 172.16.246.240:56536] AH02227: Failed to set r-&gt;user to &#39;SSL_CLIENT_S_DN_CN&#39;

(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb <fqdn> fails with:

Could not send facts to Foreman: Net::ReadTimeout

(no log messages are printed)

A 'yum install' on a client is able to download the package, but then hangs on "Uploading Package Profile" (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I'm not sure if the tomcat restart matters, as I did it first and nothing changed but after I restarted httpd - which took much longer than normal - is when everything came back to life. There were about 55 connections to httpd at the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

Thanks,

j

Jason_B_Nance · January 6, 2017, 9:15pm

A bit more information - the tomcat restart makes no difference (the httpd restart is what fixes the issue). And, this is happening every time I provision a host. O.O

j

···

----- Original Message ----- From: "Foreman Users" To: "Foreman Users" Sent: Friday, January 6, 2017 10:41:55 AM Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as well) I run into the situation where occasionally the web interface and API stop responding. Attempts to load any page, click on any button/link/whatever results in the browser just spinning and eventually timing out. The only log message printed is:

==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client 172.16.246.240:56536] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'

(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb fails with:

Could not send facts to Foreman: Net::ReadTimeout

(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then hangs on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not sure if the tomcat restart matters, as I did it first and nothing changed but after I restarted httpd - which took much longer than normal - is when everything came back to life. There were about 55 connections to httpd at the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

Thanks,

j

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

stbenjam · January 11, 2017, 12:13am

> From: "'Jason B. Nance' via Foreman users" <foreman-users@googlegroups.com>
> To: "Foreman Users" <foreman-users@googlegroups.com>
> Sent: Friday, January 6, 2017 11:41:55 AM
> Subject: [foreman-users] Web Components Stop Responding
>
> Hello Everyone,
>
> On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as
> well) I run into the situation where occasionally the web interface and API
> stop responding. Attempts to load any page, click on any
> button/link/whatever results in the browser just spinning and eventually
> timing out. The only log message printed is:
>
> ==> /var/log/httpd/foreman-ssl_error_ssl.log <==
> [Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
> 172.16.246.240:56536] AH02227: Failed to set r->user to
> 'SSL_CLIENT_S_DN_CN'
>
> (I see this frequently under normal conditions)
>
> The system is not under heavy loads during this freeze. Load average is
> about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing
> noticeable work is mongodb.
>
> Puppet runs on clients also fail during this time because /etc/puppet/node.rb
> <fqdn> fails with:
>
> Could not send facts to Foreman: Net::ReadTimeout
>
> (no log messages are printed)
>
> A 'yum install' on a client is able to download the package, but then hangs
> on "Uploading Package Profile" (and eventually times out and skips it).
>
> If I restart [tomcat and] httpd things start responding again. I'm not sure
> if the tomcat restart matters, as I did it first and nothing changed but
> after I restarted httpd - which took much longer than normal - is when
> everything came back to life. There were about 55 connections to httpd at
> the time, nearly all in CLOSE_WAIT.
>
> Anyone have ideas as to what may be happening?

This sounds an awful lot like what we fixed in Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.

Do you have a lot of puppet clients checking in? You could try to tweak the passenger
settings as described here in the Red Hat BZ (will be the default in Foreman 1.14), and
see if that helps:

https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11

Stephen

···

----- Original Message -----

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

tbrisker · January 7, 2017, 1:13pm

Hi Jason,
Could you please turn on debug logging on foreman (
Foreman :: Manual) and attach the
output in production.log during the host provisioning? That might give us a
clue as to what is failing.

···

On Fri, Jan 6, 2017 at 11:15 PM, 'Jason B. Nance' via Foreman users < foreman-users@googlegroups.com> wrote:

A bit more information - the tomcat restart makes no difference (the httpd
restart is what fixes the issue). And, this is happening every time I
provision a host. O.O

j

----- Original Message -----
From: “Foreman Users” foreman-users@googlegroups.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Friday, January 6, 2017 10:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions
as well) I run into the situation where occasionally the web interface and
API stop responding. Attempts to load any page, click on any
button/link/whatever results in the browser just spinning and eventually
timing out. The only log message printed is:
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
172.16.246.240:56536] AH02227: Failed to set r->user to
’SSL_CLIENT_S_DN_CN’

(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is
about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process
doing noticeable work is mongodb.

Puppet runs on clients also fail during this time because
/etc/puppet/node.rb fails with:
Could not send facts to Foreman: Net::ReadTimeout
(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then
hangs on “Uploading Package Profile” (and eventually times out and skips
it).

If I restart [tomcat and] httpd things start responding again. I’m not
sure if the tomcat restart matters, as I did it first and nothing changed
but after I restarted httpd - which took much longer than normal - is when
everything came back to life. There were about 55 connections to httpd at
the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

Jason_B_Nance · January 12, 2017, 8:22pm

Not sure what was going on. I typically have fewer than 10 hosts connected to this lab install which was hanging. I updated it to 3.2.2/1.13.3 on Monday and have debug logging on and haven't seen the issue reappear.

shrug

···

----- Original Message ----- From: "Stephen Benjamin" To: "Foreman Users" Sent: Tuesday, January 10, 2017 6:13:27 PM Subject: Re: [foreman-users] Web Components Stop Responding

----- Original Message -----

From: “‘Jason B. Nance’ via Foreman users” foreman-users@googlegroups.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Friday, January 6, 2017 11:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as
well) I run into the situation where occasionally the web interface and API
stop responding. Attempts to load any page, click on any
button/link/whatever results in the browser just spinning and eventually
timing out. The only log message printed is:
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
172.16.246.240:56536] AH02227: Failed to set r->user to
'SSL_CLIENT_S_DN_CN'
(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is
about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing
noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb
fails with:
Could not send facts to Foreman: Net::ReadTimeout
(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then hangs
on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not sure
if the tomcat restart matters, as I did it first and nothing changed but
after I restarted httpd - which took much longer than normal - is when
everything came back to life. There were about 55 connections to httpd at
the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

This sounds an awful lot like what we fixed in Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.

Do you have a lot of puppet clients checking in? You could try to tweak the passenger
settings as described here in the Red Hat BZ (will be the default in Foreman 1.14), and
see if that helps:

https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11

Stephen

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Jason_B_Nance · January 10, 2017, 9:49pm

I have debugging on now and am just waiting for it to happen again.

Will report back then.

j

···

From: "Tomer Brisker" To: "Foreman Users" Sent: Saturday, January 7, 2017 7:13:05 AM Subject: Re: [foreman-users] Web Components Stop Responding

Hi Jason,
Could you please turn on debug logging on foreman ( [ Foreman :: Manual | Foreman :: Manual ] ) and attach the output in production.log during the host provisioning? That might give us a clue as to what is failing.

On Fri, Jan 6, 2017 at 11:15 PM, ‘Jason B. Nance’ via Foreman users < [ mailto:foreman-users@googlegroups.com | foreman-users@googlegroups.com ] > wrote:

A bit more information - the tomcat restart makes no difference (the httpd restart is what fixes the issue). And, this is happening every time I provision a host. O.O

j

----- Original Message -----
From: “Foreman Users” < [ mailto:foreman-users@googlegroups.com | foreman-users@googlegroups.com ] >
To: “Foreman Users” < [ mailto:foreman-users@googlegroups.com | foreman-users@googlegroups.com ] >
Sent: Friday, January 6, 2017 10:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as well) I run into the situation where occasionally the web interface and API stop responding. Attempts to load any page, click on any button/link/whatever results in the browser just spinning and eventually timing out. The only log message printed is:

==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client [ http://172.16.246.240:56536/ | 172.16.246.240:56536 ] ] AH02227: Failed to set r->user to ‘SSL_CLIENT_S_DN_CN’

(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb fails with:

Could not send facts to Foreman: Net::ReadTimeout

(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then hangs on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not sure if the tomcat restart matters, as I did it first and nothing changed but after I restarted httpd - which took much longer than normal - is when everything came back to life. There were about 55 connections to httpd at the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

Thanks,

j

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [ mailto:foreman-users%2Bunsubscribe@googlegroups.com | foreman-users+unsubscribe@googlegroups.com ] .
To post to this group, send email to [ mailto:foreman-users@googlegroups.com | foreman-users@googlegroups.com ] .
Visit this group at [ https://groups.google.com/group/foreman-users | https://groups.google.com/group/foreman-users ] .
For more options, visit [ https://groups.google.com/d/optout | https://groups.google.com/d/optout ] .

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to [ mailto:foreman-users+unsubscribe@googlegroups.com | foreman-users+unsubscribe@googlegroups.com ] .
To post to this group, send email to [ mailto:foreman-users@googlegroups.com | foreman-users@googlegroups.com ] .
Visit this group at [ https://groups.google.com/group/foreman-users | https://groups.google.com/group/foreman-users ] .
For more options, visit [ https://groups.google.com/d/optout | https://groups.google.com/d/optout ] .

Jason_B_Nance · January 16, 2017, 4:36pm

Okay, this happened again today, but this time I was publishing a new version of a content view. Apache stopped responding to any/all requests and isn't logging anything, either. However, my 'set logging to debug' stuff was reverted by the upgrade I did last week and I didn't realize it (until now). SMH So I guess I'll restart and wait for the next time it happens.

Netstat says:

netstat -an | egrep '80|443'

tcp 0 0 0.0.0.0:8008 tcp 1 0 172.16.246.31:54206 tcp 1 0 172.16.246.31:49617 tcp 1 0 172.16.246.31:49616 tcp 1 0 172.16.246.31:59602 tcp 0 0 127.0.0.1:59480 tcp 1 0 172.16.246.31:54266 tcp 0 0 172.16.246.31:5671 tcp 0 0 127.0.0.1:27017 tcp 0 0 172.16.246.31:56280 tcp 1 0 172.16.246.31:36114 tcp 0 0 172.16.246.31:53508 tcp 0 0 172.16.246.31:53514 tcp 1 0 172.16.246.31:59603 tcp 0 0 172.16.246.31:53324 tcp 0 0 172.16.246.31:53340 tcp6 0 0 :::80 tcp6 0 0 :::8080 tcp6 0 0 :::443 tcp6 0 0 :::8443 tcp6 0 0 127.0.0.1:8005 tcp6 0 0 :::8009 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 0 0 172.16.246.31:443 tcp6 32 0 172.16.246.31:443 udp 0 0 0.0.0.0:51680 0.0.0.0:* LISTEN
184.51.114.249:80 CLOSE_WAIT
207.99.69.162:80 CLOSE_WAIT
207.99.69.162:80 CLOSE_WAIT
207.99.69.162:80 CLOSE_WAIT
127.0.0.1:27017 ESTABLISHED
184.51.114.249:80 CLOSE_WAIT
172.16.246.31:56280 ESTABLISHED
127.0.0.1:59480 ESTABLISHED
172.16.246.31:5671 ESTABLISHED
184.51.114.242:80 CLOSE_WAIT
172.16.246.31:443 ESTABLISHED
172.16.246.31:443 ESTABLISHED
207.99.69.162:80 CLOSE_WAIT
172.16.246.31:443 ESTABLISHED
172.16.246.31:443 ESTABLISHED
:::* LISTEN
:::* LISTEN
:::* LISTEN
:::* LISTEN
:::* LISTEN
:::* LISTEN
172.16.246.31:53482 CLOSE_WAIT
172.16.246.31:53170 CLOSE_WAIT
172.16.246.240:58842 ESTABLISHED
172.16.246.31:53476 CLOSE_WAIT
172.16.246.31:53488 CLOSE_WAIT
172.16.246.240:58778 ESTABLISHED
172.16.246.31:53376 CLOSE_WAIT
172.16.246.31:53324 ESTABLISHED
172.16.246.31:53340 ESTABLISHED
172.16.246.31:53510 CLOSE_WAIT
172.16.246.31:53256 CLOSE_WAIT
172.16.246.31:53146 CLOSE_WAIT
172.16.246.31:53502 CLOSE_WAIT
172.16.246.240:58774 ESTABLISHED
172.16.246.31:53304 CLOSE_WAIT
172.16.246.31:53450 CLOSE_WAIT
172.16.246.31:53140 CLOSE_WAIT
172.16.246.31:53486 CLOSE_WAIT
172.16.246.31:53456 CLOSE_WAIT
172.16.246.31:53454 CLOSE_WAIT
172.16.246.31:53492 CLOSE_WAIT
172.16.246.31:53370 CLOSE_WAIT
172.16.246.31:53498 CLOSE_WAIT
172.16.246.31:53156 CLOSE_WAIT
172.16.246.240:58848 ESTABLISHED
172.16.246.240:58854 ESTABLISHED
172.16.246.31:53472 CLOSE_WAIT
172.16.246.31:53506 CLOSE_WAIT
172.16.246.31:53528 CLOSE_WAIT
172.16.246.31:53344 CLOSE_WAIT
172.16.246.31:53518 CLOSE_WAIT
172.16.246.31:53500 CLOSE_WAIT
172.16.246.31:53466 CLOSE_WAIT
172.16.246.31:53504 CLOSE_WAIT
172.16.246.31:53364 CLOSE_WAIT
172.16.246.240:58838 ESTABLISHED
172.16.246.31:53428 CLOSE_WAIT
172.16.246.31:53470 CLOSE_WAIT
172.16.246.31:53494 CLOSE_WAIT
172.16.246.31:53530 CLOSE_WAIT
172.16.246.31:53378 CLOSE_WAIT
172.16.246.31:53508 ESTABLISHED
172.16.246.31:53514 ESTABLISHED
172.16.246.31:53218 CLOSE_WAIT
0.0.0.0:*

···

----- Original Message ----- From: "Jason Nance" To: "Foreman Users" Sent: Thursday, January 12, 2017 2:22:51 PM Subject: Re: [foreman-users] Web Components Stop Responding

Not sure what was going on. I typically have fewer than 10 hosts connected to this lab install which was hanging. I updated it to 3.2.2/1.13.3 on Monday and have debug logging on and haven’t seen the issue reappear.

shrug

----- Original Message -----
From: “Stephen Benjamin” stephen@redhat.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Tuesday, January 10, 2017 6:13:27 PM
Subject: Re: [foreman-users] Web Components Stop Responding

----- Original Message -----

From: “‘Jason B. Nance’ via Foreman users” foreman-users@googlegroups.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Friday, January 6, 2017 11:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as
well) I run into the situation where occasionally the web interface and API
stop responding. Attempts to load any page, click on any
button/link/whatever results in the browser just spinning and eventually
timing out. The only log message printed is:
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
172.16.246.240:56536] AH02227: Failed to set r->user to
'SSL_CLIENT_S_DN_CN'
(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is
about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing
noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb
fails with:
Could not send facts to Foreman: Net::ReadTimeout
(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then hangs
on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not sure
if the tomcat restart matters, as I did it first and nothing changed but
after I restarted httpd - which took much longer than normal - is when
everything came back to life. There were about 55 connections to httpd at
the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

This sounds an awful lot like what we fixed in Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.

Do you have a lot of puppet clients checking in? You could try to tweak the passenger
settings as described here in the Red Hat BZ (will be the default in Foreman 1.14), and
see if that helps:

https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11

Stephen

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Jason_B_Nance · January 17, 2017, 4:27pm

Hello all,

This has happened again today on my lab server while publishing content view versions and I have debug logging enabled in both Foreman and httpd, but this is the only thing reported by foreman-tail when I attempt to hit the web UI:

==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Tue Jan 17 10:06:41.022510 2017] [ssl:info] [pid 4712] [client 172.16.246.240:33626] AH01964: Connection to child 37 established (server katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.022782 2017] [ssl:debug] [pid 4712] ssl_engine_kernel.c(1886): [client 172.16.246.240:33626] AH02044: No matching SSL virtual host for servername localhost found (using default/first virtual host)
[Tue Jan 17 10:06:41.024377 2017] [ssl:debug] [pid 4712] ssl_engine_kernel.c(1812): [client 172.16.246.240:33626] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Tue Jan 17 10:06:41.024696 2017] [ssl:info] [pid 4712] (70014)End of file found: [client 172.16.246.240:33626] AH01991: SSL input filter read failed.
[Tue Jan 17 10:06:41.024714 2017] [ssl:debug] [pid 4712] ssl_engine_io.c(992): [client 172.16.246.240:33626] AH02001: Connection closed to child 37 with standard shutdown (server katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.027909 2017] [ssl:info] [pid 6040] [client 172.16.246.240:33628] AH01964: Connection to child 67 established (server katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.028336 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(1886): [client 172.16.246.240:33628] AH02044: No matching SSL virtual host for servername localhost found (using default/first virtual host)
[Tue Jan 17 10:06:41.030217 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(1812): [client 172.16.246.240:33628] AH02041: Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Tue Jan 17 10:06:41.031205 2017] [ssl:debug] [pid 6040] ssl_engine_kernel.c(224): [client 172.16.246.240:33628] AH02034: Initial (No.1) HTTPS request received for child 67 (server katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.031345 2017] [ssl:warn] [pid 6040] [client 172.16.246.240:33628] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Tue Jan 17 10:06:41.031370 2017] [authz_core:debug] [pid 6040] mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: authorization result of Require all granted: granted
[Tue Jan 17 10:06:41.031377 2017] [authz_core:debug] [pid 6040] mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626: authorization result of <RequireAny>: granted
[Tue Jan 17 10:06:41.031510 2017] [:debug] [pid 6040] mod_intercept_form_submit.c(405): intercept_form_submit_init invoked
[Tue Jan 17 10:06:41.031520 2017] [:debug] [pid 6040] mod_intercept_form_submit.c(407): skipping, no POST request

There are currently 92 Apache children (max clients is set to 256).

I'm guessing the above log messages don't really say anything. Does anyone have any other ideas as to how to troubleshoot this? I pulled an strace of all the httpd processes while trying to hit the site, but it's like drinking through a fire hose.

Thanks,

j

···

----- Original Message ----- From: "Jason Nance" To: "Foreman Users" Sent: Monday, January 16, 2017 10:36:36 AM Subject: Re: [foreman-users] Web Components Stop Responding

Okay, this happened again today, but this time I was publishing a new version of a content view. Apache stopped responding to any/all requests and isn’t logging anything, either. However, my ‘set logging to debug’ stuff was reverted by the upgrade I did last week and I didn’t realize it (until now). SMH So I guess I’ll restart and wait for the next time it happens.

Netstat says:

netstat -an | egrep ‘80|443’

tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN
tcp 1 0 172.16.246.31:54206 184.51.114.249:80 CLOSE_WAIT
tcp 1 0 172.16.246.31:49617 207.99.69.162:80 CLOSE_WAIT
tcp 1 0 172.16.246.31:49616 207.99.69.162:80 CLOSE_WAIT
tcp 1 0 172.16.246.31:59602 207.99.69.162:80 CLOSE_WAIT
tcp 0 0 127.0.0.1:59480 127.0.0.1:27017 ESTABLISHED
tcp 1 0 172.16.246.31:54266 184.51.114.249:80 CLOSE_WAIT
tcp 0 0 172.16.246.31:5671 172.16.246.31:56280 ESTABLISHED
tcp 0 0 127.0.0.1:27017 127.0.0.1:59480 ESTABLISHED
tcp 0 0 172.16.246.31:56280 172.16.246.31:5671 ESTABLISHED
tcp 1 0 172.16.246.31:36114 184.51.114.242:80 CLOSE_WAIT
tcp 0 0 172.16.246.31:53508 172.16.246.31:443 ESTABLISHED
tcp 0 0 172.16.246.31:53514 172.16.246.31:443 ESTABLISHED
tcp 1 0 172.16.246.31:59603 207.99.69.162:80 CLOSE_WAIT
tcp 0 0 172.16.246.31:53324 172.16.246.31:443 ESTABLISHED
tcp 0 0 172.16.246.31:53340 172.16.246.31:443 ESTABLISHED
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 :::443 :::* LISTEN
tcp6 0 0 :::8443 :::* LISTEN
tcp6 0 0 127.0.0.1:8005 :::* LISTEN
tcp6 0 0 :::8009 :::* LISTEN
tcp6 32 0 172.16.246.31:443 172.16.246.31:53482 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53170 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58842 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53476 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53488 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58778 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53376 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.31:53324 ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.31:53340 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53510 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53256 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53146 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53502 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58774 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53304 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53450 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53140 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53486 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53456 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53454 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53492 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53370 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53498 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53156 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58848 ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.240:58854 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53472 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53506 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53528 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53344 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53518 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53500 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53466 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53504 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53364 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58838 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53428 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53470 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53494 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53530 CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53378 CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.31:53508 ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.31:53514 ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53218 CLOSE_WAIT
udp 0 0 0.0.0.0:51680 0.0.0.0:*

----- Original Message -----
From: “Jason Nance” jason@tresgeek.net
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Thursday, January 12, 2017 2:22:51 PM
Subject: Re: [foreman-users] Web Components Stop Responding

Not sure what was going on. I typically have fewer than 10 hosts connected to this lab install which was hanging. I updated it to 3.2.2/1.13.3 on Monday and have debug logging on and haven’t seen the issue reappear.

shrug

----- Original Message -----
From: “Stephen Benjamin” stephen@redhat.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Tuesday, January 10, 2017 6:13:27 PM
Subject: Re: [foreman-users] Web Components Stop Responding

----- Original Message -----

From: “‘Jason B. Nance’ via Foreman users” foreman-users@googlegroups.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Friday, January 6, 2017 11:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions as
well) I run into the situation where occasionally the web interface and API
stop responding. Attempts to load any page, click on any
button/link/whatever results in the browser just spinning and eventually
timing out. The only log message printed is:
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
172.16.246.240:56536] AH02227: Failed to set r->user to
'SSL_CLIENT_S_DN_CN'
(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is
about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process doing
noticeable work is mongodb.

Puppet runs on clients also fail during this time because /etc/puppet/node.rb
fails with:
Could not send facts to Foreman: Net::ReadTimeout
(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then hangs
on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not sure
if the tomcat restart matters, as I did it first and nothing changed but
after I restarted httpd - which took much longer than normal - is when
everything came back to life. There were about 55 connections to httpd at
the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?

This sounds an awful lot like what we fixed in Bug #14023: puppet-foreman does not allow for configuration of PassengerMaxPoolSize - Installer - Foreman.

Do you have a lot of puppet clients checking in? You could try to tweak the passenger
settings as described here in the Red Hat BZ (will be the default in Foreman 1.14), and
see if that helps:

https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11

Stephen

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Chris_Duryee · January 17, 2017, 4:30pm

does 'passenger-status' show anything interesting?

···

On Tue, Jan 17, 2017 at 11:27 AM, 'Jason B. Nance' via Foreman users < foreman-users@googlegroups.com> wrote:

Hello all,

This has happened again today on my lab server while publishing content
view versions and I have debug logging enabled in both Foreman and httpd,
but this is the only thing reported by foreman-tail when I attempt to hit
the web UI:

==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Tue Jan 17 10:06:41.022510 2017] [ssl:info] [pid 4712] [client
172.16.246.240:33626] AH01964: Connection to child 37 established (server
katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.022782 2017] [ssl:debug] [pid 4712]
ssl_engine_kernel.c(1886): [client 172.16.246.240:33626] AH02044: No
matching SSL virtual host for servername localhost found (using
default/first virtual host)
[Tue Jan 17 10:06:41.024377 2017] [ssl:debug] [pid 4712]
ssl_engine_kernel.c(1812): [client 172.16.246.240:33626] AH02041:
Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Tue Jan 17 10:06:41.024696 2017] [ssl:info] [pid 4712] (70014)End of file
found: [client 172.16.246.240:33626] AH01991: SSL input filter read
failed.
[Tue Jan 17 10:06:41.024714 2017] [ssl:debug] [pid 4712]
ssl_engine_io.c(992): [client 172.16.246.240:33626] AH02001: Connection
closed to child 37 with standard shutdown (server
katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.027909 2017] [ssl:info] [pid 6040] [client
172.16.246.240:33628] AH01964: Connection to child 67 established (server
katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.028336 2017] [ssl:debug] [pid 6040]
ssl_engine_kernel.c(1886): [client 172.16.246.240:33628] AH02044: No
matching SSL virtual host for servername localhost found (using
default/first virtual host)
[Tue Jan 17 10:06:41.030217 2017] [ssl:debug] [pid 6040]
ssl_engine_kernel.c(1812): [client 172.16.246.240:33628] AH02041:
Protocol: TLSv1.2, Cipher: ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)
[Tue Jan 17 10:06:41.031205 2017] [ssl:debug] [pid 6040]
ssl_engine_kernel.c(224): [client 172.16.246.240:33628] AH02034: Initial
(No.1) HTTPS request received for child 67 (server
katello.ipa.centric.lab:443)
[Tue Jan 17 10:06:41.031345 2017] [ssl:warn] [pid 6040] [client
172.16.246.240:33628] AH02227: Failed to set r->user to
’SSL_CLIENT_S_DN_CN’
[Tue Jan 17 10:06:41.031370 2017] [authz_core:debug] [pid 6040]
mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626:
authorization result of Require all granted: granted
[Tue Jan 17 10:06:41.031377 2017] [authz_core:debug] [pid 6040]
mod_authz_core.c(809): [client 172.16.246.240:33628] AH01626:
authorization result of : granted
[Tue Jan 17 10:06:41.031510 2017] [:debug] [pid 6040]
mod_intercept_form_submit.c(405): intercept_form_submit_init invoked
[Tue Jan 17 10:06:41.031520 2017] [:debug] [pid 6040]
mod_intercept_form_submit.c(407): skipping, no POST request

There are currently 92 Apache children (max clients is set to 256).

I’m guessing the above log messages don’t really say anything. Does
anyone have any other ideas as to how to troubleshoot this? I pulled an
strace of all the httpd processes while trying to hit the site, but it’s
like drinking through a fire hose.

Thanks,

j

----- Original Message -----
From: “Jason Nance” jason@tresgeek.net
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Monday, January 16, 2017 10:36:36 AM
Subject: Re: [foreman-users] Web Components Stop Responding

Okay, this happened again today, but this time I was publishing a new
version of a content view. Apache stopped responding to any/all requests
and isn’t logging anything, either. However, my 'set logging to debug’
stuff was reverted by the upgrade I did last week and I didn’t realize it
(until now). SMH So I guess I’ll restart and wait for the next time it
happens.

Netstat says:

netstat -an | egrep ‘80|443’

tcp 0 0 0.0.0.0:8008 0.0.0.0:* LISTEN
tcp 1 0 172.16.246.31:54206 184.51.114.249:80
CLOSE_WAIT
tcp 1 0 172.16.246.31:49617 207.99.69.162:80
CLOSE_WAIT
tcp 1 0 172.16.246.31:49616 207.99.69.162:80
CLOSE_WAIT
tcp 1 0 172.16.246.31:59602 207.99.69.162:80
CLOSE_WAIT
tcp 0 0 127.0.0.1:59480 127.0.0.1:27017
ESTABLISHED
tcp 1 0 172.16.246.31:54266 184.51.114.249:80
CLOSE_WAIT
tcp 0 0 172.16.246.31:5671 172.16.246.31:56280
ESTABLISHED
tcp 0 0 127.0.0.1:27017 127.0.0.1:59480
ESTABLISHED
tcp 0 0 172.16.246.31:56280 172.16.246.31:5671
ESTABLISHED
tcp 1 0 172.16.246.31:36114 184.51.114.242:80
CLOSE_WAIT
tcp 0 0 172.16.246.31:53508 172.16.246.31:443
ESTABLISHED
tcp 0 0 172.16.246.31:53514 172.16.246.31:443
ESTABLISHED
tcp 1 0 172.16.246.31:59603 207.99.69.162:80
CLOSE_WAIT
tcp 0 0 172.16.246.31:53324 172.16.246.31:443
ESTABLISHED
tcp 0 0 172.16.246.31:53340 172.16.246.31:443
ESTABLISHED
tcp6 0 0 :::80 :::* LISTEN
tcp6 0 0 :::8080 :::* LISTEN
tcp6 0 0 :::443 :::* LISTEN
tcp6 0 0 :::8443 :::* LISTEN
tcp6 0 0 127.0.0.1:8005 :::* LISTEN
tcp6 0 0 :::8009 :::* LISTEN
tcp6 32 0 172.16.246.31:443 172.16.246.31:53482
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53170
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58842
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53476
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53488
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58778
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53376
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.31:53324
ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.31:53340
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53510
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53256
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53146
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53502
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58774
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53304
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53450
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53140
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53486
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53456
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53454
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53492
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53370
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53498
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53156
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58848
ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.240:58854
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53472
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53506
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53528
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53344
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53518
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53500
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53466
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53504
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53364
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.240:58838
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53428
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53470
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53494
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53530
CLOSE_WAIT
tcp6 32 0 172.16.246.31:443 172.16.246.31:53378
CLOSE_WAIT
tcp6 0 0 172.16.246.31:443 172.16.246.31:53508
ESTABLISHED
tcp6 0 0 172.16.246.31:443 172.16.246.31:53514
ESTABLISHED
tcp6 32 0 172.16.246.31:443 172.16.246.31:53218
CLOSE_WAIT
udp 0 0 0.0.0.0:51680 0.0.0.0:*

----- Original Message -----
From: “Jason Nance” jason@tresgeek.net
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Thursday, January 12, 2017 2:22:51 PM
Subject: Re: [foreman-users] Web Components Stop Responding

Not sure what was going on. I typically have fewer than 10 hosts
connected to this lab install which was hanging. I updated it to
3.2.2/1.13.3 on Monday and have debug logging on and haven’t seen the issue
reappear.

shrug

----- Original Message -----
From: “Stephen Benjamin” stephen@redhat.com
To: “Foreman Users” foreman-users@googlegroups.com
Sent: Tuesday, January 10, 2017 6:13:27 PM
Subject: Re: [foreman-users] Web Components Stop Responding

----- Original Message -----
From: “‘Jason B. Nance’ via Foreman users” <foreman-users@googlegroups.

To: “Foreman Users” foreman-users@googlegroups.com
Sent: Friday, January 6, 2017 11:41:55 AM
Subject: [foreman-users] Web Components Stop Responding

Hello Everyone,

On a Katello 3.2.1 / TFM 1.13.2 server (and noticed on previous versions
as
well) I run into the situation where occasionally the web interface and
API
stop responding. Attempts to load any page, click on any
button/link/whatever results in the browser just spinning and eventually
timing out. The only log message printed is:
==> /var/log/httpd/foreman-ssl_error_ssl.log <==
[Fri Jan 06 10:16:11.926220 2017] [ssl:warn] [pid 24448] [client
172.16.246.240:56536] AH02227: Failed to set r->user to
'SSL_CLIENT_S_DN_CN'
(I see this frequently under normal conditions)

The system is not under heavy loads during this freeze. Load average is
about 1.2 (on a 6 vCPU / 16G memory virtual server). The only process
doing
noticeable work is mongodb.

Puppet runs on clients also fail during this time because
/etc/puppet/node.rb
fails with:
Could not send facts to Foreman: Net::ReadTimeout
(no log messages are printed)

A ‘yum install’ on a client is able to download the package, but then
hangs
on “Uploading Package Profile” (and eventually times out and skips it).

If I restart [tomcat and] httpd things start responding again. I’m not
sure
if the tomcat restart matters, as I did it first and nothing changed but
after I restarted httpd - which took much longer than normal - is when
everything came back to life. There were about 55 connections to httpd
at
the time, nearly all in CLOSE_WAIT.

Anyone have ideas as to what may be happening?
This sounds an awful lot like what we fixed in http://projects.theforeman.
org/issues/14023.

Do you have a lot of puppet clients checking in? You could try to tweak
the passenger
settings as described here in the Red Hat BZ (will be the default in
Foreman 1.14), and
see if that helps:

https://bugzilla.redhat.com/show_bug.cgi?id=1163452#c11

Stephen

Thanks,

j

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Jason_B_Nance · January 17, 2017, 4:43pm

It says:

ERROR: Phusion Passenger doesn't seem to be running.

Though that is what it says when things are working, as well.

Should I be running that as a user other than root or something?

j

···

From: "Christopher Duryee" To: "Foreman Users" Sent: Tuesday, January 17, 2017 10:30:29 AM Subject: Re: [foreman-users] Web Components Stop Responding

does ‘passenger-status’ show anything interesting?