Console debug issues

Support
1

So trying to debug why the console option is not working.

  • Check for a “websockify.py” process on your Foreman server when
    opening the console page in Foreman
  • If websockify.py is missing, check /var/log/foreman/production.log for
    stderr output with logging increased to debug
  • Look at the last argument of the process command line, it will have
    the hypervisor hostname and port - ensure you can resolve and ping this
    hostname
  • Make sure you access Foreman web UI via FQDN as the certificate does
    not have shortened hostname.
  • Try a telnet/netcat connection from the Foreman host to the hypervisor
    hostname/port
  • The penultimate argument of websockify.py is the listening port number,
    check if your web client host can telnet to it
  • If using Firefox, check the known issues above and set the config
    appropriately
  • The error “WebSock error: [object Event]” can be caused by a self
    signed certificate, where the certificate`s algorithm is too weak, e.g.
    SHA1. Debugging the issue with the Firefox JavaScript Console will show a
    warning similar to “This site makes use of a SHA-1 Certificate; it’s
    recommended you use certificates with signature algorithms that use hash
    functions stronger than SHA-1”    . See Weak Signature Algorithms
    <https://developer.mozilla.org/docs/Security/Weak_Signature_Algorithm>
    on Mozilla website.

  • websockify.py exists.

  • looked at production.log for errors, saw none.
    2016-08-29 23:52:41 [app] [I] Processing by HostsController#console as HTML
    2016-08-29 23:52:41 [app] [I] Parameters: {"id"=>"george-dev.net"}
    2016-08-29 23:52:44 [app] [I] Rendered hosts/console/vnc.html.erb within
    layouts/application (6.5ms)
    2016-08-29 23:52:44 [app] [I] Rendered
    layouts/_application_content.html.erb (0.8ms)
    2016-08-29 23:52:44 [app] [I] Rendered home/_user_dropdown.html.erb
    (4.3ms)
    2016-08-29 23:52:44 [app] [I] Read fragment views/tabs_and_title_records-3
    (0.3ms)
    2016-08-29 23:52:44 [app] [I] Rendered home/_topbar.html.erb (13.6ms)
    2016-08-29 23:52:44 [app] [I] Rendered layouts/base.html.erb (16.2ms)
    2016-08-29 23:52:44 [app] [I] Completed 200 OK in 2394ms (Views: 26.4ms |
    ActiveRecord: 3.5ms)

  • there is no argument in a "process command line" in said production.log.

  • would try telnet but not sure what port it's trying to use. Does it try
    to use the web client ip for console or the foreman host?

  • I do get the "WebSock error: [object Event]" but do not use a SHA1 key.

So, where do these commands get logged? I'm kinda stuck trying to figure
out what it's trying to do.

2

> * there is no argument in a "process command line" in said production.log.

Increase logging verbosity.

> * would try telnet but not sure what port it's trying to use. Does it try
> to use the web client ip for console or the foreman host?

You will see ports in logs. It's random from a fixed range.

> So, where do these commands get logged? I'm kinda stuck trying to figure
> out what it's trying to do.

Above.

··· -- Later, Lukas #lzap Zapletal
3

This is referring to the process list from the command line of your server

  • "ps axf|grep websock" is usually the way to go. This should give you some
    detail on hostnames and ports it's trying to use.

Note that the websockify processes are very short lived when they fail,
so you may want to run "watch -n1 'ps axf|grep websock' " or similar to get
a live view of the process list and spot the data you need.

Here's an example from my process list:

[greg@topaz]$ ps ax|grep
websock ~
23002 ? S 0:00 /usr/bin/python
/usr/share/foreman/extras/noVNC/websockify.py --daemon --idle-timeout=120
–timeout=120 5919 libvirt.domain:5901 --cert topaz.domain.pem --key
topaz.pem

This is a notoriously fiddly thing to debug - good luck :slight_smile:

Greg

··· On 30 August 2016 at 00:54, Mike Wilson wrote:
  • there is no argument in a “process command line” in said production.log.
4

Thanks for the tips (yours and Lukas). I enabled debug and was quickly able
to find the port/host names it's using and start working down a path to fix
it (router ACL issue for now).

I'm just going to leave the proxy and foreman in debug mode, so much more
detail :wink:

5

> I'm just going to leave the proxy and foreman in debug mode, so much more
> detail :wink:

Or file a patch to increase verbosity of these, you are not the first
:slight_smile:

··· -- Later, Lukas #lzap Zapletal