Unable to open the VNC console

Problem:
Hi,

We have a problem that we can not open the VNC console for a new virtual machine (VM).
The necessary VNC-ports are open on the compute-node.

After some research, we saw that the VM was killed due to the cirrus video-card that is deprecated.
For this issue we applied the fix described in:

So we have edited the VM xml-config with visrh edit according the fix above.

After restarting the VM we can’t see anymore a kill in the qemu VM.log.

However, when we try to open the console it keeps hanging on “Loading …”


Expected outcome:

A working console so we can follow up the installation-process on the VM.

Foreman and Proxy versions:

Foreman 2.4 / Katello 4.0

Foreman and Proxy plugin versions:

Distribution and version:

CentOS Linux release 8.3.2011

Other relevant data:

qemu version: 4.2.0qemu-kvm-4.2.0-34.module_el8.3.0+755+88436ea4.5, kernel: 4.18.0-240.22.1.el8_3.x86_64

There is a patch you can apply:

That should work as a workaround until this is solved upstream.

@lzap ,

As written, we have already applied that patch, tha VM is started as expected and we don’t see that error anymore in the ~VM-log.

However, the VNC-console is still not working as we expect.
It keeps hanging with “Loading …”, only one small black part on the top is shown (see initial screenshot) and that is all.
So we can’t see wat is happening during the further process, if there are other problems, …

It would be fine to have a working VNC-console to follow up the further process of the VM-creation (PXE installation).
This works well on our old system with Centos 7/Foreman 2.3/Ketello 3.18 system but not on our new system with CentOS 8/Foreman 2.4/Katello 4.0

Thanks for any feedback regarding this issue.

Make sure you have a VALID certificate (not confirmed security exception), CA is installed in your browser, you are connected through HTTPS. Modern browsers do not allow javascript to do WS/WSS connections.

Then debug the session in browser, we need to see more details.

Also check if VNC port is responding on the hypervisor, check pysockify as well, firewalls etc. This is unfortunately a pain to debug. We haven’t changed anything significant in 2.4 in this regard.

@lzap ,

We ran the foreman-installer command in order to update the initial self-signed certificate on the foreman server with our own one.

Afterwards we upgraded the katello-ca-consumer-latest.noarch.rpm on the host on which we will create the VM.

However, when starting the VM-console we now get the error “The connection was closed by the browser. please verify that the certificate authority is valid”

But when checking the certificates in the browser, they are valid.

Any suggestion what we should verify further?

What’s a bit tricky is that the VNC console opens its own socket to listen on with its own certificates. That port is only open shortly after the VNC page is opened. That means you need to know which port is used (5910 - 5930 are used) and be quick. That’s usually not nice.

In browsers you can open the developer console (Tools -> Browser Tools -> Web Developer Tools in Firefox) and open the network tab. There you can find the websockets connection. Perhaps that’s better.

What you can certainly do is go to the Settings in the UI where you can see websockets settings with a CA and key file. Please verify those have the correct value that you would expect (should match what Apache uses).

1 Like

@ekohl ,

Thanks for your feedback.

We have verified the certificate in the firefox dev-tools with the one configured in foreman settings -> websockets, and at this level it is correct.

What else can we check?

@lzap , @ekohl ,

Is it possible that our own certificate can not be used as it is one for *.< domain >?

@ekohl , @lzap ,

We can also confirm that on the foreman-server a websockify.py process is started wit a port between 5910 - 5930.

@ekohl , @lzap ,

In chrome, with a websocket-plugin installed, we could see that the websocket was well started.
And on the foreman-server we could also see that a process was started using that port:

[root@foreman-server ~]# ss -tlpn | grep websockify
LISTEN 0 100 0.0.0.0:5917 0.0.0.0:* users:((“websockify.py”,pid=227009,fd=3))
[root@foreman-server ~]# ps -fp 227009
UID PID PPID C STIME TTY TIME CMD
foreman 227009 1 2 17:03 ? 00:00:00 /usr/libexec/platform-python /usr/share/foreman/extras/noVNC/webso

But the console keeps showing the error “The connection was closed by the browser. please verify that the certificate authority is valid”, and only a small part of the console is shown.

Additional info:
The host on which we will install the VM is also running with CentOS 8, and the VM should be installed over PXE.
On our old Foreman 2.1/Katello 3.15 this all worked like a charm, using the initial self-signed certificate.
However, I have seen that after upgrading it to Foreman 2.3.3/Katello 2.18, it is also no more possible to open the console. We also get the same error.

Is your websockify running on the same host as Foreman itself?

Can you check that your browser is making connection to the same hostname as you have in your location?

I am not aware of any changes in this regard, we plan some WS-WSS changes for the next release but that hasn’t been merged yet.

@lzap ,
Sorry for the late reply, had a few days off.

This is what I have:

On the Foreman-server:

[root@dc1-bootstrap1 ~]# ss -tlpn | grep 5917
LISTEN 0 100 0.0.0.0:5917 0.0.0.0:* users:((“websockify.py”,pid=359100,fd=3))

[root@dc1-bootstrap1 ~]# ps -fp 359100
UID PID PPID C STIME TTY TIME CMD
foreman 359100 1 0 11:17 ? 00:00:00 /usr/libexec/platform-python /usr/share/foreman/extras/noVNC/webso

[root@dc1-bootstrap1 ~]# ps -ef | grep webso
foreman 359100 1 3 11:17 ? 00:00:00 /usr/libexec/platform-python /usr/share/foreman/extras/noVNC/websockify.py --daemon --idle-timeout=120 --timeout=120 5917 dc1-zabbix1.cumuli.be:5900 --cert /etc/pki/katello/certs/katello-apache.crt --key /etc/pki/katello/private/katello-apache.key

On the Compute-node:

[root@dc1-zabbix1 ~]# ss -tlpn | grep 59
LISTEN 0 1 0.0.0.0:5900 0.0.0.0:* users:((“qemu-kvm”,pid=25561,fd=33))

I have no other ideas than trying a RHEL7 compute resource if that’s something that changed in libvirt.

@lzap ,
OK, I try to setup as soon as possible a CentOS 7 compute host and will share the results here.

1 Like