Console and re-build problems on VMware

Hi,
I recently started managing VMs on a VMware compute resource via Foreman,
and I still have some issues to solve, which I didn't have when using KVM.

First of all, Foreman cannot open the VM console, returning "Failed to
connect to server" on Chrome and "Connect timeout" on Firefox.
I've been told there's no active firewall on the VMware host.
The client/browser machine can connect to the Foreman machine, and the
Foreman machine can connect to the VMware host.
This is the kind of command line I can see generated on the Foreman machine:

/usr/bin/python /usr/share/foreman/extras/noVNC/websockify.py --daemon 

–idle-timeout=120 --timeout=120 5911 vmware-factory1.mydomain:5920

"vmware-factory1.mydomain" is actually the cluster/container in VMware
where the VM resides, and it resolves to the IP address of the VMware
machine.

I can't seem to find the corresponding listening open port (5920) in the
VMware host or in the VM, so I'm still a bit confused on how this all works
with the VMware computer resource: how does Foreman know which port it must
connect to? And does it connect to the virtual machine directly, or does it
go through the VMware host which in turn forwards the connection to the VM?

Also, asking Foreman to build again an existing host does not work: I see
no errors in the Foreman GUI, everything looks like it will work, but then
the VM reboots normally inside the already installed OS, the rebuild does
not happen.
In the proxy log I find lines like these:

E, [2014-04-28T10:48:03.327922 #25560] ERROR – : [704] 2014-04-28 10:48:03
URL: ftp://myrepository.mydomain/RHEL63/images/pxeboot/initrd.img
[30442765] -> "/var/lib/tftpboot//boot/RedHat-6.3-x86_64-initrd.img" [1]

E, [2014-04-28T10:48:03.328770 #25560] ERROR – : [1358] 2014-04-28
10:48:03 URL: ftp://myrepository.mydomain/RHEL63/images/pxeboot/vmlinuz
[3986992] -> "/var/lib/tftpboot//boot/RedHat-6.3-x86_64-vmlinuz" [1]

Do they indicate anything unusual? The two destination files already exist
after all, they do not need to be copied again.
Do they really need to be copied again for the re-build process to start?

Thanks for any info.
Marco

> E, [2014-04-28T10:48:03.327922 #25560] ERROR – : [704] 2014-04-28 10:48:03
> URL: ftp://myrepository.mydomain/RHEL63/images/pxeboot/initrd.img
> [30442765] -> "/var/lib/tftpboot//boot/RedHat-6.3-x86_64-initrd.img" [1]
>
> E, [2014-04-28T10:48:03.328770 #25560] ERROR – : [1358] 2014-04-28
> 10:48:03 URL: ftp://myrepository.mydomain/RHEL63/images/pxeboot/vmlinuz
> [3986992] -> "/var/lib/tftpboot//boot/RedHat-6.3-x86_64-vmlinuz" [1]
>
> Do they indicate anything unusual? The two destination files already exist
> after all, they do not need to be copied again.

Nope, wget in this case only checks if the files are up to date and does
not re-download.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> /usr/bin/python /usr/share/foreman/extras/noVNC/websockify.py
> --daemon --idle-timeout=120 --timeout=120 5911 vmware-factory1.mydomain:5920
>
> "vmware-factory1.mydomain" is actually the cluster/container in VMware
> where the VM resides, and it resolves to the IP address of the VMware
> machine.
>
> I can't seem to find the corresponding listening open port (5920) in the
> VMware host or in the VM, so I'm still a bit confused on how this all works
> with the VMware computer resource: how does Foreman know which port it must
> connect to? And does it connect to the virtual machine directly, or does it
> go through the VMware host which in turn forwards the connection to the VM?
>

That's more clear now (maybe): "vmware-factory1.mydomain" is the ESXi host
which is part of the VMware cluster, it hosts the VM and its name correctly
resolves to its IP address.
But I'm not able to connect to it using the port in the Foreman command
line.
Can anyone explain how is this supposed to work? I'm a bit clueless at the
moment, I don't know what else to look at.

Also, asking Foreman to build again an existing host does not work: I see
> no errors in the Foreman GUI, everything looks like it will work, but then
> the VM reboots normally inside the already installed OS, the rebuild does
> not happen.
>

I found the reason for this: VMs in ESX are created by default with network
boot at the last position in the boot order BIOS menu, so when the OS is
installed it is started from disk before the PXE boot image can be read.
Does anyone know how to change this default boot order, by chance? I didn't
find any way, yet.

Thanks.
Marco

··· On Monday, April 28, 2014 3:31:51 PM UTC+2, zerozer...@gmail.com wrote:

Nope, wget in this case only checks if the files are up to date and does
> not re-download.
>

So what's the meaning of those ERROR messages?

Thanks.
Marco

··· Il giorno lunedì 28 aprile 2014 18:04:58 UTC+2, Lukas Zapletal ha scritto:

Il giorno mercoledì 30 aprile 2014 11:43:40 UTC+2, zerozer...@gmail.com ha
scritto:

[Console]

> Can anyone explain how is this supposed to work? I'm a bit clueless at the
> moment, I don't know what else to look at.
>

Well… while I remain curious on the details of the Foreman<->ESX
connection (is the VNC port on ESX open by default? How does Foreman know
its number? Or does Foreman ask for it to be opened?), I discovered that
the problem was simply due to the ESX firewall. I had been told it was not
active, but it actually was instead. The console works now.
Marco

I'm not sure how to change it, we may need to change Fog first to ensure
the right parameter is exposed for us to use in Foreman.

I'd consider this a bug that we don't set network boot first, as we do
for other compute resources. See also
Feature #4317: Build button - Vmware provisioning - single click rebuild - Foreman.

··· On 30/04/14 10:43, zerozerounouno@gmail.com wrote: > On Monday, April 28, 2014 3:31:51 PM UTC+2, zerozer...@gmail.com wrote: > Also, asking Foreman to build again an existing host does not work: > I see no errors in the Foreman GUI, everything looks like it will > work, but then the VM reboots normally inside the already installed > OS, the rebuild does not happen. > > > I found the reason for this: VMs in ESX are created by default with > network boot at the last position in the boot order BIOS menu, so when > the OS is installed it is started from disk before the PXE boot image > can be read. > Does anyone know how to change this default boot order, by chance? I > didn't find any way, yet.


Dominic Cleal
Red Hat Engineering

> So what's the meaning of those ERROR messages?

It's a bug in Foreman Proxy, we currently interpret all the wget output
as ERRORs (it goes to stderr). Just ignore those.

We want to improve wgetting a lot, also making it more synchronous. If
you have time to work on this, please help us to improve this. We can
start with correct logging.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> I'm not sure how to change it, we may need to change Fog first to ensure
> the right parameter is exposed for us to use in Foreman.
>

Oh, well, I was actually looking for some ESX "default boot order" setting.
But having Foreman set it right would be great of course ;).

> I'd consider this a bug that we don't set network boot first, as we do
> for other compute resources. See also
> Feature #4317: Build button - Vmware provisioning - single click rebuild - Foreman.

Thanks.
To make the problem more visible in the tracker I filed a bug:
http://projects.theforeman.org/issues/5510

Marco

··· Il giorno mercoledì 30 aprile 2014 12:25:27 UTC+2, Dominic Cleal ha scritto:

> It's a bug in Foreman Proxy, we currently interpret all the wget output
> as ERRORs (it goes to stderr). Just ignore those.
>

Ok thanks.

> We want to improve wgetting a lot, also making it more synchronous. If
> you have time to work on this, please help us to improve this. We can
> start with correct logging.
>

Thank you for the confidence in me ;), and I really like helping open
source projects (I mostly did docs, localization and testing, I'm not a
highly experienced developer), but time definitely is not on my side in
this period.

Marco

··· Il giorno martedì 29 aprile 2014 09:58:01 UTC+2, Lukas Zapletal ha scritto: