PXE-less discovery fails to kexec on VMWare

Shri · February 16, 2018, 1:35am

Unable to complete provisioning Centos 7.4 using FDI image on vmware workstation.(guest name :mac000c29ba1add.mylabs.local)

VM Guest hung after—> fdi smart-proxy: /usr/sbin/kexec: unrecognized option ‘–args-linux’

foreman version version 1.16.0
fdi-bootable-3.4.4.iso
foreman_discovery plugin version 10.0.0

logs from VM guest.(ssh enabled on Guest VM)

https://pastebin.com/wPkUDxiR

production log
https://pastebin.com/m8Du91nt

please help.

lzap · February 16, 2018, 1:57pm

Can you remove this line:

github.com

theforeman/foreman_discovery/blob/develop/app/views/foreman_discovery/redhat_kexec.erb#L49


{
"comment": "WARNING: Both kernel and initram are not set in preview mode due to http://projects.theforeman.org/issues/19737",
"kernel": "<%= @kexec_kernel %>",
"initram": "<%= @kexec_initrd %>",
<% if (@host.operatingsystem.name == 'Fedora' and @host.operatingsystem.major.to_i > 16) or
  (@host.operatingsystem.name != 'Fedora' and @host.operatingsystem.major.to_i >= 7) -%>
"append": "ks=<%= foreman_url('provision') + "&static=yes" %> inst.ks.sendmac <%= "ip=#{ip}::#{gw}:#{mask}:::none nameserver=#{dns} ksdevice=bootif BOOTIF=#{bootif} nomodeset #{append}" %>",
<% else -%>
"append": "ks=<%= foreman_url('provision') + "&static=yes" %> kssendmac nicdelay=5 <%= "ip=#{ip} netmask=#{mask} gateway=#{gw} dns=#{dns} ksdevice=#{mac} BOOTIF=#{bootif} nomodeset #{append}" %>",
<% end -%>
"extra": ["--args-linux"]
}

or change it to:

"extra": []

Report back please, this might be important bug. IMHO kexec does not work in 1.16 due to this flag.

Shri · February 16, 2018, 4:13pm

removed line 49.“extra”: ["–args-linux"]

task failed with the following error https://pastebin.com/PmrKH6VF

Shri · February 16, 2018, 4:56pm

Below result when change it “extra”: []

Feb 16 11:51:15 fdi foreman-proxy[1487]: [1774] 2018-02-16 11:51:15 URL:http://mirror.centos.org/centos/7.4.1708/os/x86_64/images/pxeboot/vmlinuz [5877760/5877760] -> “/tmp/vmlinuz” [1]
Feb 16 11:51:46 fdi foreman-proxy[1487]: [1777] 2018-02-16 11:51:46 URL:http://mirror.centos.org/centos/7.4.1708/os/x86_64/images/pxeboot/initrd.img [48434768/48434768] -> “/tmp/initrd.img” [1]
Feb 16 11:51:46 fdi foreman-proxy[1487]: 192.168.180.100 - - [16/Feb/2018:11:51:46 +0000] “PUT /power/kexec HTTP/1.1” 200 15 35.0739
Feb 16 11:51:46 fdi smart-proxy: 192.168.180.100 - - [16/Feb/2018:11:51:11 UTC] “PUT /power/kexec HTTP/1.1” 200 15
Feb 16 11:51:46 fdi smart-proxy: - -> /power/kexec
packet_write_wait: Connection to 192.168.180.134 port 22: Broken pipe

and VM hung

lzap · February 19, 2018, 8:41am

In the first case, you had a JSON typo (comma).

In the second case, I am unable to tell what’s wrong. You need to try to follow system journal.

I suggest to try to kexec manually:

cd /tmp
wget http://mirror.centos.org/centos/7.4.1708/os/x86_64/images/pxeboot/vmlinuz
wget http://mirror.centos.org/centos/7.4.1708/os/x86_64/images/pxeboot/initrd.img
kexec -d -l kernel --initrd=initram.img --append="ks=http://foreman.mylabs.local/unattended/provision?token=55e70c52-0a43-49c8-b590-7562fc24f7c4&static=yes inst.ks.sendmac ip=192.168.180.134::192.168.180.2:255.255.255.0:::none nameserver=192.168.180.2 ksdevice=bootif BOOTIF=00-00-0c-29-ba-1a-dd nomodeset"

The -d (debug) flag should do some verbose output.

guan3387 · April 15, 2018, 2:06pm

HI Lukas,

I am facing similar issue, it stop at this

Following is the output i run with debug mode:

github.com

bluecarrot/troubleshooting/blob/master/foreman/kexec-debug.txt

[root@fdi tmp]# kexec -d --force --reset-vga --append="ks=http://sg-foreman.local:8000/unattended/provision?token=67e7eb77-0004-46f6-91c7-2284b5ebe523&static=yes inst.ks.sendmac ip=172.20.31.218::172.20.31.1:255.255.255.0:::none nameserver=172.20.31.214 ksdevice=bootif BOOTIF=00-00-50-56-89-d6-bc nomodeset" --initrd=/tmp/initrd.img /tmp/vmlinuz
Try gzip decompression.
Try LZMA decompression.
lzma_decompress_file: read on /tmp/vmlinuz of 65536 bytes failed
kernel: 0x7fe3c14ec010 kernel_size: 0x5246d0
MEMORY RANGES
0000000000000100-000000000009f7ff (0)
000000000009f800-000000000009ffff (1)
00000000000dc000-00000000000fffff (1)
0000000000100000-000000007feeffff (0)
000000007fef0000-000000007fefefff (2)
000000007feff000-000000007fefffff (3)
000000007ff00000-000000007fffffff (0)
00000000f0000000-00000000f7ffffff (1)
00000000fec00000-00000000fec0ffff (1)
00000000fee00000-00000000fee00fff (1)
00000000fffe0000-00000000ffffffff (1)
sym: sha256_starts info: 12 other: 00 shndx: 1 value: 10d0 size: 48
sym: sha256_starts value: 7fff80d0 addr: 7fff7004
R_X86_64_64

This file has been truncated. show original

lzap · April 15, 2018, 8:33pm

The screenshot shows you are using VESA/FB extensions, can you boot the FDI in 80x25 mode without any special graphics options? Is this BIOS or UEFI?

karthik · January 7, 2020, 7:42am

hi Shri,

I had similar behavior with a fork of foreman. You could find why the reboot attempt fails by checking the proxy.log(/var/log/foreman-proxy/proxy.log

In my case the foreman host wasn’t able to run the kexec on remote host due to firewall restriction, this is evident from the following proxy log snipper.

2019-12-09T06:00:25 eb8083cd [I] Started PUT /10.10.10.10/power/reboot
2019-12-09T06:01:25 0ff854ac [I] Started PUT /10.10.10.10/power/reboot
2019-12-09T06:02:33 eb8083cd [E] Connection timed out - connect(2)
2019-12-09T06:02:33 eb8083cd [I] Finished PUT /10.10.10.10/power/reboot with 500 (127304.51 ms)
2019-12-09T06:03:33 0ff854ac [E] Connection timed out - connect(2)
2019-12-09T06:03:33 0ff854ac [I] Finished PUT /10.10.10.10/power/reboot with 500 (127280.84 ms)

After the firewall change, the kexec was successful.

Note: Please ignore if this is not the case