Debug - General Topic - Feedback

Problem:
Debug - Procedure - TFTP and others
Except from this file I cannot find anything else to help troubleshooting TFTP timeout, file configuration not found …
=> /var/log/foreman/production.log (nothing in regards to tftp)
=> foreman-debug -v -a -d /tmp/Debug_2020-18-09:T05:35
On the client side I obtain a DHCP address but it stuns on TFTP and without any logs …

Expected outcome:
I would like to be autonomous in regards to troubleshooting Foreman (TFTP). I noticed that in /var/lib/tftpboot/pxelinux.cfg/ nothing is there; no file …
Secondly, is a local DNS server compulsory to make a deployment working?
Thirdly, a lot of blogs present Foreman installation but no one goes beyond (deployment). I have been working on it for approximately a week without having been able to get a deployment working (environment: WorkStationPro).
Fourthly, Ruby (Puppet project) is used but I don’t have time to spend hours in order to understand and develop in Ruby to be able to troubleshoot when I have an issue.

Lastly, this is a good development and thank you! I think that if IBM (RedHat) deployed a foreman branch (Satellite), it is because it’s worth the while.

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Distribution and version:
HOSTNAME: ***
OS: debian (Ubuntu 1804)
RELEASE: buster/sid
FOREMAN: 2.1.2
RUBY: ruby 2.5.1p57 (2018-03-29 revision 63029) [x86_64-linux-gnu]
PUPPET:

Other relevant data:

Try to use foreman-tail. The TFTP is managed by a smart-proxy, so relevant logs will be inside /var/log/foreman-proxy/proxy.log.
Also tcpdump might come handy.

Not sure what your worklow is, but pxelinux and grub configs are being deployed to tftpboot during orchestration part of provisioning.
If you want to deploy a default pxe config for unknown hosts in the subnet, navigate to Provisionimg templates page and click the Deploy PXE default button. This will put a file called default to the tftpboot and configures dhcpd to point the unknown hosts to it.

You can control how the config looks like using the provisioning templates.

1 Like

You were right I had not click on “Deploy PXE default` button”. Now I can see the default file appearing but still cannot understand why the VM cannot load the system:

cat default


DEFAULT discovery
[…]
LABEL discovery
MENU LABEL Foreman Discovery Image
KERNEL boot/fdi-image/vmlinuz0
APPEND initrd=boot/fdi-image/initrd0.img rootflags=loop root=live:/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nokaslr nomodeset proxy.url=https://maas.local.sas proxy.type=foreman
IPAPPEND 2

tail -f /var/log/foreman-proxy/proxy.log

2020-09-18T16:39:21 fe1ee70c [I] Started POST /tftp/PXELinux/create_default
2020-09-18T16:39:21 fe1ee70c [I] Finished POST /tftp/PXELinux/create_default with 200 (0.91 ms)
2020-09-18T16:39:21 fe1ee70c [I] Started POST /tftp/PXEGrub/create_default
2020-09-18T16:39:21 fe1ee70c [I] Finished POST /tftp/PXEGrub/create_default with 200 (1.36 ms)
2020-09-18T16:50:13 5f952585 [I] Started POST /tftp/PXEGrub2/create_default
2020-09-18T16:50:13 5f952585 [I] Finished POST /tftp/PXEGrub2/create_default with 200 (2.02 ms)
2020-09-18T16:50:13 5f952585 [I] Started POST /tftp/PXELinux/create_default
2020-09-18T16:50:13 5f952585 [I] Finished POST /tftp/PXELinux/create_default with 200 (0.63 ms)
2020-09-18T16:50:14 5f952585 [I] Started POST /tftp/PXEGrub/create_default
2020-09-18T16:50:14 5f952585 [I] Finished POST /tftp/PXEGrub/create_default with 200 (1.03 ms)

Installation Media for the Operating System is Ubuntu Mirror … the system cannot be discovered …

Thanks @rplevka !!! I went one step ahead … I just need to solve this loading issue and after that I’m done!!!
I was stuck on this …

Is that all you can see inside proxy.log, even after you booted the VM? As normally you should see the dhcp offer logs as well as tftp requests and responses. You mentioned that you’re getting the DHCP address - are you sure it’s foreman server that gives you the address?

I’m assuming you’re not using external smart proxy.
Verify that the VM is really deployed to the same network that the smart proxy manages as well as the firewall rules (especially around tftp).

If possible, check the serial console of the machine and check the output of the pxe boot sequence (you should see the dhcp server as well as the next-server).

1 Like

I wanted to upload the file but … lol … “Sorry new users cannot upload files”.

I have 2 NICs on Foreman VM:

  • WAN - 192.168.1.0/24
  • LAN - 10.0.0.0/24

Foreman is accessible from WAN NIC (bridge WorkStationPro)
Both VMs can communicate via an intnet connection.

How can I check if I’m using using external smart proxy and how can I activate it?

Thanks @rplevka !

I reinstalled it:

# wget -q http://deb.theforeman.org/pubkey.gpg -O- | sudo apt-key add -
OK

# apt-get update
Hit:1 http://fr.archive.ubuntu.com/ubuntu bionic InRelease
[…]
Reading package lists… Done
# apt-get install foreman-proxy

foreman-proxy is already the newest version (2.1.2-1).
foreman-proxy set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 28 not upgraded.

Smart Proxy was already installed. But I don’t know how to check the service and to restart it.

If you can help me on this @rplevka … Thanks!!

cat /etc/foreman-proxy/settings.yml | grep -v “#”

:settings_directory: /etc/foreman-proxy/settings.d
:ssl_ca_file: /etc/puppetlabs/puppet/ssl/certs/ca.pem
:ssl_certificate: /etc/puppetlabs/puppet/ssl/certs/maas.local.sas.pem
:ssl_private_key: /etc/puppetlabs/puppet/ssl/private_keys/maas.local.sas.pem
:trusted_hosts:

  • maas.local.sas
    :foreman_url: https://maas.local.sas
    :daemon: true
    :bind_host: ‘*’
    :https_port: 8443
    :http_port: 8000
    :log_file: /var/log/foreman-proxy/proxy.log
    :log_level: INFO
    :log_buffer: 2000
    :log_buffer_errors: 1000

Ah, then you probably don’t use it. The smart-proxy running on the foreman installer is the internal one. External proxy would run on a separate machine.

If you don’t want foreman to manage dhcp, you have to manually configure the dhcp server you use to point to the tftp of the foreman (next-server directive).

If you want foreman to manage it, make sure you unn foreman-installer with the appropriate --foreman-proxy-dhcp* and --foreman-proxy-tftp* parameters (use --help to list them).
Make sure no other dhcp server is running inside the subnet.

You can paste the output you’d like to share to fpaste or pastebin and link it here.

I highly recommend you to read Foreman manual:
https://theforeman.org/manuals/2.1/index.html
The very first chapter describes its architecture and is very informative.

Below is my ISC-DHCP configuration file @rplevka:

less /etc/dhcp/dhcpd.conf

default-lease-time 43200;
max-lease-time 86400;

not authoritative;

ddns-update-style none;

option domain-name “local.sas”;
option domain-name-servers 10.0.0.1;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update on; # set the “O” and “S” flag bits
option fqdn.rcode2 255;
option pxegrub code 150 = text ;

next-server 10.0.0.1;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
filename “grub2/bootia32.efi”;
} elsif option architecture = 00:07 {
filename “grub2/bootx64.efi”;
} elsif option architecture = 00:09 {
filename “grub2/bootx64.efi”;
} else {
filename “pxelinux.0”;
}

log-facility local7;

include “/etc/dhcp/dhcpd.hosts”;

If you want foreman to manage it, make sure you run foreman-installer with the appropriate --foreman-proxy-dhcp and --foreman-proxy-tftp*
I installed everything (ISC-DHCP, TFTP) with foreman-installer --*** according to the doc.

*Make sure no other dhcp server is running inside the subnet.
Both VMs can communicate via an intnet connection
=> This LAN is secluded; only both VMs can communicate so no other DHCP server can interact.

To add @rplevka

netstat -plnt
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 1195/redis-server 1
tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 810/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1202/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 1261/postgres
tcp 0 0 127.0.0.1:3000 0.0.0.0:* LISTEN 1/init
tcp 0 0 0.0.0.0:8443 0.0.0.0:* LISTEN 1064/ruby
tcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 1064/ruby
tcp 0 0 0.0.0.0:7911 0.0.0.0:* LISTEN 995/dhcpd

=> tcp 0 0.0.0.0:8443 - LISTEN - 0.0.0.0:*

The dhcp config seems to be correct.
Can you confirm that you can see the dhcp handshake logs and the tftp logs in your proxy.log?
Also verify that there is a dhcp record in the /var/lib/dhcpd/dhcpd.leases once your host obtains IP.

Can you check the serial console of your VM while pxe-booting (if available)?

From the information you provided, I still have some doubts about where your VM gets the ip from.
I have no experience with VMware workstation, but from what i have read, even if you use “Host Only networking” there is still a DHCP service provided by the vmnet1 virtual switch.
Ignore me if i’m completely off and you’re sure there’s no such thing in your setup.

Last, but not least: verify the firewall rules for TFTP on your sat server. Thanks!

Now I realized I should also ask you to set logging level to DEBUG for foreman-proxy here:

/etc/foreman-proxy/settings.yml

Don’t forget to restart foreman-proxy service after updating its config.

cat /var/lib/dhcp/dhcpd.leases
lease 10.0.0.2 {
starts 5 2020/09/18 16:43:47;
ends 6 2020/09/19 04:43:47;
tstp 6 2020/09/19 04:43:47;
cltt 5 2020/09/18 16:43:47;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet 00:0c:29:2b:98:0f;
set vendor-class-identifier = “PXEClient:Arch:00000:UNDI:002001”;
set vendor-string = “PXEClient:Arch:00000:UNDI:002001”;
}

I modified this file as you told me:
=> vi /etc/foreman-proxy/settings.yml

“# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN”
:log_level: DEBUG

I have no experience with VMware workstation, but from what i have read, even if you use “Host Only networking” there is still a DHCP service provided by the vmnet1 virtual switch.

=> I was TSE at VMware not long ago … lol

The boot still stuns on …

vmware_qsvKhWsAVr

On the other hand I don’t know if it’s correct:

Wait, that’s it! Tha bootmenu you’re getting is the default one, that you deployed. So it works just fine. Depending on what you’re trying to achieve now, your options are eithre loading the Discovery image and let it to send the VM facts to the foreman, where you’ll find your VM under “discovered hosts” and provision it from there. Or you can simply navigate to New Host dialog and provision it from there directly.
You get to define your subnet in the UI (infrastructure/subnets) and define some installation media + OS. After filling the info in the dialog foreman will render appropriate pxelinux/grub configs and deploy it to the tftp same way as it deployed the default one. You just reboot your VM and it should pick up its config. That’s the whole magic behind it.
You can also add integration with your VMware instance (infrastructure/compute resources) but i guess you need to install extra plugin to enable it…then foreman will take care of creating of the vm aswell.

Is that all you needed?

1 Like

Sorry @rplevka for this belated answer,
This incident is solved.
Thanks for everything!!!

I am working on script automation now …

Gregory