Is it possible to run tftp server on a libvirt client VM?

techietubby · November 20, 2020, 10:34am

Problem:

I have the latest release 2.21 of Foreman running on CentOS 7.9 inside a libvirt VM (the host is Cent 8.2). Everything works except tftp. I can run tftp on the host machine and can access it from the VM on the command line e.g.

[root@foreman ~]# tftp -4 -v 192.168.0.150 -c get
Connected to 192.168.0.150 (192.168.0.150), port 69
(files) pxelinux.0
getting from 192.168.0.150:pxelinux.0 to pxelinux.0 [netascii]
Received 43347 bytes in 0.1 seconds [2582678 bit/s]

The client libvirt VMs all get created by Foreman and everything works, and the last stage is to run TFTP on the Foreman server but no matter what I try the TFTP from the host to the VM I see this:

tcpdump -i eth0 -vv port 69

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:29:51.532577 IP (tos 0x0, ttl 64, id 30935, offset 0, flags [DF], proto UDP (17), length 50)
vhost.test.local.50628 > foreman.test.local.tftp: [bad udp cksum 0x8279 -> 0x40a2!] 22 RRQ “pxelinux.0” netascii
11:29:56.532645 IP (tos 0x0, ttl 64, id 31303, offset 0, flags [DF], proto UDP (17), length 50)
vhost.test.local.50628 > foreman.test.local.tftp: [bad udp cksum 0x8279 -> 0x40a2!] 22 RRQ “pxelinux.0” netascii
…

I want to install as little as possible on the vhost system and where possible have everything in containers or VMs. Is there any way around this?

Expected outcome:
The file is transferred so the new VM boots

Foreman and Proxy versions:

ansible-collection-theforeman-foreman-1.4.0-1.el8.noarch
ansiblerole-foreman_scap_client-0.0.6-1.el7.noarch
candlepin-3.1.22-1.el7.noarch
candlepin-selinux-3.1.22-1.el7.noarch
foreman-2.2.1-1.el7.noarch
foreman-bootloaders-redhat-202005201200-1.el7.noarch
foreman-bootloaders-redhat-tftpboot-202005201200-1.el7.noarch
foreman-cli-2.2.1-1.el7.noarch
foreman-console-2.2.1-1.el7.noarch
foreman-debug-2.2.1-1.el7.noarch
foreman-discovery-image-service-1.0.0-3.el7.x86_64
foreman-dynflow-sidekiq-2.2.1-1.el7.noarch
foreman-ec2-2.2.1-1.el7.noarch
foreman-installer-2.2.1-1.el7.noarch
foreman-installer-katello-2.2.1-1.el7.noarch
foreman-libvirt-2.2.1-1.el7.noarch
foreman-postgresql-2.2.1-1.el7.noarch
foreman-proxy-2.2.1-1.el7.noarch
foreman-release-2.2.1-1.el7.noarch
foreman-selinux-2.2.1-1.el7.noarch
foreman-service-2.2.1-1.el7.noarch
foreman.persephone.biz-apache-1.0-1.noarch
foreman.persephone.biz-foreman-client-1.0-1.noarch
foreman.persephone.biz-foreman-proxy-1.0-1.noarch
foreman.persephone.biz-foreman-proxy-client-1.0-1.noarch
foreman.persephone.biz-puppet-client-1.0-1.noarch
foreman.persephone.biz-qpid-broker-1.0-1.noarch
foreman.persephone.biz-qpid-client-cert-1.0-1.noarch
foreman.persephone.biz-qpid-router-client-1.0-1.noarch
foreman.persephone.biz-qpid-router-server-1.0-1.noarch
katello-3.17.0-1.el7.noarch
katello-certs-tools-2.7.1-2.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.17.0-1.el7.noarch
katello-debug-3.17.0-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-repos-3.17.0-1.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-admin-client-2.21.4-2.el7.noarch
pulp-client-1.0-1.noarch
pulp-consumer-client-2.21.4-2.el7.noarch
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.8-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.4-1.el7.noarch
pulp-puppet-tools-2.21.4-1.el7.noarch
pulp-rpm-admin-extensions-2.21.4-1.el7.noarch
pulp-rpm-consumer-extensions-2.21.4-1.el7.noarch
pulp-rpm-handlers-2.21.4-1.el7.noarch
pulp-rpm-plugins-2.21.4-1.el7.noarch
pulp-rpm-yumplugins-2.21.4-1.el7.noarch
pulp-selinux-2.21.4-2.el7.noarch
pulp-server-2.21.4-2.el7.noarch
pulpcore-selinux-1.0.0-2.el7.x86_64
puppet-foreman_scap_client-0.4.0-1.el7.noarch
python-gofer-qpid-2.12.5-3.el7.noarch
python-pulp-agent-lib-2.21.4-2.el7.noarch
python-pulp-bindings-2.21.4-2.el7.noarch
python-pulp-client-lib-2.21.4-2.el7.noarch
python-pulp-common-2.21.4-2.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.8-1.el7.noarch
python-pulp-oid_validation-2.21.4-2.el7.noarch
python-pulp-puppet-common-2.21.4-1.el7.noarch
python-pulp-repoauth-2.21.4-2.el7.noarch
python-pulp-rpm-common-2.21.4-1.el7.noarch
python-pulp-streamer-2.21.4-2.el7.noarch
python2-qpid-1.37.0-4.el7.noarch
python2-qpid-proton-0.32.0-2.el7.x86_64
python2-qpid-qmf-1.39.0-1.el7.x86_64
python3-pulp-2to3-migration-0.5.0-1.el7.noarch
python3-pulp-certguard-1.0.2-1.el7.noarch
python3-pulp-container-2.0.1-1.el7.noarch
python3-pulp-file-1.2.0-1.el7.noarch
python3-pulp-rpm-3.6.2-1.el7.noarch
python3-pulpcore-3.6.3-2.el7.noarch
qpid-cpp-client-1.39.0-1.el7.x86_64
qpid-cpp-client-devel-1.39.0-1.el7.x86_64
qpid-cpp-server-1.39.0-1.el7.x86_64
qpid-cpp-server-linearstore-1.39.0-1.el7.x86_64
qpid-dispatch-router-1.14.0-1.el7.x86_64
qpid-proton-c-0.32.0-2.el7.x86_64
qpid-qmf-1.39.0-1.el7.x86_64
qpid-tools-1.39.0-1.el7.noarch
rubygem-foreman_maintain-0.6.13-1.el7.noarch
rubygem-foreman_scap_client-0.4.0-1.el7.noarch
tfm-rubygem-actioncable-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailbox-6.0.3.1-1.el7.noarch
tfm-rubygem-actionmailer-6.0.3.1-1.el7.noarch
tfm-rubygem-actionpack-6.0.3.1-1.el7.noarch
tfm-rubygem-actiontext-6.0.3.1-1.el7.noarch
tfm-rubygem-actionview-6.0.3.1-1.el7.noarch
tfm-rubygem-activejob-6.0.3.1-1.el7.noarch
tfm-rubygem-activemodel-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-6.0.3.1-1.el7.noarch
tfm-rubygem-activerecord-import-1.0.0-2.el7.noarch
tfm-rubygem-activerecord-session_store-1.1.1-4.el7.noarch
tfm-rubygem-activestorage-6.0.3.1-1.el7.noarch
tfm-rubygem-activesupport-6.0.3.1-1.el7.noarch
tfm-rubygem-addressable-2.6.0-2.el7.noarch
tfm-rubygem-algebrick-0.7.3-7.el7.noarch
tfm-rubygem-amazing_print-1.1.0-1.el7.noarch
tfm-rubygem-ancestry-3.0.7-1.el7.noarch
tfm-rubygem-anemone-0.7.2-17.el7.noarch
tfm-rubygem-angular-rails-templates-1.1.0-1.el7.noarch
tfm-rubygem-ansi-1.5.0-2.el7.noarch
tfm-rubygem-apipie-bindings-0.4.0-1.el7.noarch
tfm-rubygem-apipie-dsl-2.2.9-1.el7.noarch
tfm-rubygem-apipie-params-0.0.5-4.el7.noarch
tfm-rubygem-apipie-rails-0.5.17-3.el7.noarch
tfm-rubygem-audited-4.9.0-3.el7.noarch
tfm-rubygem-bcrypt-3.1.12-3.el7.x86_64
tfm-rubygem-builder-3.2.4-1.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-5.el7.noarch
tfm-rubygem-clamp-1.1.2-6.el7.noarch
tfm-rubygem-concurrent-ruby-1.1.6-2.el7.noarch
tfm-rubygem-concurrent-ruby-edge-0.6.0-2.fm2_1.el7.noarch
tfm-rubygem-connection_pool-2.2.2-2.el7.noarch
tfm-rubygem-crass-1.0.6-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-4.el7.noarch
tfm-rubygem-daemons-1.2.3-6.el7.noarch
tfm-rubygem-deacon-1.0.0-4.el7.noarch
tfm-rubygem-deep_cloneable-3.0.0-3.el7.noarch
tfm-rubygem-deface-1.5.3-2.el7.noarch
tfm-rubygem-domain_name-0.5.20160310-4.el7.noarch
tfm-rubygem-dynflow-1.4.7-1.fm2_2.el7.noarch
tfm-rubygem-erubi-1.9.0-1.el7.noarch
tfm-rubygem-excon-0.76.0-1.el7.noarch
tfm-rubygem-facter-2.4.0-7.el7.x86_64
tfm-rubygem-faraday-0.15.4-2.el7.noarch
tfm-rubygem-fast_gettext-1.4.1-4.el7.noarch
tfm-rubygem-ffi-1.12.2-1.el7.x86_64
tfm-rubygem-fog-aws-3.6.5-1.el7.noarch
tfm-rubygem-fog-core-2.1.0-3.el7.noarch
tfm-rubygem-fog-json-1.2.0-3.el7.noarch
tfm-rubygem-fog-libvirt-0.7.0-2.el7.noarch
tfm-rubygem-fog-xml-0.1.2-8.el7.noarch
tfm-rubygem-foreman-tasks-3.0.1-1.fm2_2.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible-6.0.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.4-1.fm2_2.el7.noarch
tfm-rubygem-foreman_bootdisk-17.0.2-2.fm2_2.el7.noarch
tfm-rubygem-foreman_dhcp_browser-0.0.8-4.fm2_1.el7.noarch
tfm-rubygem-foreman_discovery-16.2.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_openscap-4.0.4-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution-cockpit-4.1.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.1-1.el7.noarch
tfm-rubygem-foreman_snapshot_management-1.7.1-1.fm2_1.el7.noarch
tfm-rubygem-formatador-0.2.1-12.el7.noarch
tfm-rubygem-friendly_id-5.3.0-1.el7.noarch
tfm-rubygem-fx-0.5.0-1.el7.noarch
tfm-rubygem-get_process_mem-0.2.1-4.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.8.0-2.el7.noarch
tfm-rubygem-gitlab-sidekiq-fetcher-0.6.0-1.el7.noarch
tfm-rubygem-globalid-0.4.2-1.el7.noarch
tfm-rubygem-graphql-1.8.14-2.el7.noarch
tfm-rubygem-graphql-batch-0.3.10-2.el7.noarch
tfm-rubygem-gssapi-1.2.0-7.el7.noarch
tfm-rubygem-hammer_cli-2.2.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.7-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.11-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.1.2-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.15-1.fm2_2.el7.noarch
tfm-rubygem-hammer_cli_katello-0.23.2-1.el7.noarch
tfm-rubygem-hashie-3.6.0-2.el7.noarch
tfm-rubygem-highline-1.7.8-5.el7.noarch
tfm-rubygem-http-cookie-1.0.2-4.el7.noarch
tfm-rubygem-i18n-1.8.2-1.el7.noarch
tfm-rubygem-ipaddress-0.8.0-12.el7.noarch
tfm-rubygem-jwt-2.2.1-2.el7.noarch
tfm-rubygem-kafo-5.0.1-1.el7.noarch
tfm-rubygem-kafo_parsers-1.1.0-3.el7.noarch
tfm-rubygem-kafo_wizards-0.0.1-4.el7.noarch
tfm-rubygem-katello-3.17.0-1.el7.noarch
tfm-rubygem-ldap_fluff-0.4.7-5.el7.noarch
tfm-rubygem-little-plugger-1.1.4-2.el7.noarch
tfm-rubygem-locale-2.0.9-14.el7.noarch
tfm-rubygem-logging-2.2.2-5.el7.noarch
tfm-rubygem-loofah-2.4.0-1.el7.noarch
tfm-rubygem-mail-2.7.1-1.el7.noarch
tfm-rubygem-marcel-0.3.3-1.el7.noarch
tfm-rubygem-method_source-0.9.2-1.el7.noarch
tfm-rubygem-mime-types-3.2.2-4.el7.noarch
tfm-rubygem-mime-types-data-3.2018.0812-4.el7.noarch
tfm-rubygem-mimemagic-0.3.5-1.el7.noarch
tfm-rubygem-mini_mime-1.0.2-1.el7.noarch
tfm-rubygem-mini_portile2-2.4.0-1.el7.noarch
tfm-rubygem-multi_json-1.14.1-2.el7.noarch
tfm-rubygem-multipart-post-2.0.0-2.el7.noarch
tfm-rubygem-mustermann-1.0.2-4.el7.noarch
tfm-rubygem-net-ldap-0.16.1-2.el7.noarch
tfm-rubygem-net-ping-2.0.1-4.el7.noarch
tfm-rubygem-net-scp-1.2.1-4.el7.noarch
tfm-rubygem-net-ssh-4.2.0-2.el7.noarch
tfm-rubygem-netrc-0.11.0-5.el7.noarch
tfm-rubygem-nio4r-2.5.2-1.el7.x86_64
tfm-rubygem-nokogiri-1.10.9-1.el7.x86_64
tfm-rubygem-oauth-0.5.4-4.el7.noarch
tfm-rubygem-openscap-0.4.9-3.el7.noarch
tfm-rubygem-paint-0.8.7-9.el7.noarch
tfm-rubygem-parse-cron-0.1.4-4.fm2_1.el7.noarch
tfm-rubygem-pg-1.1.4-3.el7.x86_64
tfm-rubygem-polyglot-0.3.5-2.el7.noarch
tfm-rubygem-powerbar-2.0.1-2.el7.noarch
tfm-rubygem-promise.rb-0.7.4-2.el7.noarch
tfm-rubygem-public_suffix-3.0.3-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.5.0-1.el7.noarch
tfm-rubygem-pulp_ansible_client-0.3.0-1.el7.noarch
tfm-rubygem-pulp_certguard_client-1.0.2-1.el7.noarch
tfm-rubygem-pulp_container_client-2.0.0-1.el7.noarch
tfm-rubygem-pulp_file_client-1.2.0-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.6.2-1.el7.noarch
tfm-rubygem-pulpcore_client-3.6.0-1.el7.noarch
tfm-rubygem-puma-4.3.3-4.el7.x86_64
tfm-rubygem-rabl-0.14.3-1.el7.noarch
tfm-rubygem-rack-2.2.3-1.el7.noarch
tfm-rubygem-rack-cors-1.0.2-2.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-9.el7.noarch
tfm-rubygem-rack-protection-2.0.3-4.el7.noarch
tfm-rubygem-rack-test-1.1.0-4.el7.noarch
tfm-rubygem-rails-6.0.3.1-1.el7.noarch
tfm-rubygem-rails-dom-testing-2.0.3-6.el7.noarch
tfm-rubygem-rails-html-sanitizer-1.3.0-1.el7.noarch
tfm-rubygem-rails-i18n-6.0.0-2.el7.noarch
tfm-rubygem-railties-6.0.3.1-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-3.el7.noarch
tfm-rubygem-rb-inotify-0.9.7-5.el7.noarch
tfm-rubygem-record_tag_helper-1.0.1-3.el7.noarch
tfm-rubygem-redis-4.1.2-2.el7.noarch
tfm-rubygem-responders-3.0.0-3.el7.noarch
tfm-rubygem-rest-client-2.0.2-3.el7.noarch
tfm-rubygem-rkerberos-0.1.5-18.el7.x86_64
tfm-rubygem-roadie-3.4.0-3.el7.noarch
tfm-rubygem-roadie-rails-2.1.1-2.el7.noarch
tfm-rubygem-robotex-1.0.0-21.el7.noarch
tfm-rubygem-rsec-0.4.3-4.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.1-1.el7.x86_64
tfm-rubygem-ruby2ruby-2.4.2-3.el7.noarch
tfm-rubygem-ruby_parser-3.10.1-3.el7.noarch
tfm-rubygem-rubyipmi-0.10.0-6.el7.noarch
tfm-rubygem-runcible-2.13.1-1.el7.noarch
tfm-rubygem-safemode-1.3.5-3.el7.noarch
tfm-rubygem-scoped_search-4.1.9-1.el7.noarch
tfm-rubygem-secure_headers-6.3.0-2.el7.noarch
tfm-rubygem-sequel-5.7.1-3.el7.noarch
tfm-rubygem-sexp_processor-4.10.0-6.el7.noarch
tfm-rubygem-sidekiq-5.2.7-3.el7.noarch
tfm-rubygem-sinatra-2.0.3-4.el7.noarch
tfm-rubygem-smart_proxy_ansible-3.0.1-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery-1.0.5-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_discovery_image-1.2.1-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow-0.2.4-6.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_dynflow_core-0.2.6-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_openscap-0.7.4-1.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_2.el7.noarch
tfm-rubygem-smart_proxy_remote_execution_ssh-0.3.0-4.fm2_2.el7.noarch
tfm-rubygem-sprockets-4.0.2-1.el7.noarch
tfm-rubygem-sprockets-rails-3.2.1-6.el7.noarch
tfm-rubygem-sqlite3-1.3.13-6.el7.x86_64
tfm-rubygem-sshkey-1.9.0-4.el7.noarch
tfm-rubygem-statsd-instrument-2.1.4-3.el7.noarch
tfm-rubygem-stomp-1.4.9-1.el7.noarch
tfm-rubygem-thor-1.0.1-2.el7.noarch
tfm-rubygem-thread_safe-0.3.6-5.el7.noarch
tfm-rubygem-tilt-2.0.8-4.el7.noarch
tfm-rubygem-tzinfo-1.2.6-1.el7.noarch
tfm-rubygem-unf-0.1.3-8.el7.noarch
tfm-rubygem-unf_ext-0.0.7.2-3.el7.x86_64
tfm-rubygem-unicode-0.4.4.4-3.el7.x86_64
tfm-rubygem-unicode-display_width-1.0.5-4.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-7.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-5.el7.noarch
tfm-rubygem-websocket-driver-0.7.1-1.el7.x86_64
tfm-rubygem-websocket-extensions-0.1.5-1.el7.noarch
tfm-rubygem-will_paginate-3.1.7-3.el7.noarch
tfm-rubygem-wirb-1.0.3-6.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-5.el7.noarch
tfm-rubygem-xmlrpc-0.3.0-2.el7.noarch
tfm-rubygem-zeitwerk-2.2.2-1.el7.noarch
tfm-runtime-6.1-3.el7.x86_64

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data:

I am running the OS standard TFTP.socket on both machines:

]# systemctl status tftp.socket
● tftp.socket - Tftp Server Activation Socket
Loaded: loaded (/usr/lib/systemd/system/tftp.socket; enabled; vendor preset:>
Active: active (listening) since Fri 2020-11-20 10:47:21 CET; 50min ago
Listen: [::]:69 (Datagram)
Tasks: 0 (limit: 822644)
Memory: 0B
CGroup: /system.slice/tftp.socket

I was also wondering if it were possible to somehow configure the host as some kind of tftp-proxy so Foreman could dynamically update the boot files?

lzap · November 20, 2020, 12:40pm

When you say “libvirt” it’s a very broad specification. You can configure bridged networks, NAT networks or isolated and other network types in libvirt. Where is Foreman? Where is the VM? Are they on the same network segment? Is there a router or NAT?

If I assume you use the “default” network and Foreman is running outside of this network, then this is NAT. TFTP protocol will not work over NAT because it’s stateless, you can get it working only if you enable Linux connection tracking module. But then you can run into other issues caused by NAT IP masking.

Another thing to consider is that libvirt “default” network is running its own DHCP server, that will cause other issues. Make sure you create a network without any libvirt-managed DHCP or TFTP servers.

techietubby · November 20, 2020, 12:47pm

Hi Izap,

I have a libvirt network with a bridge-device that uses NAT and this seems to be the issue:

# virsh net-dumpxml default
<network>
  <name>default</name>
  <uuid>312ccbda-b0fa-46c0-b7bd-b403f9ef41cb</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:56:2b:57'/>
  <domain name='test.local'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/tftpboot/pxeboot/'/>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <bootp file='pxelinux.0'/>
    </dhcp>
  </ip>
</network>

I can transfer a TFTP from my host system to my Foreman VM but not the other way around. I have tried disabling all the tftp.socket processes except the one on Foreman but the problem seems to be with the way that the virtual bridge NATs the UDP reply.
I have tried various combinations and the only thing that is totally fixed is that the TFTP only works when on the server and not on a VM.

My host is 192.168.0.150
Foreman is a VM on 192.168.0.99
I only have the one libvirt bridge that links my VMs.

I tried booting with addreses in the .122 NAT network but that does not work so I have created everything in Foreman in the 192.168.0.x network and I say everything works except TFTP.

techietubby · November 20, 2020, 12:55pm

When you say libvirt without NAT do you mean to remove the lines from the default network and restart, or something else. I also read that I should perhaps be using dnsmasq?

<network>
  <name>default</name>
  <uuid>312ccbda-b0fa-46c0-b7bd-b403f9ef41cb</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:56:2b:57'/>
  <domain name='test.local'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
  </ip>
</network>

techietubby · November 20, 2020, 1:03pm

Perhaps a really random thought but NAT is set for 1024 to 65535. If I moved it to say 1048 and then configured my TFTP server with restricted ports e.g. “tftp 192.168.0.99 -R 1024:1047” would that mean the packets weren’t NAT’D or this there some way that I can tell Foreman that my TFTP server is running on the host?

Dirk · November 20, 2020, 2:15pm

I prefer to use a network without dnsmasq to run bind and dhcpd myself on my virtual Foreman, so my network looks like this:

<network>
  <name>foreman</name>
  <uuid>e2866ece-7343-4569-9392-bdfd17308efa</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr1' stp='on' delay='0'/>
  <mac address='52:54:00:97:f2:80'/>
  <domain name='localdomain'/>
  <ip address='192.168.142.1' netmask='255.255.255.0'>
  </ip>
</network>

I have several of this networks running for additional demo environments and a similar setup for our training environments. So I at least can confirm that it is able to run such an environment.

If using dnsmasq has some advantages or disadvantages over this I am not sure as I always used a setup without as my setup predates dnsmasq provider in Foreman.

techietubby · November 20, 2020, 5:06pm

Hi Dirk,
Thanks for your reply. I shall update my network and reboot tomorrow and let you know.
Regards,
Andrew

techietubby · November 21, 2020, 12:08pm

Hi Dirk,

I have now updated my default virtual-network as follows:

<network>
  <name>default</name>
  <uuid>312ccbda-b0fa-46c0-b7bd-b403f9ef41cb</uuid>
  <forward mode='nat'>
<nat>
  <port start='1024' end='65535'/>
</nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:56:2b:57'/>
  <domain name='test.local'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
  </ip>
</network>

and restarted everything. Unfortunately it has made no difference:

tcpdump -i eth0 -vv port 69

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:02:48.170821 IP (tos 0x0, ttl 64, id 63332, offset 0, flags [DF], proto UDP (17), length 52)
xeon.persephone.local.56987 > foreman.local.test.tftp: [bad udp cksum 0x827b -> 0xc12c!] 24 RRQ “pxelinux.cfg” netascii
13:02:53.170908 IP (tos 0x0, ttl 64, id 65535, offset 0, flags [DF], proto UDP (17), length 52)
xeon.persephone.local.56987 > foreman.local.test.tftp: [bad udp cksum 0x827b -> 0xc12c!] 24 RRQ “pxelinux.cfg” netascii
13:02:58.171038 IP (tos 0x0, ttl 64, id 449, offset 0, flags [DF], proto UDP (17), length 52)
xeon.persephone.local.56987 > foreman.local.test.tftp: [bad udp cksum 0x827b -> 0xc12c!] 24 RRQ “pxelinux.cfg” netascii
13:03:03.171121 IP (tos 0x0, ttl 64, id 2081, offset 0, flags [DF], proto UDP (17), length 52)
xeon.persephone.local.56987 > foreman.local.test.tftp: [bad udp cksum 0x827b -> 0xc12c!] 24 RRQ “pxelinux.cfg” netascii
13:03:08.171191 IP (tos 0x0, ttl 64, id 4733, offset 0, flags [DF], proto UDP (17), length 52)
xeon.persephone.local.56987 > foreman.local.test.tftp: [bad udp cksum 0x827b -> 0xc12c!] 24 RRQ “pxelinux.cfg” netascii
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel

systemctl status tftp.socket

● tftp.socket - Tftp Server Activation Socket
Loaded: loaded (/usr/lib/systemd/system/tftp.socket; disabled; vendor preset: disabled)
Active: active (running) since Sat 2020-11-21 13:00:44 CET; 6min ago
Listen: [::]:69 (Datagram)

Is there anything else I could check?

techietubby · November 21, 2020, 5:19pm

I managed to get a lot further today. I have set: net.netfilter.nf_conntrack_helper in sysctl.conf and restarted everything on Foreman. Now I can tftp and if I do a create-host/PXEBoot through Foreman it gets right to the end of the ftfp and returns:

tftp://192.168.0.99/pxelinux.0…Connection timed out

If I do a tftp 192.168.0.99 from the host to Foreman it gets the pxelinux.0 file almost instantly.

TCPDump now shows that the packets are being received:

tcpdump -i eth0 -vv port 69

tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes

18:12:00.339096 IP (tos 0x0, ttl 64, id 1161, offset 0, flags [none], proto UDP (17), length 68)
192.168.0.200.32854 > foreman.test.local.tftp: [udp sum ok] 40 RRQ “pxelinux.0” octet blksize 1432 tsize 0
18:12:00.609528 IP (tos 0x0, ttl 64, id 1419, offset 0, flags [none], proto UDP (17), length 68)
192.168.0.200.32854 > foreman.test.local.tftp: [udp sum ok] 40 RRQ “pxelinux.0” octet blksize 1432 tsize 0
…

Any ideas?

Dirk · November 23, 2020, 7:58am

I have not touched my host’s firewall and have not enabled the firewall at the Foreman vm. Conntrack helper should not be necessary I think because tftp is udp, but if you can connect from the host at least the correct firewall rules should be present.

So it could be firewall rules on the host denying communication between VMs or routing not being allowed. On the vm it could be a problem with the source network in the rule like having firewalld with multiple zones and the rule being in the wrong one.

techietubby · November 23, 2020, 8:31am

Hi Dirk,

Thanks for your input and help.

I have managed to make this work and it seems it was a mixture of removing the DHCP from my virtual network and also tinkering with SELinux. Strangely the firewall never seems to have been an issue but the strange thing is that I had to “enable” in.tftpd as when I started it manually something stops it periodically?

We can now marked this issue as solved.