Foreman and SP on the same host with dual network attachement (one private)

Hello,

This is my first post, here is some background before asking my questions :

I’m a member of a sysadmin team in charge of provisionning a CentOS Linux slurm scheduled HPC cluster of about 400 nodes.
We do the initial bootstrap (from PXE/iPXE to OS basic install, configuring IPMI in between) with xCAT and then configure/further install the OS using Ansible.

We used to use Cobbler before that and are now considering switching from xCAT to Foreman to perform the same operations. So we’re new to Formean.

I’ve played with a test installation I did from scratch on a vm using the foreman-installer, basically following the steps described in the quickstart guide and documentation.

So I decided to enable dhcp, tftp, dns and ansible smart proxies on the same host as the foreman server but since this host has a dual network attachement, one interface being linked to a public routable network, the other linked to a private non routable one on which the hosts to provision are, I’m not sure what I’m supposed to do.

What I’d like to achieve is to have foreman manage pxe, dhcp, dns, tfp for this private network.
I did create in the UI a subnet and a domain matching this network but the proxies are seen in the public network (i.e. with their public domain name).

Depending on if I use or not the --foreman-dhcp-gateway (to point to the private ip of the server host), I manage or not to tftp the pxe config file.
Then again, depending if I provide or not a “Gateway Address” in the subnet setting inside the UI I manage or not to get the kickstart file.

None of this makes sense since the host I try to provision does not need a gateway anyway. Obviously my setup is just bogus.

Here’s what I did:

  • Host I installed Foreman and the Smart Proxies on :
[root@foreman hummel]# cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core) 

[root@foreman hummel]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:8a:91:b0 brd ff:ff:ff:ff:ff:ff
    inet 157.99.101.21/24 brd 157.99.101.255 scope global eth0
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
    link/ether 00:50:56:8a:65:16 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.10/24 brd 192.168.10.255 scope global eth1
       valid_lft forever preferred_lft forever

[root@foreman etc]# hostname
foreman

[root@foreman etc]# cat /etc/resolv.conf 
; generated by /usr/sbin/dhclient-script
search it.pasteur.fr
nameserver 157.99.64.65
nameserver 157.99.64.64

[root@foreman hummel]# rpm -qa | grep -i foreman-
foreman-installer-1.17.1-1.el7.noarch
foreman-postgresql-1.17.1-1.el7.noarch
foreman-1.17.1-1.el7.noarch
foreman-proxy-1.17.1-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.12.0-2.el7.noarch
foreman-release-1.17.1-1.el7.noarch
tfm-rubygem-foreman-tasks-0.11.2-1.fm1_17.el7.noarch
foreman-cli-1.17.1-1.el7.noarch
foreman-release-scl-4-1.el7.noarch
foreman-debug-1.17.1-1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.2.4-1.fm1_17.el7.noarch
foreman-selinux-1.17.1-1.el7.noarch
  • foreman-installer options used :
foreman-installer \
--foreman-proxy-dhcp=true \ 
- -foreman-proxy-dhcp-interface=eth1 \
--foreman-proxy-dhcp-managed=true \
--foreman-proxy-dhcp-subnets="192.168.10.0/24" \
--foreman-proxy-dhcp-gateway="192.168.10.10" \
--foreman-proxy-dns=true \
--foreman-proxy-dns-forwarders="157.99.64.64"  \
--foreman-proxy-dns-managed=true \
--foreman-proxy-dns-zone=test.cluster.pasteur.fr \
--foreman-proxy-bmc=true \
--enable-foreman-proxy-plugin-ansible \
--enable-foreman-plugin-ansible \
--foreman-proxy-dns-reverse=10.168.192.in-addr.arpa

In the UI, I created the 192.168.10.0/24 subnet and the test.cluster.pasteur.fr domain and in the subnet settings, I provided 192.168.10.10 (the private ip address of the server) as the DNS servers.

smartproxies however are “discovered” as beeing in .it.pasteur.fr zone.
I also tried to add a static line in /etc/hosts with

192.168.10.10 foreman.test.cluster.pasteur.fr

but it seems to conflict to the (puppet reconfigured ?) foreman.it.pasteur.fr

so, to sum up, my questions are

  1. in my use case, do I have to use an additionnal machine to host the Smart Proxies ? Otherwise, how am I supposed to deal with the network settings ?

  2. obviously, the --foreman-dhcp-gateway corresponds to the options routers for the dhcp subnet in the dhcpd.conf : what is the difference between this parameter and the Gateway Address we can provide in the subnet settings in the UI ?

By the way : if I provide the dhcp range to the foreman-installer, I see them in dhcpd.conf but if I remove them in the answer yml file, re-rerun the installer and provide them in the UI, I don’t see them in the dhcpd.conf file : does that workd as expected ?

    • in general : how can we reset the foreman config (and it’s proxies configs) to do a fresh start ?
      I only managed to edit the answer yml file to delete options I set up on previous installer runs but didn’t want anymore ?

Thanks you and sorry for the long post.


TH.

Hello, welcome to the community. Thanks for introduction on what you do, it’s always good to hear use cases of our project.

Let me answer you in general first. Although it is possible to achieve multi-nic setups and we have some plans in improving this area in the future (HA deployments, multi-nic proxies), our Foreman installer primarily targets on single-nic Foreman server and single-nic smart-proxies. The idea is to have one Foreman server which can run one smart-proxy managing one instance of TFTP/DHCP/DNS server (one subnet) and have multiple smart-proxied deployed for each individual subnet.

The concept is simple, there is one or more Smart Proxies registered into Foreman, each one have list of feature, the feature you are lookging for is called DHCP. When you create a Subnet, you associate DHCP feature to it, this means when Foreman will be asked to create DHCP reservation it will send request to particular proxy.

Each smart proxy can be configured with multiple DHCP modules, we have ISC DHCP (installed by default), MSDHCP and several plugin modules. If you use our installer, it will set you up ISC DHCP with one subnet by default, you can create more however. Each subnet you create in dhcpd.conf you must enter into Foreman DB inventory manually.

It does not really matter where Smart Proxies are running, Foreman talks to them via HTTPS, it can be the Foreman host itself or remote site over VPN. Obviously the idea is to put Smart Proxies as close as possible to the managed clients and services.

I suggest to avoid hosts with multiple NICs if you can to avoid problems with multihoming. With some Foreman plugins installed, it can be tough to get things working as plugins rely on TLS/SSL, you can end up regenerating some certificates with aliases and stuff like that. It’s doable, but still a challenge for even experienced Foreman user.

This is one of the most common misunderstandings in our project, Subnet DB entry and subnet defined on DHCP server are two separate things. They must correspond, but creating Subnet entry in Foreman UI does not create subnet in your dhcpd.conf. You must use our installer to create entry, it can set you up one subnet, others via hiera yaml.

The gateway which goes into dhcpd.conf is gateway for leases and reservation while Gateway field in Subnet in DB is used for various things, for example for static configuration of hosts. You’d need to grep our codebase and templates to find all the places, I don’t remember. The same is for the rest, keep in mind that reservation range should not share IP addresses with pool range defined in dhcpd.conf. We have that in our docs.

Our installer is puppet-based, every time you re-run it with different options it will “do the job” and reconfigure things for you. You can run it as many times as you want until you get a working setup. Not sure what happens when you remove from YAML file, but simply use the options.

Hello Izap,

thanks for your reply.

Just to get it straight : I only have one subnet to manage, the thing is that this subnet is a private non-routable subnet.
So at some point, either foreman server itself or the Smart Proxies will have to be dual nic : one nic to reach the private subet (where the to-be-provisionned nodes reside) and one nic on a public routable network for any admin to log (directly) into the machine. How am I supposed to deal with that using the foreman-installer ?
One thing seems clear though is that only one Smart Proxy should be enough for me, be it on the same host as foreman server or on a dedicated host.

Note : my clients are all single nic, on the private network.

Get it for the possible confusion : I didn’t confuse the subnet UI object and the dhcpd subnet though I admit I did confuse the gateway.

Am I correct assuming the client talks to the proxy and not directly to the foreman server ?
If so, maybe the problem I had is that foreman puppeted-configured at install time the public side (hostname) of my server (I see foreman.it.pasteur.fr in the UI) when I said with install options "I want a dhcp proxy, I want a dns proxy, …etc…).
Maybe because of my search it.pasteur.fr in resolv.conf or maybe because eth0 was my public nic ?

Basically, what I am saying is that I don’t see a problem in server <-> proxy talking to each other (in HTTPS) through the public nic : thus, for that matter, it may be legitimate to have foreman.it.pasteur.fr (public) as a Smart Proxy (as seen in the UI)).
But there may be a problem if the client tries to route itself to the public side of the proxy : it then may be a problem for that matter to have foreman.it.pasteur.fr instead of foreman.test.cluster.fr (private) configured as a Smart Proxy (as seen in the UI)).
As a matter of fact, there should be no routing involved between the client and the proxy since they are on the same (private) non-routable network.
I wonder if that’s not what is happening since, without --foreman-dhcp-gateway I cannot pass the tftp stage, with it, the installer starts (and for some reason I need to have the “other” gateway you talked about (the one in the subnet in the UI) for kickstart file to be downloaded.

But it does that only for added features, doesn’t it ? For instance, if for that particular option (--foreman-dhcp-gateway), once set, I don’t think it is “removed” after another foreman-installer run without it : am I mistaking (could well be ;-)).

What would be the correct method to wipe out the foreman config to start as frech as at first install time ?

One last thing about the confusion between subnets : does that explain (subnets in DB is not the same as in dhcp) that when I leave the --foreman-proxy -dhcp-range out I don’t see it in the dhcpd config file ? If so, what is the range in the subnet UI we can fill out ?

Thanks again and sorry for the confusion. I may have started with a non trivial use case :wink:


TH

We don’t have documented unfortunately, depends on services you plan to use. I can tell that for TFTP and DHCP you simply configure the server with hostname set to the interface which is Foreman-facing so installer generates HTTPS certificate for Foreman-Proxy communication and you give TFTP and DHCP services inteface which is client (server) facing.

But we have services which are talking to smart proxy hostname directly, for example template endpoint (provision template download). Foreman renders URL which you want to download from as the Foreman-facing hostname which will not work (resolves to wrong IP address). There are various workarounds, the best is I think changing the template and using host/hostgroup/subnet parameter to override the hostname. @sean797 ?

Since Foreman is not aware of multi-home setups, you are on your own here and you will run into issues you need to resolve. But it’s been done in the past, we can help you when you hit those. I recommend to do POC installation in clean network without multi-homing and learn about how Foreman works first.

All client communication is not via proxy by default, but you can configure this. If you had single NIC on proxy, installer can easily set you things up. But in case of multi-nic this is more complex problem. I am familiar with PXE provisioning, so in that case TFTP, DHCP, DNS and kickstart request can be all completely proxy only. This is same for Puppet - master runs on Proxy itself. @ekohl do we have working installer setups for multi-nic proxies?

To force clients to download kickstarts from proxy instead of foreman, you need to install module called “smart proxy template”. Here is a snippet I use to install PXE setup on my single-nic environment:

 foreman-installer -v --scenario $SCENARIO \
  --foreman-admin-password=$PASS \
  --foreman-organizations-enabled true \
  --foreman-initial-organization=$ORG \
  --foreman-locations-enabled true \
  --foreman-initial-location=$LOC \
  --enable-foreman-plugin-discovery \
  --enable-foreman-plugin-bootdisk \
  --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh \
  --enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible \
  --foreman-proxy-http=true \
  --foreman-proxy-dns true \
  --foreman-proxy-dns-interface eth0 \
  --foreman-proxy-dns-forwarders 192.168.${NATLAN}.1 \
  --foreman-proxy-dns-zone nat.lan \
  --foreman-proxy-dns-reverse ${NATLAN}.168.192.in-addr.arpa \
  --foreman-proxy-dhcp true \
  --foreman-proxy-dhcp-interface eth0 \
  --foreman-proxy-dhcp-gateway=192.168.${NATLAN}.1 \
  --foreman-proxy-dhcp-range="192.168.${NATLAN}.10 192.168.${NATLAN}.109" \
  --foreman-proxy-dhcp-nameservers="192.168.${NATLAN}.${IP}" \
  --foreman-proxy-tftp true \
  --foreman-proxy-tftp-servername=192.168.${NATLAN}.${IP} \
  --foreman-proxy-puppet true \
  --foreman-proxy-puppetca true \
  --foreman-proxy-templates true \
  --foreman-proxy-logs true \
  --foreman-proxy-register-in-foreman true

The option you are looking for probably is “–foreman-proxy-templates true”.

I admit this is confusing and @ekohl correct me if I am wrong, but everytime you call foreman-installer command it adds new options to YAML and reruns puppet. So basically it “remembers” already overriden options.

Hmm question or kafo devs (foreman-installer framework), maybe @mbacovsky or @ekohl? I have no idea. If I mess up, I dig in shell history and “correct” all options by providing them again.

Not puppet expert, not sure why you don’t see dhcpd.config file. Define “to see” verb, like the file literally disappears?

Foreman is a beast! Really, it has hundreds of features, plugins, hidden features and (few) bugs :slight_smile: Absolutely do start with simple use case and learn.

Ok thanks for your time.
I’d try to mix all these ideas.


Thomas.

We’ve run into a similar issue before as we have a similar setup. The solution we arrived on was leaving foreman and the proxy configured entirely with it’s public facing name/domain, and allowing it to listen/act on the private interface. Two options for allowing foreman to function in this configuration:

  1. Use bind views to present the private IP address of the foreman server against the public FQDN. Not sure if the puppet-dns module (used by the foreman-installer script) is capable of configuring this through the foreman-installer or whether it’ll need to be a custom job. Anything on your private subnet will then be able to route to the private address of your foreman server.
  2. Use local name resolution (/etc/hosts) on the private nodes to resolve the public hostname/domain to the private address of the foreman server

Incrementally, the project is improving in this space but it’s not perfect yet…

Hello again,

I started my setup all over again. I’m still installing/running on a single host with 2 nics but I switched their order and set up the hostname, so as almost eveything resolves/points to the private subnet (the one I want to install clients on). It looks like this :

  • eth0 : private IP
  • eth1 : public IP
  • hostname : private hostname (foreman.dev.cluster.pasteur.fr) set via hostnamectl set-hostname --static (CentOS 7)
  • /etc/sysconfig/network : no hostname
  • /etc/sysconfig/network-scripts/ifcfg-eth0 : DOMAIN=“dev.cluster.pasteur.fr
  • /etc/sysconfig/network-scripts/ifcfg-eth1 : DOMAIN=“it.pasteur.fr

And I installed with :
# foreman-installer
–foreman-proxy-dhcp=true
–foreman-proxy-dhcp-interface=eth0
–foreman-proxy-dhcp-managed=true
–foreman-proxy-dhcp-subnets=“192.168.10.0/24”
–foreman-proxy-dhcp-gateway=“192.168.10.10”
–foreman-proxy-tftp=true
–foreman-proxy-tftp-managed=true
–foreman-proxy-tftp-servername=“192.168.10.10”
–foreman-proxy-dns=true
–foreman-proxy-dns-forwarders=“157.99.64.64”
–foreman-proxy-dns-managed=true
–foreman-proxy-dns-zone=dev.cluster.pasteur.fr
–foreman-proxy-bmc=true
–enable-foreman-proxy-plugin-ansible
–enable-foreman-plugin-ansible
–foreman-proxy-dns-reverse=10.168.192.in-addr.arpa

I created in the UI the needed objects to install a client. All in the same private domain or network.
So basically nothing foreman related states the public name or ip.

Thus I, as before, am unable to fetch the kickstart file after the PXE step.
As a matter of fact, I see this line in the named zone file :

$ORIGIN dev.cluster.pasteur.fr.
foreman                 A       157.99.101.21
$TTL 86400      ; 1 day
test-409                A       192.168.10.200

-> this seems to be the problem. Do you know where the public ip for foreman comes from here (instead of the private ip) ?

I think it could work if I put in the UI subnet form the foreman private ip as the gateway but the correct way to proceed seems to register the good IP in the zone file. Do you have any idea of how to do that ?

Thanks


TH

Two things to check:

  1. The unattended_url setting in Foreman (probably not the issue, but worth checking)
  2. The “tftp_servername” setting on the proxy (I think it’s in the dhcp.yml plugin config)

(2) is usually the issue for PXE problems. Try setting it to the IP you want to use for the TFTP server. If you change it, you’ll need to rewrite the lease for the test host, use the Rebuild Config option in the Foreman UI.

Thanks for your reply

This one is correct.

I explicitly passed this on the command line to the private ip I was interested in.

Could the reason be that the public nic is the only one to have a default gateway ?

It’s possible. I’ve re-read the post, and realised I misunderstood - I thought PXE was failing, but you’re getting past that.

If you can boot an installer, can you get a shell on it? I know the Debian installer has a few virtual terminals you can switch to, but I’m less familiar with the Anaconda ones. If you can, it might be worth checking the network config live, and seeing where the issue is.

17:58:37.552276 ARP, Reply 192.168.10.200 is-at 0c:c4:7a:1e:8a:5d (oui Unknown), length 46
17:58:37.555320 ARP, Request who-has foreman.dev.cluster.pasteur.fr tell 192.168.10.200, length 46
17:58:37.555340 ARP, Reply foreman.dev.cluster.pasteur.fr is-at 00:50:56:8a:8b:09 (oui Unknown), length 28

17:58:37.555526 IP 192.168.10.200.44340 > foreman.dev.cluster.pasteur.fr.domain: 30597+ A? foreman.dev.cluster.pasteur.fr. (48)
17:58:37.555547 IP 192.168.10.200.44340 > foreman.dev.cluster.pasteur.fr.domain: 61248+ AAAA? foreman.dev.cluster.pasteur.fr. (48)
17:58:37.558082 IP foreman.dev.cluster.pasteur.fr.domain > 192.168.10.200.44340: 61248* 0/1/0 (89)
17:58:37.558101 IP foreman.dev.cluster.pasteur.fr.domain > 192.168.10.200.44340: 30597* 1/1/0 A 157.99.101.21 (78)

This tcpdump snippet shows to me that the client gets the right dns address but the wrong answer ? Which matches what’s in the zone db file of named ?

Sorry, it was my mistake : for some reason, in the answers.yml file was a foreman-proxy-dns-interface set to the wrong interface…
It echoes however my question above about how to completely reset a foreman config and if puppet just adds/change features provided on the installer of the command line…

Thanks anyway…

I’m almost there :

  • PXE works
  • kickstart file download works

given that I provide the private non-routed ip/name of foreman in /etc/hosts.

For the same reason (I’m installing clients on a private non-routed subnet) I also add a nic in that subnet on my media host (vmware http CentOS mirror) and the corresponding entry in /etc/hosts.

but now anaconda wants to access this host : can I add this host entry in the foreman managed dns via the UI or do I have to externally (nsupdate for instance) add it to the dynamic zone ?

Thanks


TH
but now

For this one : I’m not really sure why it’s needed : as a matter of fact, the next-server in the generated dhcpd.conf file is an IP address, not a name.
However, I notice that if I don’t provide this command line option, the “Services” tab for the Smart Proxies section in the UI shows only tftp “Version” but “false” for “tftp server”.

What am I missing ?

Thanks.


TH

Next server must be an IP this is required by PXE. When this is not set via tftp_servername then Foreman grabs smart proxy hostname and does reverse DNS PTR search for an IP and use that IP for the filename option.

It was irrelevant, I asked for it based on mis-reading the question. It only impacts PXE, which you say is working, so it’s not the problem.

As for your DNS, we don’t really have an interface for adding arbitrary records, AFAIK - only the host records that are automated when creating/updating hosts. So you’ll want to use nsupdate or whatever your favourite tool is for adding this (this irritates me every time I need to add some CNAMES…)

This explains my problem then (tftp works only when I provide this command line option). As a matter of fact, I can see a A record for forman in my forward dns zone but no ptr record in my reverse dns zone : is that normal ?

# grep -i foreman /etc/hosts
192.168.10.10	foreman.dev.cluster.pasteur.fr

# cat db.dev.cluster.pasteur.fr
$ORIGIN .
$TTL 10800	; 3 hours
dev.cluster.pasteur.fr	IN SOA	foreman.dev.cluster.pasteur.fr. root.dev.cluster.pasteur.fr. (
				26         ; serial
				86400      ; refresh (1 day)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				3600       ; minimum (1 hour)
				)
			NS	foreman.dev.cluster.pasteur.fr.
$ORIGIN dev.cluster.pasteur.fr.
foreman			A	192.168.10.10
$TTL 600	; 10 minutes
mirrors			A	192.168.10.3

]# cat db.10.168.192.in-addr.arpa 
$ORIGIN .
$TTL 10800	; 3 hours
10.168.192.in-addr.arpa	IN SOA	foreman.dev.cluster.pasteur.fr. root.10.168.192.in-addr.arpa. (
				20         ; serial
				86400      ; refresh (1 day)
				3600       ; retry (1 hour)
				604800     ; expire (1 week)
				3600       ; minimum (1 hour)
				)
			NS	foreman.dev.cluster.pasteur.fr.
$ORIGIN 10.168.192.in-addr.arpa.
$TTL 600	; 10 minutes
3			PTR	mirrors.dev.cluster.pasteur.fr.

That’s how I added mirros in the above zones snippets.

Thanks.

I must add this is strange though since I can however see a correct next-server in my dhcpd.conf file

# Bootfile Handoff
next-server 192.168.10.10;

But again, I’m not really sure if it doesn’t come from a previous foreman-installer run with the --foreman-proxy-tftp-servername option set…Could that be possible ?

If this zone is managed by foreman, then this is possible. You associated Proxy for Domain but not for Subnet. In that case, Foreman manages only forward records. Associate both and you will get PTR as well.

This is probably set by our installer, yes. Plus it also sets the YAML configu for the TFTP module.

I started from a fresh install (I’m playing/experimenting installation scenarii) : I didn’t create any subnet yet in the UI indeed but neither did I create a domain.
Here was my command line

foreman-installer --foreman-proxy-dhcp=true --foreman-proxy-dhcp-interface=eth0 --foreman-proxy-dhcp-managed=true --foreman-proxy-dhcp-subnets="192.168.10.0/24" --foreman-proxy-tftp=true --foreman-proxy-tftp-managed=true --foreman-proxy-dns=true --foreman-proxy-dns-interface=eth0 --foreman-proxy-dns-forwarders="157.99.64.64" --foreman-proxy-dns-managed=true --foreman-proxy-dns-zone=dev.cluster.pasteur.fr --foreman-proxy-bmc=true --enable-foreman-proxy-plugin-ansible --enable-foreman-plugin-ansible --foreman-proxy-dns-reverse=10.168.192.in-addr.arpa