Primary Interface magically changed IP address during host creation

Support
1

Problem:
Has anyone seen this…
I have been testing Rocky 9 builds today and after having to make a few changes to the kickstart template, I finally was about finished. One more try…same host, but I did a Cancel Build / Build. No changes to the host details. This was it before I closed up for the day.
Build failed during Anaconda Software Source selection.
Then my ssh session dropped. Reconnect, and a minute later it drops again.
Start running pings on the console and see pings dropping for 40-60 seconds.

Long story short - I discovered the “new” host had automatically somehow changed the Primary interface IP address to the same IP as the Foreman host!
This has actually happened before once or twice (on another Foreman/Katello server). We just assumed in the prior occurrence that we must have inadvertently changed it ourselves, but this time I know I never did.

And holy cow, I just looked at it and it has changed back again! The server is already up and running, so it still has the IP address I assigned, but in the Host Details in Foreman it is now the Foreman server IP address.

root@fkserver:/var/log/foreman# hammer host info --name fkserver.linuxlab.local | grep -A1 "Network:"
Network:
    IPv4 address: 10.x.y.10
root@fkserver:/var/log/foreman# hammer host info --name tlrl9003.linuxlab.local | grep -A1 "Network:"
Network:
    IPv4 address: 10.x.y.10

root@tlrl9003:~# nmcli -g ipv4.addresses connection show ens160
10.x.y.55/24

This is on a Rocky 8 server, Foreman 3.5.3, Katello 4.7.4

The previous occurrence(s) were on CentOS 7, Foreman 3.3.1, Katello 4.5.1

Any ideas on this?

2

A wild guess what might have happened, it’s hard to tell what actually might have happened without knowing your provisioning workflow.

  1. “Something” (misconfigured DHCP, a script gone rogue, etc) has caused the host to get the IP of the Foreman server.
  2. Either Puppet or subscription manager (maybe also Ansible, I don’t use that myself so I don’t know how that works) was run, probably during provisioning, and sent a facts report to Foreman.
  3. Foreman parsed that facts report and you probably have the setting “Ignore interfaces facts for provisioning” set to “No” (this is the default as far as I am aware, so I assume it’s set to No)
  4. Foreman recognized that the facts reported a different IP for your host, and because of the above mentioned setting, changed the IP of the host to what has been reported by the facts.

This is all “works as designed”, except for step 1. So you should probably try and figure out what is causing the IP address to change in the first place.
Since this happened (as far as I understand) during provisioning, my first guess would be a misconfiguration on the DHCP side, maybe a missing reservation for the Foreman IP or something like that.

3

Good suggestions…just got back from a little “beach therapy”, so I’ll start testing again.
1.) I had built 5 test servers and only one exhibited this issue. They are all using the same provisioning scripts/templates, but I was tweaking as I was going along. Entirely possible “something” was different in that one.
2/3.) Puppet and subscription-manager were run…MAYBE the facts came up with the Foreman host IP during that test, but not sure how. And yep, we do have “ignore interface facts during provisioning” set to no, so I’ll do a little more digging on that end.

Also, there are two interfaces - one for DHCP on a non-routable layer 2 network, which essentially becomes unnecessary and unused once anaconda kicks in, and the other is the static IP interface which is configured in the UI. And that static IP is the one that changes.

Anyway, more testing today! Thanks!

1 Like