Primary virtual interface detection

lzap · August 24, 2018, 11:34am

Hello,

I am fighting with a bug in discovery - when primary/provisioning interface is on VLAN (or it’s virtual), Foreman core detection does not set the virtual NIC as primary, but the NIC that the interface is attached to.

I am wondering if this is a known bug. When I am debugging it and everything seems to be correct during parsing phase and even after reload call. But in later stage (populate_discovery_fields_from_facts) primary interface is wrongly associated. This looks like some ActiveRecord magic, detection code see everything correctly but it’s actually stored incorrectly.

Any advice?

lzap · August 24, 2018, 11:41am

Here is the result, note eno2.2905 is supposed to be primary not eno2.

 [[430, "eno1", "24:6e:96:5a:f7:34", false],
 [429, "eno2", "24:6e:96:5a:f7:35", true],
 [437, "eno2.2905", "24:6e:96:5a:f7:35", false],
 [431, "eno3", "24:6e:96:5a:f7:36", false],
 [432, "eno4", "24:6e:96:5a:f7:37", false],
 [433, "enp129s0f0", "a0:36:9f:e5:96:24", false],
 [434, "enp129s0f1", "a0:36:9f:e5:96:25", false],
 [435, "enp129s0f2", "a0:36:9f:e5:96:26", false],
 [436, "enp129s0f3", "a0:36:9f:e5:96:27", false],
 [438, "ipmi", "84:7b:eb:f6:aa:c0", false]]

Thanks to audit, I am able to easily track how we create NICs:

We have following interfaces 'eno1, eno2, eno3, eno4, eno2_2905, enp129s0f0, enp129s0f1, enp129s0f2, enp129s0f3' based on facts
Interface eno1 facts: {"macaddress"=>"24:6e:96:5a:f7:34", "mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"true", "lldp_neighbor_chassisid"=>"a0:e0:af:0d:9c:62", "lldp_neighbor_portid"=>"Eth1/17", "lldp_neighbor_sysname"=>"CLFFLPRDLEF93108-109", "lldp_neighbor_mngaddr_ipv4"=>"10.101.57.109", "lldp_neighbor_pvid"=>176}
Interface eno2 facts: {"macaddress"=>"24:6e:96:5a:f7:35", "mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"true", "lldp_neighbor_chassisid"=>"a0:e0:af:0c:aa:c6", "lldp_neighbor_portid"=>"Eth1/17", "lldp_neighbor_sysname"=>"CLFFLPRDLEF93108-110", "lldp_neighbor_mngaddr_ipv4"=>"10.101.57.110", "lldp_neighbor_pvid"=>176}
Interface eno3 facts: {"macaddress"=>"24:6e:96:5a:f7:36", "mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"true"}
Interface eno4 facts: {"macaddress"=>"24:6e:96:5a:f7:37", "mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"false"}
Interface eno2_2905 facts: {"ipaddress"=>"10.26.48.106", "macaddress"=>"24:6e:96:5a:f7:35", "netmask"=>"255.255.255.0", "mtu"=>1500, "network"=>"10.26.48.0", "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"true"}
Interface enp129s0f0 facts: {"macaddress"=>"a0:36:9f:e5:96:24", "mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"false"}
Interface enp129s0f1 facts: {"macaddress"=>"a0:36:9f:e5:96:25", "mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"false"}
Interface enp129s0f2 facts: {"macaddress"=>"a0:36:9f:e5:96:26", "mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"false"}
Interface enp129s0f3 facts: {"macaddress"=>"a0:36:9f:e5:96:27", "mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"false"}
Discovery fact parser detected primary interface: ["eno2.2905", {"ipaddress"=>"10.26.48.106", "macaddress"=>"24:6e:96:5a:f7:35", "netmask"=>"255.255.255.0", "mtu"=>1500, "network"=>"10.26.48.0", "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "link"=>"true", "keep_subnet"=>true, "virtual"=>true, "attached_to"=>"eno2", "tag"=>"2905"}]
Nic::Managed (429) update event on mac , 24:6e:96:5a:f7:35
Nic::Managed (429) update event on identifier , eno2.2905
Saving eno1 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (430) create event on mac 24:6e:96:5a:f7:34
Nic::Managed (430) create event on ip 
Nic::Managed (430) create event on type Nic::Managed
Nic::Managed (430) create event on name 
Nic::Managed (430) create event on host_id 285
Nic::Managed (430) create event on subnet_id 
Nic::Managed (430) create event on domain_id 
Nic::Managed (430) create event on attrs {"mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "lldp_neighbor_chassisid"=>"a0:e0:af:0d:9c:62", "lldp_neighbor_portid"=>"Eth1/17", "lldp_neighbor_sysname"=>"CLFFLPRDLEF93108-109", "lldp_neighbor_mngaddr_ipv4"=>"10.101.57.109", "lldp_neighbor_pvid"=>176}
Nic::Managed (430) create event on provider 
Nic::Managed (430) create event on username 
Nic::Managed (430) create event on password [redacted]
Nic::Managed (430) create event on virtual false
Nic::Managed (430) create event on link true
Nic::Managed (430) create event on identifier eno1
Nic::Managed (430) create event on tag 
Nic::Managed (430) create event on attached_to 
Nic::Managed (430) create event on managed false
Nic::Managed (430) create event on mode balance-rr
Nic::Managed (430) create event on attached_devices 
Nic::Managed (430) create event on bond_options 
Nic::Managed (430) create event on primary false
Nic::Managed (430) create event on provision false
Nic::Managed (430) create event on compute_attributes {}
Nic::Managed (430) create event on ip6 
Nic::Managed (430) create event on subnet6_id 
Saving eno2 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (429) update event on identifier eno2.2905, eno2
Nic::Managed (429) update event on attrs {}, {"mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true", "lldp_neighbor_chassisid"=>"a0:e0:af:0c:aa:c6", "lldp_neighbor_portid"=>"Eth1/17", "lldp_neighbor_sysname"=>"CLFFLPRDLEF93108-110", "lldp_neighbor_mngaddr_ipv4"=>"10.101.57.110", "lldp_neighbor_pvid"=>176}
B PRIMARY INTERFACE IDENT: 
Saving eno3 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (431) create event on mac 24:6e:96:5a:f7:36
Nic::Managed (431) create event on ip 
Nic::Managed (431) create event on type Nic::Managed
Nic::Managed (431) create event on name 
Nic::Managed (431) create event on host_id 285
Nic::Managed (431) create event on subnet_id 
Nic::Managed (431) create event on domain_id 
Nic::Managed (431) create event on attrs {"mtu"=>1500, "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (431) create event on provider 
Nic::Managed (431) create event on username 
Nic::Managed (431) create event on password [redacted]
Nic::Managed (431) create event on virtual false
Nic::Managed (431) create event on link true
Nic::Managed (431) create event on identifier eno3
Nic::Managed (431) create event on tag 
Nic::Managed (431) create event on attached_to 
Nic::Managed (431) create event on managed false
Nic::Managed (431) create event on mode balance-rr
Nic::Managed (431) create event on attached_devices 
Nic::Managed (431) create event on bond_options 
Nic::Managed (431) create event on primary false
Nic::Managed (431) create event on provision false
Nic::Managed (431) create event on compute_attributes {}
Nic::Managed (431) create event on ip6 
Nic::Managed (431) create event on subnet6_id 
Saving eno4 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (432) create event on mac 24:6e:96:5a:f7:37
Nic::Managed (432) create event on ip 
Nic::Managed (432) create event on type Nic::Managed
Nic::Managed (432) create event on name 
Nic::Managed (432) create event on host_id 285
Nic::Managed (432) create event on subnet_id 
Nic::Managed (432) create event on domain_id 
Nic::Managed (432) create event on attrs {"mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (432) create event on provider 
Nic::Managed (432) create event on username 
Nic::Managed (432) create event on password [redacted]
Nic::Managed (432) create event on virtual false
Nic::Managed (432) create event on link false
Nic::Managed (432) create event on identifier eno4
Nic::Managed (432) create event on tag 
Nic::Managed (432) create event on attached_to 
Nic::Managed (432) create event on managed false
Nic::Managed (432) create event on mode balance-rr
Nic::Managed (432) create event on attached_devices 
Nic::Managed (432) create event on bond_options 
Nic::Managed (432) create event on primary false
Nic::Managed (432) create event on provision false
Nic::Managed (432) create event on compute_attributes {}
Nic::Managed (432) create event on ip6 
Nic::Managed (432) create event on subnet6_id 
Saving enp129s0f0 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (433) create event on mac a0:36:9f:e5:96:24
Nic::Managed (433) create event on ip 
Nic::Managed (433) create event on type Nic::Managed
Nic::Managed (433) create event on name 
Nic::Managed (433) create event on host_id 285
Nic::Managed (433) create event on subnet_id 
Nic::Managed (433) create event on domain_id 
Nic::Managed (433) create event on attrs {"mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (433) create event on provider 
Nic::Managed (433) create event on username 
Nic::Managed (433) create event on password [redacted]
Nic::Managed (433) create event on virtual false
Nic::Managed (433) create event on link false
Nic::Managed (433) create event on identifier enp129s0f0
Nic::Managed (433) create event on tag 
Nic::Managed (433) create event on attached_to 
Nic::Managed (433) create event on managed false
Nic::Managed (433) create event on mode balance-rr
Nic::Managed (433) create event on attached_devices 
Nic::Managed (433) create event on bond_options 
Nic::Managed (433) create event on primary false
Nic::Managed (433) create event on provision false
Nic::Managed (433) create event on compute_attributes {}
Nic::Managed (433) create event on ip6 
Nic::Managed (433) create event on subnet6_id 
Saving enp129s0f1 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (434) create event on mac a0:36:9f:e5:96:25
Nic::Managed (434) create event on ip 
Nic::Managed (434) create event on type Nic::Managed
Nic::Managed (434) create event on name 
Nic::Managed (434) create event on host_id 285
Nic::Managed (434) create event on subnet_id 
Nic::Managed (434) create event on domain_id 
Nic::Managed (434) create event on attrs {"mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (434) create event on provider 
Nic::Managed (434) create event on username 
Nic::Managed (434) create event on password [redacted]
Nic::Managed (434) create event on virtual false
Nic::Managed (434) create event on link false
Nic::Managed (434) create event on identifier enp129s0f1
Nic::Managed (434) create event on tag 
Nic::Managed (434) create event on attached_to 
Nic::Managed (434) create event on managed false
Nic::Managed (434) create event on mode balance-rr
Nic::Managed (434) create event on attached_devices 
Nic::Managed (434) create event on bond_options 
Nic::Managed (434) create event on primary false
Nic::Managed (434) create event on provision false
Nic::Managed (434) create event on compute_attributes {}
Nic::Managed (434) create event on ip6 
Nic::Managed (434) create event on subnet6_id 
Saving enp129s0f2 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (435) create event on mac a0:36:9f:e5:96:26
Nic::Managed (435) create event on ip 
Nic::Managed (435) create event on type Nic::Managed
Nic::Managed (435) create event on name 
Nic::Managed (435) create event on host_id 285
Nic::Managed (435) create event on subnet_id 
Nic::Managed (435) create event on domain_id 
Nic::Managed (435) create event on attrs {"mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (435) create event on provider 
Nic::Managed (435) create event on username 
Nic::Managed (435) create event on password [redacted]
Nic::Managed (435) create event on virtual false
Nic::Managed (435) create event on link false
Nic::Managed (435) create event on identifier enp129s0f2
Nic::Managed (435) create event on tag 
Nic::Managed (435) create event on attached_to 
Nic::Managed (435) create event on managed false
Nic::Managed (435) create event on mode balance-rr
Nic::Managed (435) create event on attached_devices 
Nic::Managed (435) create event on bond_options 
Nic::Managed (435) create event on primary false
Nic::Managed (435) create event on provision false
Nic::Managed (435) create event on compute_attributes {}
Nic::Managed (435) create event on ip6 
Nic::Managed (435) create event on subnet6_id 
B PRIMARY INTERFACE IDENT: 
Saving enp129s0f3 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (436) create event on mac a0:36:9f:e5:96:27
Nic::Managed (436) create event on ip 
Nic::Managed (436) create event on type Nic::Managed
Nic::Managed (436) create event on name 
Nic::Managed (436) create event on host_id 285
Nic::Managed (436) create event on subnet_id 
Nic::Managed (436) create event on domain_id 
Nic::Managed (436) create event on attrs {"mtu"=>1500, "duplex"=>"unknown! (255)", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (436) create event on provider 
Nic::Managed (436) create event on username 
Nic::Managed (436) create event on password [redacted]
Nic::Managed (436) create event on virtual false
Nic::Managed (436) create event on link false
Nic::Managed (436) create event on identifier enp129s0f3
Nic::Managed (436) create event on tag 
Nic::Managed (436) create event on attached_to 
Nic::Managed (436) create event on managed false
Nic::Managed (436) create event on mode balance-rr
Nic::Managed (436) create event on attached_devices 
Nic::Managed (436) create event on bond_options 
Nic::Managed (436) create event on primary false
Nic::Managed (436) create event on provision false
Nic::Managed (436) create event on compute_attributes {}
Nic::Managed (436) create event on ip6 
Nic::Managed (436) create event on subnet6_id 
Saving eno2.2905 NIC for host kelly-alvin-vlahos-alconcel
Nic::Managed (437) create event on mac 24:6e:96:5a:f7:35
Nic::Managed (437) create event on ip 10.26.48.106
Nic::Managed (437) create event on type Nic::Managed
Nic::Managed (437) create event on name 
Nic::Managed (437) create event on host_id 285
Nic::Managed (437) create event on subnet_id 
Nic::Managed (437) create event on domain_id 
Nic::Managed (437) create event on attrs {"netmask"=>"255.255.255.0", "mtu"=>1500, "network"=>"10.26.48.0", "speed"=>"1000", "duplex"=>"full", "port"=>"Twisted Pair", "auto_negotiation"=>"true"}
Nic::Managed (437) create event on provider 
Nic::Managed (437) create event on username 
Nic::Managed (437) create event on password [redacted]
Nic::Managed (437) create event on virtual true
Nic::Managed (437) create event on link true
Nic::Managed (437) create event on identifier eno2.2905
Nic::Managed (437) create event on tag 2905
Nic::Managed (437) create event on attached_to eno2
Nic::Managed (437) create event on managed false
Nic::Managed (437) create event on mode balance-rr
Nic::Managed (437) create event on attached_devices 
Nic::Managed (437) create event on bond_options 
Nic::Managed (437) create event on primary false
Nic::Managed (437) create event on provision false
Nic::Managed (437) create event on compute_attributes {}
Nic::Managed (437) create event on ip6 
Nic::Managed (437) create event on subnet6_id 
Saving ipmi NIC for host kelly-alvin-vlahos-alconcel
Nic::BMC (438) create event on mac 84:7b:eb:f6:aa:c0
Nic::BMC (438) create event on ip 10.101.67.236
Nic::BMC (438) create event on type Nic::BMC
Nic::BMC (438) create event on name 
Nic::BMC (438) create event on host_id 285
Nic::BMC (438) create event on subnet_id 
Nic::BMC (438) create event on domain_id 
Nic::BMC (438) create event on attrs {"enabled"=>true, "ipaddress_source"=>"Static Address", "subnet_mask"=>"255.255.240.0", "gateway"=>"10.101.64.1", "1_ipaddress_source"=>"Static Address", "1_ipaddress"=>"10.101.67.236", "1_subnet_mask"=>"255.255.240.0", "1_macaddress"=>"84:7b:eb:f6:aa:c0", "1_gateway"=>"10.101.64.1"}
Nic::BMC (438) create event on provider IPMI
Nic::BMC (438) create event on username 
Nic::BMC (438) create event on password [redacted]
Nic::BMC (438) create event on virtual false
Nic::BMC (438) create event on link true
Nic::BMC (438) create event on identifier ipmi
Nic::BMC (438) create event on tag 
Nic::BMC (438) create event on attached_to 
Nic::BMC (438) create event on managed false
Nic::BMC (438) create event on mode balance-rr
Nic::BMC (438) create event on attached_devices 
Nic::BMC (438) create event on bond_options 
Nic::BMC (438) create event on primary false
Nic::BMC (438) create event on provision false
Nic::BMC (438) create event on compute_attributes {}
Nic::BMC (438) create event on ip6 
Nic::BMC (438) create event on subnet6_id 
Unable to assign subnet - reboot trigger may not be possible, primary interface is missing IP address

lzap · August 24, 2018, 12:03pm

And my analysis follows, @Marek_Hulan might know better.

It looks like we create primary interface in advance, then we save identifier and mac there and then there’s loop over all interfaces which creates all the others but overwrites the primary one completely.

Marek_Hulan · August 24, 2018, 12:27pm

I don’t think this is known issue. I think the problem comes from the log quoted above. It seems when eno2 is being updated, it searches interfaces by MAC and in fact updates eno2.2905 instead. The logic to find the interface object to be updated is here [1]. In this case, I think we didn’t detect the vlan interface as virtual therefore it searches using mac also among physical. It would help to have a reproducer and put some pry to the place.

[1] https://github.com/theforeman/foreman/blob/dc603185819618d28c406fef2245b53745ed423d/app/models/host/base.rb#L411-L425

lzap · August 24, 2018, 2:02pm

Here is a patch that contains new test.

But my solution breaks two other tests and I am not really sure if this is a good direction tho. I am giving up for today, this needs fresh brain on Monday.

https://github.com/theforeman/foreman/pull/5989