Host puppet yaml foreman_interfaces is missing bindings

gvde · May 9, 2022, 11:05am

Problem:
I am configuring the network connections of my client hosts with puppet using the enc information which I can check with the “Puppet YAML” button in the GUI, parameters - foreman_interfaces - attrs - bindings/bindings6

I have noticed that in some cases some information is missing for my hosts. One case is when the interface is set to virtual, then attrs is empty, i.e. “attrs: {}” although there are bindings in the facts.

For instance, facts for a client host contains this:

networking => {
  domain => "os.example.com",
  fqdn => "ostest.os.example.com",
  hostname => "ostest",
  interfaces => {
    bond0 => {
      bindings6 => [
        {
          address => "fe80::e63d:1aff:fedc:1e60",
          netmask => "ffff:ffff:ffff:ffff::",
          network => "fe80::"
        }
      ],
      ip6 => "fe80::e63d:1aff:fedc:1e60",
      mac => "e4:3d:1a:dc:1e:60",
      mtu => 9000,
      netmask6 => "ffff:ffff:ffff:ffff::",
      network6 => "fe80::",
      scope6 => "link"
    },
    bond0.601 => {
      bindings => [
        {
          address => "10.60.1.101",
          netmask => "255.255.255.0",
          network => "10.60.1.0"
        }
      ],
      bindings6 => [
        {
          address => "fe80::e63d:1aff:fedc:1e60",
          netmask => "ffff:ffff:ffff:ffff::",
          network => "fe80::"
        }
      ],
      ip => "10.60.1.101",
      ip6 => "fe80::e63d:1aff:fedc:1e60",
      mac => "e4:3d:1a:dc:1e:60",
      mtu => 9000,
      netmask => "255.255.255.0",
      netmask6 => "ffff:ffff:ffff:ffff::",
      network => "10.60.1.0",
      network6 => "fe80::",
      scope6 => "link"
    },
...

Checking facts for the client host in foreman also shows the bindings listed for interface bond0.601. However, the generated puppet yaml for this host looks like this:

  foreman_interfaces:
  - ip: 
    ip6: 
    mac: e4:3d:1a:dc:1e:60
    name: ''
    attrs:
      bindings6:
      - address: fe80::e63d:1aff:fedc:1e60
        netmask: 'ffff:ffff:ffff:ffff::'
        network: 'fe80::'
      mtu: 9000
      netmask6: 'ffff:ffff:ffff:ffff::'
      network6: 'fe80::'
      scope6: link
    virtual: true
    link: true
    identifier: bond0
    managed: true
    primary: false
    provision: false
    subnet: 
    subnet6: 
    tag: ''
    attached_to: ''
    type: Bond
    attached_devices: ens1f0np0,ens1f1np1
    mode: 802.3ad
    bond_options: xmit_hash_policy=layer2+3 miimon=100
  - ip: 10.60.1.101
    ip6: ''
    mac: 
    name: ostest.os.example.com
    attrs: {}
    virtual: true
    link: true
    identifier: bond0.601
    managed: true
    primary: true
    provision: true
    subnet:
      name: OS MGMT 601 IPv4
      network: 10.60.1.0
      mask: 255.255.255.0
      gateway: 10.60.1.230
      dns_primary: 10.60.1.1
      dns_secondary: 10.60.1.2
      from: ''
      to: ''
      boot_mode: Static
      ipam: None
      vlanid: 601
      mtu: 9000
      nic_delay:
      network_type: IPv4
      description: ''
    subnet6: 
    tag: '601'
    attached_to: bond0
    type: Interface

For whatever reasons attrs is empty for bond0.601 while bond0 shows some bindings.

Expected outcome:
Virtual interfaces should also list bindings and other interface facts.

Foreman and Proxy versions:

Foreman 3.1.2, Katello 4.3.1

Distribution and version:

CentOS 7.9

nadjaheitmann · May 11, 2022, 8:56am

@ezr-ondrej Any idea what is going on here?

ezr-ondrej · May 11, 2022, 11:37am

I’m in the dark, as I did not touch the code for this. Following the code trail tho gives me this:

attrs is just a serialized hash and thus it depends who (or rather what fact parser) have created the given interface. I’d guess that the interfaces differ in the discovery origin (maybe one created manualy and second discovered from facts?)

gvde · May 11, 2022, 12:45pm

Actually, both interfaces were created manually in the gui. You are right. After manual creation it has attrs: {}.

So the problem seems to be that the puppet facts got correctly mapped for bond0 but not for bond0.601. I guess that’s why it’s also missing the mac address.

ezr-ondrej · May 11, 2022, 10:01pm

Yeah, that sounds like a culprit. I do not understand the bond naming and it’s resolving, so can’t help you there, but you can try to inspect the puppet facts to see what name has been reported, and then manually change the name to the one puppet reports?

gvde · May 12, 2022, 4:09am

I have posted the puppet facts in the initial post. The interface is called bond0.601 on the client, in the facts and in the host identifier. So it’s identical, just like all the other interfaces.

gvde · May 12, 2022, 10:02am

From where exactly it is retrieving the information?

I have noticed that there are “networking” and “net” facts, from puppet and from rhsm.

networking.interfaces from puppet contains all interfaces including bond0.601

net.interfaces from rhsm has all interfaces except bond0.601. 601 is a child node of bond0 there.

ezr-ondrej · May 12, 2022, 1:27pm

Puppet facts are generated by running facter you can take a look there.
It is entirely possible, that the additional info is generated from rhsm facts and that puppet parser does not populate the additional attrs, thus it is available just for bond0, generated from rhsm, but not for the second bond as that is not found in rhsm facts

ezr-ondrej · May 12, 2022, 1:51pm

So what I read here:

github.com

theforeman/foreman/blob/75e7c0bf6d3fa1049931ec9000a0819e72201a05/app/models/host/base.rb#L496

    
      
            bond.errors.full_messages.each { |message| logger.warn " #{message}" }
          end
          
          
def update_subnet_from_facts(iface, keep_subnet)
            return if Setting[:update_subnets_from_facts] == 'none' || keep_subnet
            return if Setting[:update_subnets_from_facts] == 'provision' && !iface.provision
            iface.subnet = Subnet.subnet_for(iface.ip) if iface.ip_changed? && !iface.matches_subnet?(:ip, :subnet)
            iface.subnet6 = Subnet.subnet_for(iface.ip6) if iface.ip6_changed? && !iface.matches_subnet?(:ip6, :subnet6)
          end
          
          
def set_interface(attributes, name, iface)
            # update bond.attached_interfaces when interface is in the list and identifier has changed
            update_bonds(iface, name, attributes) if iface.identifier != name && !iface.virtual? && iface.persisted?
            attributes = attributes.clone
            iface.mac = attributes.delete(:macaddress)
            iface.ip = parse_ip_address(attributes.delete(:ipaddress))
            iface.ip6 = parse_ip_address(attributes.delete(:ipaddress6))
          
          
  update_subnet_from_facts(iface, attributes.delete(:keep_subnet))
          
          
  iface.virtual = attributes.delete(:virtual) || false

I think that missing attrs means no facts have been attached to the interface, if there would be any match from facts, it would populate attrs.

gvde · May 12, 2022, 1:51pm

facter facts are O.K. and contain the bindings information. rhsm/subscription-manager facts don’t contain anything like that.

If I create a vlan interface on the client host and then run puppet, it creates the interface in foreman including the bindings information.

If I create the vlan interface on the foreman server then run puppet, it creates the interface on the client (that’s my puppet class) but foreman doesn’t add the bindings from the puppet facts.

However, if I create a bond interface on the foreman server then run puppet, it creates the interface on the client and foreman adds the bindings from the puppet facts.

So it seems to me that puppet facts are not added on vlan interfaces which have been manually created on the foreman server.

ezr-ondrej · May 12, 2022, 2:33pm

So it seems to me, that the puppet facts for the second bond, just get ignored for some reason.
Could you take a look if you do not have that in ignored_interfaces (under Administer > Settings)?

I do not see any other reason why it should get ignored, but it seems like the reason behind all this

gvde · May 12, 2022, 3:32pm

That only contains this:

lo,en*v*,usb*,vnet*,macvtap*,;vdsmdummy;,veth*,docker*,tap*,qbr*,qvb*,qvo*,qr-*,qg-*,vlinuxbr*,vovsbr*,br-int

It’s not the bond but the virtual vlan interface. I have configured a vlan interface directly on an ethernet interface and I see the same problem: puppet facts are correctly loaded into foreman and bindings facts are available. But the host interface doesn’t have it:

  - ip: 10.60.2.101
    ip6: ''
    mac: 
    name: ostest.osint.example.com
    attrs: {}
    virtual: true
    link: true
    identifier: ens1f2np2.602
    managed: true
    primary: false
    provision: false
    subnet:
      name: OS INT 602 IPv4
      network: 10.60.2.0
      mask: 255.255.255.0
      gateway: ''
      dns_primary: ''
      dns_secondary: ''
      from: ''
      to: ''
      boot_mode: Static
      ipam: None
      vlanid: 
      mtu: 9000
      nic_delay: 
      network_type: IPv4
      description: Openstack Internal Network 602
    subnet6: 
    tag: '602'
    attached_to: ens1f2np2
    type: Interface

So the obvious differences: the interface has a tag which is not ‘’ and it’s type Interface and virtual…

ezr-ondrej · May 12, 2022, 3:59pm

That is different from what you’ve posted ^^ … if the identifiers do not match, the interface is not updated from facts I belive

gvde · May 12, 2022, 4:17pm

Maybe I wasn’t clear: I have added ens1f2np2.602 for testing purposes to narrow down the problem. The issue is related to the vlan interfaces.

Originally with bond0.601 but also with ens1f2np2.602. It doesn’t matter if the vlan interface is on a bond interface or a ethernet interface. The problem is that it’s a virtual vlan interface.

After seeing the set_interface function I have search through the production log and found this when puppet runs on the client:

2022-05-12T18:06:58 [I|app|70a1b145] Started POST "/api/hosts/facts" for 2001:638:70e:19::2 at 2022-05-12 18:06:58 +0200
2022-05-12T18:06:58 [I|app|70a1b145] Processing by Api::V2::HostsController#facts as JSON
2022-05-12T18:06:58 [I|app|70a1b145]   Parameters: {"facts"=>"[FILTERED]", "name"=>"ostest.os.example.com", "certname"=>"ostest.os.example.com", "apiv"=>"v2", "host"=>{"certname"=>"ostest.os.example.com ", "name"=>"ostest.os.example.com"}}
2022-05-12T18:06:58 [I|app|70a1b145] Import facts for 'ostest.os.example.com' completed. Added: 0, Updated: 10, Deleted 0 facts
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Saving bond0.601 NIC for host ostest.os.example.com failed, skipping because:
2022-05-12T18:06:58 [W|app|70a1b145]  Identifier has already been taken
2022-05-12T18:06:58 [W|app|70a1b145]  IP address has already been taken
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Not queueing Nic::Managed: ["Identifier has already been taken", "IP address has already been taken"]
2022-05-12T18:06:58 [W|app|70a1b145] Saving ens1f2np2.602 NIC for host ostest.os.example.com failed, skipping because:
2022-05-12T18:06:58 [W|app|70a1b145]  Identifier has already been taken
2022-05-12T18:06:58 [W|app|70a1b145]  IP address has already been taken
2022-05-12T18:06:58 [I|app|70a1b145] Completed 201 Created in 191ms (Views: 5.0ms | ActiveRecord: 35.8ms | Allocations: 72466)

Looks like it’s not matching the vlan interface in the facts correctly to the existing interface in the database.

gvde · May 12, 2022, 5:16pm

O.K. I found out how to get it working: as it’s a virtual interface I did not enter a mac address when creating the interface for the host, just like I did when creating the bond interface.

Now it seems for the bond interface, the mac address will be merge into the interface information, i.e. after running puppet the mac address from the facts will be in the bond interface in foreman.

For the vlan interface, however, this doesn’t work. Even though facts for the vlan interface contain a mac address it’s never entered into the interface information.

Now, I have manually copied the mac address of the vlan interface from the facts into foreman and now after the next puppet run bindings are merged into the interface information.

I guess that means it does the mapping of the vlan interface to the foreman interface using the mac address. Before, that wasn’t possible because the foreman interface didn’t have a mac address configured and didn’t import it from facts either.

Relying on the mac address is obviously flawed. The vlan interface should not rely on the mac address as the mac address can change.

For the bond interface it works fine. It learns the current mac address from the facts and keeps it up-to-date in case the mac address is changing.

gvde · May 12, 2022, 5:24pm

Not surprising: if the mac address of the base interface for the vlan interface changes (and the used mac address for the vlan interface changes on the client as well) it isn’t in foreman and thus the error message in production log returns…

ezr-ondrej · May 12, 2022, 9:21pm

Yup, looking through the code knowing what am I looking for, it is correct, both mac and identifier needs to match for virtual interface to get updated from those facts.

This was probably introduced because identifier and mac can both change for virtual interface and thus we wanted to be sure we are updating correct interface.

github.com

theforeman/foreman/blob/75e7c0bf6d3fa1049931ec9000a0819e72201a05/app/models/host/base.rb#L448

    
      
              # for other interfaces we distinguish between virtual and physical interfaces
              # for virtual devices we don't check only mac address since it's not unique,
              # if we want to update the device it must have same identifier
              else
                begin
                  macaddress = Net::Validations.normalize_mac(attributes[:macaddress])
                rescue Net::Validations::Error
                  logger.debug "invalid mac during parsing: #{attributes[:macaddress]}"
                end
          
          
      mac_based = base.where(:mac => macaddress)
                if attributes[:virtual]
                  mac_based.virtual.where(:identifier => name) || find_by_attached_mac(base, mac_based, identifier, attributes)
                elsif mac_based.physical.any?
                  mac_based.physical
                elsif !managed
                  # Unmanaged host's interfaces are just used for reporting, so overwrite based on identifier first
                  base.where(:identifier => name)
                end
            end
          end

Would you have an idea how to improve on that?

ezr-ondrej · May 12, 2022, 9:33pm

I’ve looked through history and it lives in the same form from Fixes #6444 - add support for virtual NICs · theforeman/foreman@d455f32 · GitHub mostly unchallanged ever since, just moved around few times.
Prior this commit (8 years ago), we’ve always overriden all interfaces from facts (if interface would have been missing in facts, it would get deleted).

I do not draw any conclusion out of that, I was just curious and sharing my findings here

gvde · May 13, 2022, 9:05am

Well, technically, “virtual” interfaces like tagged vlans, bonds and probably bridges as well don’t need a mac address configured. It’s optional. If I am not mistaken, it will use a specific mac address if configure otherwise it will choose one.

Usually, you won’t configure the mac address and a bond will pick a mac address from the bonded physical interfaces and a vlan will inherit the mac address from the main interface.

Thus, I think the “correct” way to do this would be to make the configuration of the mac address optional for these kind of interfaces. The used mac address shouldn’t be imported from facts in that case.

And the only way to properly identify the interface would be the identifier/name if the mac address is not configured, just like for the bonds and bridges.