Foreman networking tasks

[Marek_Hulan]

Hi all

as I promised during sprint demo 28 [1] in which I showed recent changes to
networking I'd like to start discussion over items that we've identified as
important regarding further networking stuff.

The goal is to find out which features are most important (feel free to +1) and
to find out how they should be implemented. There are many open questions. If
this is too complex for email discussion we could do some deep dive and
discuss each task separately. When we are clear on items I'll clean up redmine
and convert tasks to new issues.

Here's the list of items:

(1) Refactoring of existing code

• extract primary interface from host
• rework static flag in provisioning
• allow searching of hosts based on mac address (including all interfaces)
• rename Nic::Managed or merge with Nic::Interface class
• properly detect primary interface among facts, never use ipaddress and
  macaddress facts
• open question: what is primary interface, the one used for provisioning?

(2) Configuration

• currently we configure it by generating config files in provisioning templates
• open questions:
  • should we try to configure it using puppet - I see several issues
  • should we use some library (aka, nominate you favorite one)
  • if any of previous question is yes, how and when do we get it to host

(3) User interface

• display all networking information in host detail so we don't have to edit
• improve host edit form (modal window? advanced link?)

(4) Should all interfaces be modeled in Foreman?

• we may want to manage vlan interfaces but not necessarily manage the
  physical to which they are attached, same with bridges and bonds
• open question: should we validate all linked devices or allow user to
  specify e.g. bridge containing interfaces that do not exist in Foreman?

(5) Connect the form with interfaces from compute resource, so we can map them
(MAC address is not known at this moment)

(6) Support for bridge devices

• importing more information, like bridged devices
• configure it

(7) Support for bond devices

• importing
• configure it
• open question: what modes should we support?

(8) Support for VPN devices (tun/tap)

• importing
• open question: configure it?

(9) Publishing networking information in ENC output

• namespace it with a configurable string so users can avoid potential clashes
  with other networking facts

(10) Custom facts

• we support parsing of custom facts for IPMI devices and link status in
  discovery, how do propagate such facts to normal puppet agents?
• this is needed for fixing vlan and aliases fact conflicts until they fix it in
  upstream [2]

(11) Other

• should we allow boot mode to be overridden on interface level?
• fix alias vs. vlan detection (see custom facts)
• suggest new IP for interface link, similar to what we have for host
• should we delete interfaces that are not present in facts? it may be
  reasonable for physical devices, probably not for virtuals as your vlan
  interfaces can be down for a while
• improve UI so user see more information about network he's assigning (e.g.
  vlanid, whether it's tagged and such)

(12) Debian support

• currently we have provisioning templates for both, but wheezy installer
  overrides configuration before reboot, so we need to find some workaround if we
  want to use provisioning templates for configuration

I made a copy of this list on public pad [3]. Is there anything else not
covered by these tasks?

[1] https://www.youtube.com/watch?v=yCCViaCaQRo
[2] https://tickets.puppetlabs.com/browse/FACT-701
[3] https://titanpad.com/urGYwL9cDf

Thanks

···

-- Marek

[Claer]

> Hi all
Hello,

> as I promised during sprint demo 28 [1] in which I showed recent changes to
> networking I'd like to start discussion over items that we've identified as
> important regarding further networking stuff.
>
> The goal is to find out which features are most important (feel free to +1) and
> to find out how they should be implemented. There are many open questions. If
> this is too complex for email discussion we could do some deep dive and
> discuss each task separately. When we are clear on items I'll clean up redmine
> and convert tasks to new issues.
>
> Here's the list of items:
>
> (1) Refactoring of existing code
> * extract primary interface from host
> * rework static flag in provisioning
> * allow searching of hosts based on mac address (including all interfaces)
> * rename Nic::Managed or merge with Nic::Interface class
> * properly detect primary interface among facts, never use ipaddress and
> macaddress facts
> * open question: what is primary interface, the one used for provisioning?

It is often the interface that has the default route

> (2) Configuration
> * currently we configure it by generating config files in provisioning templates
> * open questions:
> - should we try to configure it using puppet - I see several issues
> - should we use some library (aka, nominate you favorite one)
> - if any of previous question is yes, how and when do we get it to host
I'd answer no to these questions. Last time I looked at (6months+ ago) there was
no puppet module for managing both redhad and debian interfaces in a clean way.

> (3) User interface
> * display all networking information in host detail so we don't have to edit
++1
> * improve host edit form (modal window? advanced link?)
+1

> (4) Should all interfaces be modeled in Foreman?
> * we may want to manage vlan interfaces but not necessarily manage the
> physical to which they are attached, same with bridges and bonds
> * open question: should we validate all linked devices or allow user to
> specify e.g. bridge containing interfaces that do not exist in Foreman?
I don't think so. Bonding prod, backup and admin are the main use I had for
several NICs on a server. Bridges are for more advanced setups that can be done
after installation imo (they are useful for hypervisors for ex.)
Managing bonding and specific ip routes for additionnal NICs is what I see
really interesting at the installation phase. VLAN would be a plus but is not
often used for deployment.

> (5) Connect the form with interfaces from compute resource, so we can map them
> (MAC address is not known at this moment)
Downloading the networks list from the compute ressources and attach it to the
vm can be useful.

> (6) Support for bridge devices
> * importing more information, like bridged devices
> * configure it

> (7) Support for bond devices
> * importing
Interesting only if you can replay it at install phase
> * configure it
> * open question: what modes should we support?
Dunno, I saw mode 1 not working on some switches where mode 6 worked. No sane
default known to me.

> (8) Support for VPN devices (tun/tap)
> * importing
> * open question: configure it?
I wouldn't try to support network configuration that far, there is so much
difference between all OS.

> (9) Publishing networking information in ENC output
> * namespace it with a configurable string so users can avoid potential clashes
> with other networking facts
+1
It would definitively helps templating firewall rules.

> (10) Custom facts
> * we support parsing of custom facts for IPMI devices and link status in
> discovery, how do propagate such facts to normal puppet agents?
> * this is needed for fixing vlan and aliases fact conflicts until they fix it in
> upstream [2]
>
> (11) Other
> * should we allow boot mode to be overridden on interface level?
> * fix alias vs. vlan detection (see custom facts)
> * suggest new IP for interface link, similar to what we have for host
> * should we delete interfaces that are not present in facts? it may be
> reasonable for physical devices, probably not for virtuals as your vlan
> interfaces can be down for a while
> * improve UI so user see more information about network he's assigning (e.g.
> vlanid, whether it's tagged and such)
>
> (12) Debian support
> * currently we have provisioning templates for both, but wheezy installer
> overrides configuration before reboot, so we need to find some workaround if we
> want to use provisioning templates for configuration
+1

···

On Tue, Sep 09 2014 at 48:16, Marek Hulan wrote:

I made a copy of this list on public pad [3]. Is there anything else not
covered by these tasks?

[1] https://www.youtube.com/watch?v=yCCViaCaQRo
[2] https://tickets.puppetlabs.com/browse/FACT-701
[3] https://titanpad.com/urGYwL9cDf

Thanks

–
Marek

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gwmngilfen]

>
> Here's the list of items:
>
> (1) Refactoring of existing code
> * extract primary interface from host
> * rework static flag in provisioning
> * allow searching of hosts based on mac address (including all interfaces)
> * rename Nic::Managed or merge with Nic::Interface class
> * properly detect primary interface among facts, never use ipaddress and
> macaddress facts
> * open question: what is primary interface, the one used for provisioning?

+1 to this. The two interface flags we need are:

Primary - this gives the host it's identity, and the DNS record for
this interface is how the host is referred to / displayed in the UI.
There must always be a primary, so that a host has a name (except for
unamanged, wher the host.name is the fqdn as usual)

Provisioning - the interface used for writing tftp files based it's
mac, and configured as the primary during kickstart so it can reach
Foreman. Defaults to the Primary interface above if not set.

This is because we often hear of people having separate provisioning
networks, with hosts moved to the correct vlans/subnets after build.
As such, you want to see "foo.example.com" in Foreman, not
"foo.provisioning.example.com".

> (2) Configuration
> * currently we configure it by generating config files in provisioning templates
> * open questions:
> - should we try to configure it using puppet - I see several issues
> - should we use some library (aka, nominate you favorite one)
> - if any of previous question is yes, how and when do we get it to host

I don't see how you'd get a library like netcf built into Anaconda,
and once the host has rebooted, we no longer have a way interact with
it - other than Puppet, but we should not be dictating which Puppet
modules the user must install.

I don't see any solution apart from doing it in Kictstart/Preseed.
Anaconda/D-I should be configured to use the provisioning interface
for install, so that the host can retrieve templates from Foreman. The
Finish / %post should then correctly lay down the remaining network
configuration for reboot. We could potentially install some tool and
use it in the %post to ensure we generate the network files correctly,
that will need testing

> (3) User interface
> * display all networking information in host detail so we don't have to edit
> * improve host edit form (modal window? advanced link?)

+1 - I like what Kyle was suggesting for the OS page a few weeks back
(regarding creating objects of one type directly in the edit page of
another). Perhaps we could use this idea to create network objects on
the fly, and then display a more condensed version as each is saved,
to reduce screen clutter?

> (4) Should all interfaces be modeled in Foreman?
> * we may want to manage vlan interfaces but not necessarily manage the
> physical to which they are attached, same with bridges and bonds
> * open question: should we validate all linked devices or allow user to
> specify e.g. bridge containing interfaces that do not exist in Foreman?

I think we're being quite flexible, and can add interface types as the
community starts to play with this. It's not as high on the priority
as the things above.

> (5) Connect the form with interfaces from compute resource, so we can map them
> (MAC address is not known at this moment)

Nice-to-have, but we've survived with the duplication between tabs this far

> (6) Support for bridge devices
> * importing more information, like bridged devices
> * configure it

See (4)

> (7) Support for bond devices
> * importing
> * configure it
> * open question: what modes should we support?

See (4)

> (8) Support for VPN devices (tun/tap)
> * importing
> * open question: configure it?

See (4)

> (9) Publishing networking information in ENC output
> * namespace it with a configurable string so users can avoid potential clashes
> with other networking facts

+1 this will be very helpful I suspect. We should also add some
helpers to that existing puppet classes can be fed with data in a
better form that <%= @host.interfaces.first.subnet.gateway %>

> (10) Custom facts
> * we support parsing of custom facts for IPMI devices and link status in
> discovery, how do propagate such facts to normal puppet agents?
> * this is needed for fixing vlan and aliases fact conflicts until they fix it in
> upstream [2]

We should not be requiring custom facts, in my opinion. We currently
don't mandate that users must install any puppet code at all. Like
the foreman api function, we could add custom facts to our foreman
module, and people can use it if they like, but our code needs to work
with vanilla Facter.

> (12) Debian support
> * currently we have provisioning templates for both, but wheezy installer
> overrides configuration before reboot, so we need to find some workaround if we
> want to use provisioning templates for configuration

There's a few possibilities here, I can look into it.

Greg

···

On 9 September 2014 15:48, Marek Hulan wrote:

[iNecas]

> > Hi all
> Hello,
>
> > as I promised during sprint demo 28 [1] in which I showed recent changes to
> > networking I'd like to start discussion over items that we've identified as
> > important regarding further networking stuff.
> >
> > The goal is to find out which features are most important (feel free to +1)
> > and
> > to find out how they should be implemented. There are many open questions.
> > If
> > this is too complex for email discussion we could do some deep dive and
> > discuss each task separately. When we are clear on items I'll clean up
> > redmine
> > and convert tasks to new issues.
> >
> > Here's the list of items:
> >
> > (1) Refactoring of existing code
> > * extract primary interface from host
> > * rework static flag in provisioning
> > * allow searching of hosts based on mac address (including all interfaces)
> > * rename Nic::Managed or merge with Nic::Interface class
> > * properly detect primary interface among facts, never use ipaddress and
> > macaddress facts
> > * open question: what is primary interface, the one used for provisioning?
>

+1 - since we are touching it already, let's also make it better

> It is often the interface that has the default route
>
> > (2) Configuration
> > * currently we configure it by generating config files in provisioning
> > templates
> > * open questions:
> > - should we try to configure it using puppet - I see several issues
> > - should we use some library (aka, nominate you favorite one)
> > - if any of previous question is yes, how and when do we get it to host
> I'd answer no to these questions. Last time I looked at (6months+ ago) there
> was
> no puppet module for managing both redhad and debian interfaces in a clean
> way.

Also, using the puppet has the issues that you change the facter values
(such as ipaddress_eth0) while converging, so one would need separate
that from the actual puppet run for the hostgroup it self (sounds quite
complicated to me).

Dominic: what was the library you suggested at the demo?

Anyway, the possibility to fallback to plain old config files should still
be there if needed.

>
> > (3) User interface
> > * display all networking information in host detail so we don't have to
> > edit
> ++1

+1
That doesn't apply only for the networking, but in general: it's quite inconvenient
to need to go to edit mode to find some details: getting as much as possible
to show page would be big +1 (and loading the comute-resource information asynchronously:
as sometimes, it can take quite long time to load the data from compute resource)

> > * improve host edit form (modal window? advanced link?)
> +1
>
> > (4) Should all interfaces be modeled in Foreman?
> > * we may want to manage vlan interfaces but not necessarily manage the
> > physical to which they are attached, same with bridges and bonds
> > * open question: should we validate all linked devices or allow user to
> > specify e.g. bridge containing interfaces that do not exist in Foreman?
> I don't think so. Bonding prod, backup and admin are the main use I had for
> several NICs on a server. Bridges are for more advanced setups that can be
> done
> after installation imo (they are useful for hypervisors for ex.)
> Managing bonding and specific ip routes for additionnal NICs is what I see
> really interesting at the installation phase. VLAN would be a plus but is not
> often used for deployment.

IMO there should be possibility to record the information about unmanaged
devices, just for filtering purposes. OTOH validation might be tricky,
there might be a possibilty to select existing interfaces in Foreman, but still,
there should be possibility of free form string.

>
> > (5) Connect the form with interfaces from compute resource, so we can map
> > them
> > (MAC address is not known at this moment)
> Downloading the networks list from the compute ressources and attach it to
> the
> vm can be useful.

Could you elaborate a bit more on this?

>
> > (6) Support for bridge devices
> > * importing more information, like bridged devices
> > * configure it

+1 - useful for provisioning hypervisors

>
> > (7) Support for bond devices
> > * importing
> Interesting only if you can replay it at install phase
> > * configure it
> > * open question: what modes should we support?
> Dunno, I saw mode 1 not working on some switches where mode 6 worked. No sane
> default known to me.
>
> > (8) Support for VPN devices (tun/tap)
> > * importing
> > * open question: configure it?
> I wouldn't try to support network configuration that far, there is so much
> difference between all OS.
>
> > (9) Publishing networking information in ENC output
> > * namespace it with a configurable string so users can avoid potential
> > clashes
> > with other networking facts
> +1
> It would definitively helps templating firewall rules.
>
> > (10) Custom facts
> > * we support parsing of custom facts for IPMI devices and link status in
> > discovery, how do propagate such facts to normal puppet agents?
> > * this is needed for fixing vlan and aliases fact conflicts until they fix
> > it in
> > upstream [2]
> >
> > (11) Other
> > * should we allow boot mode to be overridden on interface level?
> > * fix alias vs. vlan detection (see custom facts)
> > * suggest new IP for interface link, similar to what we have for host
> > * should we delete interfaces that are not present in facts? it may be
> > reasonable for physical devices, probably not for virtuals as your vlan
> > interfaces can be down for a while

There should be definitely a mode, that doesn't delete the interfaces
based on facts: maybe some "cleanup" action might be useful, but I generally
don't like the idea of too close coupling between facter facts and the other
metadata (especially as the facter has sometimes quite surprising values)

···

----- Original Message ----- > On Tue, Sep 09 2014 at 48:16, Marek Hulan wrote:

• improve UI so user see more information about network he’s assigning
  (e.g.
  vlanid, whether it’s tagged and such)

(12) Debian support

• currently we have provisioning templates for both, but wheezy installer
  overrides configuration before reboot, so we need to find some workaround
  if we
  want to use provisioning templates for configuration
  +1

I made a copy of this list on public pad [3]. Is there anything else not
covered by these tasks?

[1] https://www.youtube.com/watch?v=yCCViaCaQRo
[2] https://tickets.puppetlabs.com/browse/FACT-701
[3] https://titanpad.com/urGYwL9cDf

Thanks

–
Marek

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ori_Rabin]

> (3) User interface
> * display all networking information in host detail so we don't have to
edit
> * improve host edit form (modal window? advanced link?)

It was mentioned during the demo that there is a PR for host details which
is a readonly version of edit:
https://github.com/theforeman/foreman/pull/1743

···

On Wed, Sep 10, 2014 at 3:04 PM, Greg Sutcliffe wrote:

On 9 September 2014 15:48, Marek Hulan mhulan@redhat.com wrote:

Here’s the list of items:

(1) Refactoring of existing code

• extract primary interface from host
• rework static flag in provisioning
• allow searching of hosts based on mac address (including all
  interfaces)
• rename Nic::Managed or merge with Nic::Interface class
• properly detect primary interface among facts, never use ipaddress and
  macaddress facts
• open question: what is primary interface, the one used for
  provisioning?

+1 to this. The two interface flags we need are:

Primary - this gives the host it’s identity, and the DNS record for
this interface is how the host is referred to / displayed in the UI.
There must always be a primary, so that a host has a name (except for
unamanged, wher the host.name is the fqdn as usual)

Provisioning - the interface used for writing tftp files based it’s
mac, and configured as the primary during kickstart so it can reach
Foreman. Defaults to the Primary interface above if not set.

This is because we often hear of people having separate provisioning
networks, with hosts moved to the correct vlans/subnets after build.
As such, you want to see “foo.example.com” in Foreman, not
“foo.provisioning.example.com”.

(2) Configuration

• currently we configure it by generating config files in provisioning
  templates
• open questions:
  • should we try to configure it using puppet - I see several issues
  • should we use some library (aka, nominate you favorite one)
  • if any of previous question is yes, how and when do we get it to host

I don’t see how you’d get a library like netcf built into Anaconda,
and once the host has rebooted, we no longer have a way interact with
it - other than Puppet, but we should not be dictating which Puppet
modules the user must install.

I don’t see any solution apart from doing it in Kictstart/Preseed.
Anaconda/D-I should be configured to use the provisioning interface
for install, so that the host can retrieve templates from Foreman. The
Finish / %post should then correctly lay down the remaining network
configuration for reboot. We could potentially install some tool and
use it in the %post to ensure we generate the network files correctly,
that will need testing

(3) User interface

• display all networking information in host detail so we don’t have to
  edit
• improve host edit form (modal window? advanced link?)

+1 - I like what Kyle was suggesting for the OS page a few weeks back
(regarding creating objects of one type directly in the edit page of
another). Perhaps we could use this idea to create network objects on
the fly, and then display a more condensed version as each is saved,
to reduce screen clutter?

(4) Should all interfaces be modeled in Foreman?

• we may want to manage vlan interfaces but not necessarily manage the
  physical to which they are attached, same with bridges and bonds
• open question: should we validate all linked devices or allow user to
  specify e.g. bridge containing interfaces that do not exist in Foreman?

I think we’re being quite flexible, and can add interface types as the
community starts to play with this. It’s not as high on the priority
as the things above.

(5) Connect the form with interfaces from compute resource, so we can
map them
(MAC address is not known at this moment)

Nice-to-have, but we’ve survived with the duplication between tabs this
far

(6) Support for bridge devices

• importing more information, like bridged devices
• configure it

See (4)

(7) Support for bond devices

• importing
• configure it
• open question: what modes should we support?

See (4)

(8) Support for VPN devices (tun/tap)

• importing
• open question: configure it?

See (4)

(9) Publishing networking information in ENC output

• namespace it with a configurable string so users can avoid potential
  clashes
  with other networking facts

+1 this will be very helpful I suspect. We should also add some
helpers to that existing puppet classes can be fed with data in a
better form that <%= @host.interfaces.first.subnet.gateway %>

(10) Custom facts

• we support parsing of custom facts for IPMI devices and link status in
  discovery, how do propagate such facts to normal puppet agents?
• this is needed for fixing vlan and aliases fact conflicts until they
  fix it in
  upstream [2]

We should not be requiring custom facts, in my opinion. We currently
don’t mandate that users must install any puppet code at all. Like
the foreman api function, we could add custom facts to our foreman
module, and people can use it if they like, but our code needs to work
with vanilla Facter.

(12) Debian support

• currently we have provisioning templates for both, but wheezy installer
  overrides configuration before reboot, so we need to find some
  workaround if we
  want to use provisioning templates for configuration

There’s a few possibilities here, I can look into it.

Greg

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marek_Hulan]

> > Here's the list of items:
> >
> > (1) Refactoring of existing code
> > * extract primary interface from host
> > * rework static flag in provisioning
> > * allow searching of hosts based on mac address (including all interfaces)
> > * rename Nic::Managed or merge with Nic::Interface class
> > * properly detect primary interface among facts, never use ipaddress and
> > macaddress facts
> > * open question: what is primary interface, the one used for provisioning?
>
> +1 to this. The two interface flags we need are:
>
> Primary - this gives the host it's identity, and the DNS record for
> this interface is how the host is referred to / displayed in the UI.
> There must always be a primary, so that a host has a name (except for
> unamanged, wher the host.name is the fqdn as usual)
>
> Provisioning - the interface used for writing tftp files based it's
> mac, and configured as the primary during kickstart so it can reach
> Foreman. Defaults to the Primary interface above if not set.
>
> This is because we often hear of people having separate provisioning
> networks, with hosts moved to the correct vlans/subnets after build.
> As such, you want to see "foo.example.com" in Foreman, not
> "foo.provisioning.example.com".

This sounds reasonable to me. Few ideas

• primary interface should always be managed (flag added recently)
• same applies to provisioning interface
• we can easily support moving of provisioning flag between interfaces
• we don't want to enable primary interface flag to be moved (at least for now)
• while flags on interface seems as simple solution for now, we should think
  about subnet roles so by assigning a subnet with role "provisioning", we'd
  know the default value for provisioning flag, this brings several things to
  consider, like multiple provisioning subnets

>
> > (2) Configuration
> > * currently we configure it by generating config files in provisioning
> > >
> > * open questions:
> > - should we try to configure it using puppet - I see several issues
> > - should we use some library (aka, nominate you favorite one)
> > - if any of previous question is yes, how and when do we get it to host
>
> I don't see how you'd get a library like netcf built into Anaconda,
> and once the host has rebooted, we no longer have a way interact with
> it - other than Puppet, but we should not be dictating which Puppet
> modules the user must install.

> I don't see any solution apart from doing it in Kictstart/Preseed.
> Anaconda/D-I should be configured to use the provisioning interface
> for install, so that the host can retrieve templates from Foreman. The
> Finish / %post should then correctly lay down the remaining network
> configuration for reboot. We could potentially install some tool and
> use it in the %post to ensure we generate the network files correctly,
> that will need testing

Basically I agree, we could install it by specifying it in %packages or d-i
pkgsel/include in case of preseed and use it in %post. Even if you specify
interface configuration to Anaconda it won't bring up interfaces other than it
booted from, it will just create config scripts and bring them up after the
installation. Not sure about debian installer. Also I think we won't be able
to fully configure advanced things like bond/bridges (even though Anaconda has
some support for this already). But I thought we could reuse netcf in %post so
we don't generate config files manually.

> > (3) User interface
> > * display all networking information in host detail so we don't have to
> > edit * improve host edit form (modal window? advanced link?)
>
> +1 - I like what Kyle was suggesting for the OS page a few weeks back
> (regarding creating objects of one type directly in the edit page of
> another). Perhaps we could use this idea to create network objects on
> the fly, and then display a more condensed version as each is saved,
> to reduce screen clutter?
>
> > (4) Should all interfaces be modeled in Foreman?
> > * we may want to manage vlan interfaces but not necessarily manage the
> > physical to which they are attached, same with bridges and bonds
> > * open question: should we validate all linked devices or allow user to
> > specify e.g. bridge containing interfaces that do not exist in Foreman?
>
> I think we're being quite flexible, and can add interface types as the
> community starts to play with this. It's not as high on the priority
> as the things above.
>
> > (5) Connect the form with interfaces from compute resource, so we can map
> > them (MAC address is not known at this moment)
>
> Nice-to-have, but we've survived with the duplication between tabs this far
>

It's not just about duplication but we can't configure interfaces for virtual
hosts now. We don't know the MAC address of those interfaces yet therefore we
can't create the right configuration in network form thus we can't create config
files during provisioning. If we linked these two "interface" forms, we would
be manage it after compute resource allocates MAC addresses.

> > (6) Support for bridge devices
> > * importing more information, like bridged devices
> > * configure it
>
> See (4)
>
> > (7) Support for bond devices
> > * importing
> > * configure it
> > * open question: what modes should we support?
>
> See (4)
>
> > (8) Support for VPN devices (tun/tap)
> > * importing
> > * open question: configure it?
>
> See (4)
>
> > (9) Publishing networking information in ENC output
> > * namespace it with a configurable string so users can avoid potential
> > clashes with other networking facts
>
> +1 this will be very helpful I suspect. We should also add some
> helpers to that existing puppet classes can be fed with data in a
> better form that <%= @host.interfaces.first.subnet.gateway %>
>
> > (10) Custom facts
> > * we support parsing of custom facts for IPMI devices and link status in
> > discovery, how do propagate such facts to normal puppet agents?
> > * this is needed for fixing vlan and aliases fact conflicts until they fix
> > it in upstream [2]
>
> We should not be requiring custom facts, in my opinion. We currently
> don't mandate that users must install any puppet code at all. Like
> the foreman api function, we could add custom facts to our foreman
> module, and people can use it if they like, but our code needs to work
> with vanilla Facter.

I don't say we should require it. But without some custom facts we can't
support alias and vlans reliably (at least with puppet). So I agree with some
optional but reasonable easy to use solution. By foreman module, do you mean
puppet module? Or creating a new one? So users would have to assign to all
host groups manually?

> > (12) Debian support
> > * currently we have provisioning templates for both, but wheezy installer
> > overrides configuration before reboot, so we need to find some workaround
> > if we want to use provisioning templates for configuration
>
> There's a few possibilities here, I can look into it.
>
> Greg

Thanks for comments!

···

On Wednesday 10 of September 2014 13:04:31 Greg Sutcliffe wrote: > On 9 September 2014 15:48, Marek Hulan wrote:

–
Marek

[Marek_Hulan]

Thanks for comments so far, my replies below in text

>
> > > Hi all
> >
> > Hello,
> >
> > > as I promised during sprint demo 28 [1] in which I showed recent changes
> > > to
> > > networking I'd like to start discussion over items that we've identified
> > > as
> > > important regarding further networking stuff.
> > >
> > > The goal is to find out which features are most important (feel free to
> > > +1)
> > > and
> > > to find out how they should be implemented. There are many open
> > > questions.
> > > If
> > > this is too complex for email discussion we could do some deep dive and
> > > discuss each task separately. When we are clear on items I'll clean up
> > > redmine
> > > and convert tasks to new issues.
> > >
> > > Here's the list of items:
> > >
> > > (1) Refactoring of existing code
> > > * extract primary interface from host
> > > * rework static flag in provisioning
> > > * allow searching of hosts based on mac address (including all
> > > interfaces)
> > > * rename Nic::Managed or merge with Nic::Interface class
> > > * properly detect primary interface among facts, never use ipaddress and
> > > macaddress facts
> > > * open question: what is primary interface, the one used for
> > > provisioning?
>
> +1 - since we are touching it already, let's also make it better
>
> > It is often the interface that has the default route

The primary interface is kind of a relative term. If we say that it's the
interface that has the default route, what happens when we have two interfaces
each with own default gateway?

> >
> > > (2) Configuration
> > > * currently we configure it by generating config files in provisioning
> > > templates
> > >
> > > * open questions:
> > > - should we try to configure it using puppet - I see several issues
> > > - should we use some library (aka, nominate you favorite one)
> > > - if any of previous question is yes, how and when do we get it to
> > > host
> >
> > I'd answer no to these questions. Last time I looked at (6months+ ago)
> > there was
> > no puppet module for managing both redhad and debian interfaces in a clean
> > way.
>
> Also, using the puppet has the issues that you change the facter values
> (such as ipaddress_eth0) while converging, so one would need separate
> that from the actual puppet run for the hostgroup it self (sounds quite
> complicated to me).

Thanks for mentioning this, my feeling is that puppet is not the right tool to
manage network configuration because of it can't refresh facts. Maybe we could
add some script using $library that would configure networking just before
puppet run. One benefit is that it would be cfgmgmt agnostic.

>
> Dominic: what was the library you suggested at the demo?

sorry for answering for you
I believe it was netcf - https://fedorahosted.org/netcf/

> Anyway, the possibility to fallback to plain old config files should still
> be there if needed.
>
> > > (3) User interface
> > > * display all networking information in host detail so we don't have to
> > > edit
> >
> > ++1
>
> +1
> That doesn't apply only for the networking, but in general: it's quite
> inconvenient to need to go to edit mode to find some details: getting as
> much as possible to show page would be big +1 (and loading the
> comute-resource information asynchronously: as sometimes, it can take quite
> long time to load the data from compute resource)
> > > * improve host edit form (modal window? advanced link?)
> >
> > +1
> >
> > > (4) Should all interfaces be modeled in Foreman?
> > > * we may want to manage vlan interfaces but not necessarily manage the
> > > physical to which they are attached, same with bridges and bonds
> > > * open question: should we validate all linked devices or allow user to
> > > specify e.g. bridge containing interfaces that do not exist in Foreman?
> >
> > I don't think so. Bonding prod, backup and admin are the main use I had
> > for
> > several NICs on a server. Bridges are for more advanced setups that can be
> > done
> > after installation imo (they are useful for hypervisors for ex.)
> > Managing bonding and specific ip routes for additionnal NICs is what I see
> > really interesting at the installation phase. VLAN would be a plus but is
> > not often used for deployment.
>
> IMO there should be possibility to record the information about unmanaged
> devices, just for filtering purposes. OTOH validation might be tricky,
> there might be a possibilty to select existing interfaces in Foreman, but
> still, there should be possibility of free form string.

If we allow to use identifiers of interfaces not present in Foreman I think we
shouldn't validate identifier at all (except it's present, maybe some regexp),
Tricky would be possible usage of such association, we'll have to wrap
everything in if conditions.

> > > (5) Connect the form with interfaces from compute resource, so we can
> > > map
> > > them
> > > (MAC address is not known at this moment)
> >
> > Downloading the networks list from the compute ressources and attach it to
> > the
> > vm can be useful.
>
> Could you elaborate a bit more on this?
>
> > > (6) Support for bridge devices
> > > * importing more information, like bridged devices
> > > * configure it
>
> +1 - useful for provisioning hypervisors
>
> > > (7) Support for bond devices
> > > * importing
> >
> > Interesting only if you can replay it at install phase
> >
> > > * configure it
> > > * open question: what modes should we support?
> >
> > Dunno, I saw mode 1 not working on some switches where mode 6 worked. No
> > sane default known to me.
> >
> > > (8) Support for VPN devices (tun/tap)
> > > * importing
> > > * open question: configure it?
> >
> > I wouldn't try to support network configuration that far, there is so much
> > difference between all OS.

nods

> >
> > > (9) Publishing networking information in ENC output
> > > * namespace it with a configurable string so users can avoid potential
> > > clashes
> > > with other networking facts
> >
> > +1
> > It would definitively helps templating firewall rules.
> >
> > > (10) Custom facts
> > > * we support parsing of custom facts for IPMI devices and link status in
> > > discovery, how do propagate such facts to normal puppet agents?
> > > * this is needed for fixing vlan and aliases fact conflicts until they
> > > fix
> > > it in
> > > upstream [2]
> > >
> > > (11) Other
> > > * should we allow boot mode to be overridden on interface level?
> > > * fix alias vs. vlan detection (see custom facts)
> > > * suggest new IP for interface link, similar to what we have for host
> > > * should we delete interfaces that are not present in facts? it may be
> > > reasonable for physical devices, probably not for virtuals as your vlan
> > > interfaces can be down for a while
>
> There should be definitely a mode, that doesn't delete the interfaces
> based on facts: maybe some "cleanup" action might be useful, but I generally
> don't like the idea of too close coupling between facter facts and the
> other metadata (especially as the facter has sometimes quite surprising
> values)

so same settings that enables/disables auto creation and update of attributes?
Or rather separate option so you get new interfaces but they are not cleaned.

···

> ----- Original Message ----- > > On Tue, Sep 09 2014 at 48:16, Marek Hulan wrote:

• improve UI so user see more information about network he’s assigning
  (e.g.
  vlanid, whether it’s tagged and such)

(12) Debian support

• currently we have provisioning templates for both, but wheezy
  installer
  overrides configuration before reboot, so we need to find some
  workaround
  if we
  want to use provisioning templates for configuration

+1

I made a copy of this list on public pad [3]. Is there anything else not
covered by these tasks?

[1] https://www.youtube.com/watch?v=yCCViaCaQRo
[2] https://tickets.puppetlabs.com/browse/FACT-701
[3] https://titanpad.com/urGYwL9cDf

Thanks

–
Marek

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Marek

[Gwmngilfen]

> This sounds reasonable to me. Few ideas
> - primary interface should always be managed (flag added recently)
> - same applies to provisioning interface
> - we can easily support moving of provisioning flag between interfaces
> - we don't want to enable primary interface flag to be moved (at least for now)
> - while flags on interface seems as simple solution for now, we should think
> about subnet roles so by assigning a subnet with role "provisioning", we'd
> know the default value for provisioning flag, this brings several things to
> consider, like multiple provisioning subnets

Mostly agree - although we can edit a host today and change it's
network details, so I don't agree with not being able to move the
primary interface. From a complexity viewpoint, I agree it's further
down the refactoring list though

>> We should not be requiring custom facts, in my opinion. We currently
>> don't mandate that users must install any puppet code at all. Like
>> the foreman api function, we could add custom facts to our foreman
>> module, and people can use it if they like, but our code needs to work
>> with vanilla Facter.
>
> I don't say we should require it. But without some custom facts we can't
> support alias and vlans reliably (at least with puppet). So I agree with some
> optional but reasonable easy to use solution. By foreman module, do you mean
> puppet module? Or creating a new one? So users would have to assign to all
> host groups manually?

I'm talking about the function at [1] - we could add more things to
this, or as you say, we could create a separate module
(foreman_functions?) which contain all of this stuff, but contains no
classes (thuse preventing namespace conflict or dependency clashes).
As discussed though, it must be optional.

Greg
[1] https://github.com/theforeman/puppet-foreman/blob/master/lib/puppet/parser/functions/foreman.rb

···

On 12 September 2014 09:44, Marek Hulan wrote:

[iNecas]

> Thanks for comments so far, my replies below in text
>
> >
> > > > Hi all
> > >
> > > Hello,
> > >
> > > > as I promised during sprint demo 28 [1] in which I showed recent
> > > > changes
> > > > to
> > > > networking I'd like to start discussion over items that we've
> > > > identified
> > > > as
> > > > important regarding further networking stuff.
> > > >
> > > > The goal is to find out which features are most important (feel free to
> > > > +1)
> > > > and
> > > > to find out how they should be implemented. There are many open
> > > > questions.
> > > > If
> > > > this is too complex for email discussion we could do some deep dive and
> > > > discuss each task separately. When we are clear on items I'll clean up
> > > > redmine
> > > > and convert tasks to new issues.
> > > >
> > > > Here's the list of items:
> > > >
> > > > (1) Refactoring of existing code
> > > > * extract primary interface from host
> > > > * rework static flag in provisioning
> > > > * allow searching of hosts based on mac address (including all
> > > > interfaces)
> > > > * rename Nic::Managed or merge with Nic::Interface class
> > > > * properly detect primary interface among facts, never use ipaddress
> > > > and
> > > > macaddress facts
> > > > * open question: what is primary interface, the one used for
> > > > provisioning?
> >
> > +1 - since we are touching it already, let's also make it better
> >
> > > It is often the interface that has the default route
>
> The primary interface is kind of a relative term. If we say that it's the
> interface that has the default route, what happens when we have two
> interfaces
> each with own default gateway?
>
> > >
> > > > (2) Configuration
> > > > * currently we configure it by generating config files in provisioning
> > > > templates
> > > >
> > > > * open questions:
> > > > - should we try to configure it using puppet - I see several issues
> > > > - should we use some library (aka, nominate you favorite one)
> > > > - if any of previous question is yes, how and when do we get it to
> > > > host
> > >
> > > I'd answer no to these questions. Last time I looked at (6months+ ago)
> > > there was
> > > no puppet module for managing both redhad and debian interfaces in a
> > > clean
> > > way.
> >
> > Also, using the puppet has the issues that you change the facter values
> > (such as ipaddress_eth0) while converging, so one would need separate
> > that from the actual puppet run for the hostgroup it self (sounds quite
> > complicated to me).
>
> Thanks for mentioning this, my feeling is that puppet is not the right tool
> to
> manage network configuration because of it can't refresh facts. Maybe we
> could
> add some script using $library that would configure networking just before
> puppet run. One benefit is that it would be cfgmgmt agnostic.
>
> >
> > Dominic: what was the library you suggested at the demo?
>
> sorry for answering for you
> I believe it was netcf - https://fedorahosted.org/netcf/
>
> > Anyway, the possibility to fallback to plain old config files should still
> > be there if needed.
> >
> > > > (3) User interface
> > > > * display all networking information in host detail so we don't have to
> > > > edit
> > >
> > > ++1
> >
> > +1
> > That doesn't apply only for the networking, but in general: it's quite
> > inconvenient to need to go to edit mode to find some details: getting as
> > much as possible to show page would be big +1 (and loading the
> > comute-resource information asynchronously: as sometimes, it can take quite
> > long time to load the data from compute resource)
> > > > * improve host edit form (modal window? advanced link?)
> > >
> > > +1
> > >
> > > > (4) Should all interfaces be modeled in Foreman?
> > > > * we may want to manage vlan interfaces but not necessarily manage the
> > > > physical to which they are attached, same with bridges and bonds
> > > > * open question: should we validate all linked devices or allow user to
> > > > specify e.g. bridge containing interfaces that do not exist in Foreman?
> > >
> > > I don't think so. Bonding prod, backup and admin are the main use I had
> > > for
> > > several NICs on a server. Bridges are for more advanced setups that can
> > > be
> > > done
> > > after installation imo (they are useful for hypervisors for ex.)
> > > Managing bonding and specific ip routes for additionnal NICs is what I
> > > see
> > > really interesting at the installation phase. VLAN would be a plus but is
> > > not often used for deployment.
> >
> > IMO there should be possibility to record the information about unmanaged
> > devices, just for filtering purposes. OTOH validation might be tricky,
> > there might be a possibilty to select existing interfaces in Foreman, but
> > still, there should be possibility of free form string.
>
> If we allow to use identifiers of interfaces not present in Foreman I think
> we
> shouldn't validate identifier at all (except it's present, maybe some
> regexp),
> Tricky would be possible usage of such association, we'll have to wrap
> everything in if conditions.
>
> > > > (5) Connect the form with interfaces from compute resource, so we can
> > > > map
> > > > them
> > > > (MAC address is not known at this moment)
> > >
> > > Downloading the networks list from the compute ressources and attach it
> > > to
> > > the
> > > vm can be useful.
> >
> > Could you elaborate a bit more on this?
> >
> > > > (6) Support for bridge devices
> > > > * importing more information, like bridged devices
> > > > * configure it
> >
> > +1 - useful for provisioning hypervisors
> >
> > > > (7) Support for bond devices
> > > > * importing
> > >
> > > Interesting only if you can replay it at install phase
> > >
> > > > * configure it
> > > > * open question: what modes should we support?
> > >
> > > Dunno, I saw mode 1 not working on some switches where mode 6 worked. No
> > > sane default known to me.
> > >
> > > > (8) Support for VPN devices (tun/tap)
> > > > * importing
> > > > * open question: configure it?
> > >
> > > I wouldn't try to support network configuration that far, there is so
> > > much
> > > difference between all OS.
>
> nods
>
> > >
> > > > (9) Publishing networking information in ENC output
> > > > * namespace it with a configurable string so users can avoid potential
> > > > clashes
> > > > with other networking facts
> > >
> > > +1
> > > It would definitively helps templating firewall rules.
> > >
> > > > (10) Custom facts
> > > > * we support parsing of custom facts for IPMI devices and link status
> > > > in
> > > > discovery, how do propagate such facts to normal puppet agents?
> > > > * this is needed for fixing vlan and aliases fact conflicts until they
> > > > fix
> > > > it in
> > > > upstream [2]
> > > >
> > > > (11) Other
> > > > * should we allow boot mode to be overridden on interface level?
> > > > * fix alias vs. vlan detection (see custom facts)
> > > > * suggest new IP for interface link, similar to what we have for host
> > > > * should we delete interfaces that are not present in facts? it may be
> > > > reasonable for physical devices, probably not for virtuals as your vlan
> > > > interfaces can be down for a while
> >
> > There should be definitely a mode, that doesn't delete the interfaces
> > based on facts: maybe some "cleanup" action might be useful, but I
> > generally
> > don't like the idea of too close coupling between facter facts and the
> > other metadata (especially as the facter has sometimes quite surprising
> > values)
>
> so same settings that enables/disables auto creation and update of
> attributes?
> Or rather separate option so you get new interfaces but they are not cleaned.

When I create the interfaces in foreman manually, I would hate to get
them deleted once the facter calls in. If the interfaces get from facter
I don't care that much (and might want to use something that would prevent
the syncing if needed). Having "facter_managed" flag on every interface
that would determine this behavior would probably make sense to me.

···

----- Original Message ----- > > ----- Original Message ----- > > > On Tue, Sep 09 2014 at 48:16, Marek Hulan wrote:

• improve UI so user see more information about network he’s assigning
  (e.g.
  vlanid, whether it’s tagged and such)

(12) Debian support

• currently we have provisioning templates for both, but wheezy
  installer
  overrides configuration before reboot, so we need to find some
  workaround
  if we
  want to use provisioning templates for configuration

+1

I made a copy of this list on public pad [3]. Is there anything else
not
covered by these tasks?

[1] https://www.youtube.com/watch?v=yCCViaCaQRo
[2] https://tickets.puppetlabs.com/browse/FACT-701
[3] https://titanpad.com/urGYwL9cDf

Thanks

–
Marek

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Marek

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lzap]

> The primary interface is kind of a relative term. If we say that it's the
> interface that has the default route, what happens when we have two interfaces
> each with own default gateway?

Having multiple default gateways with same metrics, OS chooses randomly
I think. Technically, there is no such a thing like primary interface…

In real life, it is possible to make such a constraint that there is
only one but we should count with the edge case.

···

-- Later, Lukas #lzap Zapletal

[Marek_Hulan]

Hello,

thanks to everyone who commented. I cleaned up networking issues in redmine
and created new issues for all tasks we discussed here. There's one tracking
issue [1] that wraps them all. I upvoted those that were mostly discussed,
feel free to do the same. Also let's put any other comments or concerns to
specific tasks.

[1] Tracker #2409: Networking - Foreman

···

-- Marek

On Friday 12 of September 2014 17:38:17 Greg Sutcliffe wrote:

On 12 September 2014 09:44, Marek Hulan mhulan@redhat.com wrote:

This sounds reasonable to me. Few ideas

• primary interface should always be managed (flag added recently)
• same applies to provisioning interface
• we can easily support moving of provisioning flag between interfaces
• we don’t want to enable primary interface flag to be moved (at least for
  now) - while flags on interface seems as simple solution for now, we
  should think about subnet roles so by assigning a subnet with role
  "provisioning", we’d know the default value for provisioning flag, this
  brings several things to consider, like multiple provisioning subnets

Mostly agree - although we can edit a host today and change it’s
network details, so I don’t agree with not being able to move the
primary interface. From a complexity viewpoint, I agree it’s further
down the refactoring list though

We should not be requiring custom facts, in my opinion. We currently
don’t mandate that users must install any puppet code at all. Like
the foreman api function, we could add custom facts to our foreman
module, and people can use it if they like, but our code needs to work
with vanilla Facter.

I don’t say we should require it. But without some custom facts we can’t
support alias and vlans reliably (at least with puppet). So I agree with
some optional but reasonable easy to use solution. By foreman module, do
you mean puppet module? Or creating a new one? So users would have to
assign to all host groups manually?

I’m talking about the function at [1] - we could add more things to
this, or as you say, we could create a separate module
(foreman_functions?) which contain all of this stuff, but contains no
classes (thuse preventing namespace conflict or dependency clashes).
As discussed though, it must be optional.

Greg
[1]
https://github.com/theforeman/puppet-foreman/blob/master/lib/puppet/parser/
functions/foreman.rb