What is the future of Infoblox smartproxy plugins?

Hey everyone,

after a meeting with our Infoblox admin today about some new values they wanted to be set an hosts, I took a look at the current state of the smartproxy DHCP Infoblox plugin and realized it is not really in a good state right now.
First off, the plugin itself had it’s last commit in May 2020, which is not necessarily bad since it still works, but to me this is somewhat of a bad sign already.
Second and more concerning, the infoblox gem it relies on has also had it’s last commit during that timeframe and has since been archived by the author. The API version the gem defaults to (WAPI 2.0 if I understand the code correct) is somewhat old by now and is expected to be deprecated by Infoblox with the next major release.

So, a question to whomever feels somewhat responsible for this plugin: Do you have any plans for how to handle this situation?
Are there plans to actually continue the work on the plugin (if at all possible considering the corresponding gem is not updated anymore) or is the plugin on it’s sunset and we need to look for other solutions on how to integrate our Foreman and Infoblox?

Regards

3 Likes

Thanks for looking into this.

First of all, smart_proxy_dhcp_infoblox is under the Foreman umbrella. That means we can easily hand over maintenance to anyone interested.

I was not aware the gem we rely on is archived, which is concerning. Typically there are two options: fork it or try to take over maintenance. From the Foreman organization we can try to facilitate this if needed. We’ve done this before.

For now I made some people aware of this thread, but consider this thread an invitation for people to take up development. We can help with the Foreman (Proxy) side of things but not everyone has experience with the Infoblox side of things.

Foreman DHCP management capabilities is a clunky feature, it is what
it is, it dates back to the initial days when provisioning was added
by the community. The DHCP Smart-Proxy has a very simple API: get me all
IPs, get me IP by MAC, get me MAC by IP, add a new record (MAC+IP).
Foreman uses this to manage DHCP records on the ISC DHCP server that
ships with RHEL.

See, I don’t think Foreman should be part of the IPAM business, it’s
a complicated thing and the market is pretty much all filled up.
Foreman only offers poor’s-man “give me next IP address in sequence”
of “give me random IP address from this pool”. But IPAM is much more
than that, you have organizations, groups, subnets, pools, segments
and projects, you want to know who owns which IP address and typically
you want IP addresses to be associated with DNS records or integrated
with identity management.

The community only recently added External IPAM feature, the Smart-Proxy
API is still incubating. It has been a long process due to lack of time of the
main author. In the meantime, we have added webhooks which could be a good
alternative to do DHCP/DNS integration, not perfect but doable with advantage
that users can write scripts/services for any kind of providers easily.

One might think that DHCP management is a feature that is needed for
provisioning, but that is not the case. In order to do PXE/HTTP BOOT
provisioning, Foreman only need to manage TFTP/HTTP bootloader
config files (PXELinux, Grub2) and as long as the networking team
ensures that a DHCP server replies with PXE options (which TFTP/HTTP
to use essentially). To find associated host information, we have
provisioning tokens which are placed into TFTP PXE configuration
files, and even if this is turned off Anaconda installer is able to
figure out the details as it sends MAC address in the kickstart
request in HTTP headers and Foreman can leverage this to find the host
details in its inventory when tokens expire or are turned off.

Granted, some operating systems offer neither MAC address sending nor
provisioning tokens, therefore not managing DHCP would mean that
customers would be unable to do provisioning of niché operating
systems. In these scenarios, Foreman has a fallback mechanism of
detecting the host which is booting via its (REMOTE) IP address. This
information is matched against the inventory of provisioning NICs to
match the host. But Red Hat based or Debian based all do work fine,
most linuxes as well.

When creating a new managed host, users are required to associate a
Subnet (and Domain), this is essential for other features (e.g. which
Smart-Proxys to use), however, it automatically assumes that you want to
do provisioning and enforces to fill in an IP address - either via
DHCP “IPAM” feature (pick the next free IP from DHCP Smart-Proxy or use
internal DB table) or manually. I saw several BZs reported by
customers through the years asking to relax this requirement as they
never intended to use Foreman for provisioning and this is exactly
what I am proposing. We should allow hosts with blank IP addresses.

Infoblox dependencies dying is not the only reason why I propose this
change, there is one far more important. In the IPv6 world, you cannot
do IP address pre-reservation for unknown hosts (yet to be
provisioned), DHCPv6 protocol is vastly different from its older
brother and they deliberately omitted MAC addresses from it (it has
DUID now instead). Even if we want to keep DHCPv4 management in
Foreman, customers who already use DHCPv6 cannot use it with
Foreman and need to use the approach I am proposing here - unmanaged
DHCPv6 server that “points” to the Foreman TFTP Smart-Proxy.

And there is more, not only is the DHCPv6 protocol not capable of
associating host records without DUIDs, RHEL does not ship the
next-generation DHCPv6 server ISC Kea for various (license, technical)
reasons. The one that is included in RHEL8/9 is ISC DHCP which does
not offer any DHCPv6 management capabilities. So even if we added some
DUID support into Foreman, we cannot technically create those
records on the ISC DHCP server today.

And there is the old and famous “single subnet installer problem”. Our
Puppet installer only sets up a single subnet, if you want multiple
you need to use the (now documented) Hiera YAML approach. It’s because
our limited DHCP API does not offer any kind of subnet management
capabilities, you need to configure subnets externally and only then
Foreman can manage records within those subnets. This is not a bad
practice as it allows more simple API and less code to worry about,
however, it’s confusing and our workaround is the foreman_setup plugin
which often breaks upstream (it is not shipped with Foreman AFAIK).

This change would not come free of course, we have many places (e.g.
in templates) when we test @host.subnet.present? and if so, we assume
an IP address is also present. We would need to change this to
@host.ip.present? or @host.ip6.present? (@host.any_ip?) but that is an
easy change.

If this is successful we could even take this further and remove DNS
management capabilities and replace this with webhooks so customers
would have a fully tested webhook-based approach so they could use
pretty much any DNS provider they want. Today, DHCP and DNS
orchestration is separate, but this does not make sense (actually is a
known problem for Infoblox integration today) for some IPAM systems
where hosts are single entries for both DHCP and DNS (e.g. again
Infoblox), therefore they would simplement just a single webhook
script/service to create both DHCP and DNS.

Hello, we have no information about deprecation of WAPI 2.0, the official and latest manual says that the latest is 2.12 and it is supposed to be fully backward compatible with 2.x series. Can you elaborate on that?

We can take over the maintainance of this plugin, it`s not the first and definitely not the last. We will most likely do just security bugs tho.

this is exactly why this feature was added, to reduce the need to depend on other teams to do something as simple as setup a new host / vm… I dont think it was ever considered a full IPAM (its not) but, leverage one that can help (or allow basic api’s as you described with ISC dhcp or dnsmasq) is imho very useful (and the same goes for DHCP) I would be very much against dropping those features.

This is just what my coworker maintaining our infoblox has told me. From what I understand, 2.0 is currently the oldest fully supported API version. Some 1.X versions seem to be still work, too, but are currently deprecated and will probably be removed with the next major release. I will ask if he has any further information on this topic and come back on that.

I agree to the point that Foreman should not try to implement/mimic a fullblown IPAM system. Yet I think it is a essential feature for a provisioning and management system (what imo Foreman is at it’s core) to have the ability to integrate with these systems, at least for the basic features like IP management.

I have read about both, and while the External IPAM feature would probably be the right way to go, we are pretty much still stuck without it. I have looked into web-/shellhooks as a possible replacement, but from a sysadmin standpoint this sounds pretty horrible. The fact that the new hooks do not give any return values makes it impossible to determine whether or not your request worked (from how I understand the documentation). This would mean I need to wait for the deployment to finish to see if something went wrong, check DNS entries by hand, etc, simply because I cannot rely on the integration to warn me if something went wrong. Plus, you cannot write back info into Foreman from webhooks (I think there was a topic about this just a few days ago), which means we also cannot write the IP information back to Foreman to use during/after provisioning. Permanently using dynamic IP configuration is not necessarily what network admins like in their enterprise environment (to my experience).
Do not get me wrong, there are more than enough problems with the current approach, but webhooks are not the solution imho.

I understand your point, but adding new systems to IPAM is pretty much a must in large environments. Once several subnets and VLANs get involved, even getting a proper working DHCP reply for each of them becomes a hurdle of it self. More than one of my installations in the past broke just because the VMWare VLAN I assigned a system did not match the servers designated subnet as per DHCP. I highly doubt this would work better if not even the IPAM system knew where to put the systems.

100% agree on this. Forcing the use of IP management is for sure problematic for different use-cases.

I have never looked at DHCPv6, so I cannot comment on this. Do you have any experience how others are handling this issue in an IPv6 world?

I would assume this is more of a lab/small scale issue. Having your DHCP and DNS being completely managed by foreman-installer never struck me as the approach people would take in production environments, but since you mentioned it here I assume I am probably wrong.

The plugin itself is already under the foreman umbrella, it has just not been updated in years, probably because there was simply no need to. My main concern is actually with the dependency gem.

This would only work if you can live with the fact that DHCP/DNS information is not needed for provisioning, of course if you want this to be orchestrated by Foreman then it would need to be request-response.

I think Foreman should leverage the fact that Ruby is a scripting and extensive language. There should be DHCP/DNS and External IPAM provider that is able to convert these requests into simple scriptable actions. Users should be able to easily (after reading like one page of HOWTO) write implementation for any kind of DHCP/DNS/IPAM service.

Sure, I am not saying let’s not integrate with IPAM, I am saying that DHCP records are not needed in order to provision hosts via PXE.

There is no solution other than https://dhcpy6d.de if you really want to do the same thing which you do for IPv4 networks. I am not sure how reliable this is in practice tho.

Yeah, we can take that under our umbrella too, or fork in when a security vulnerability is found and upstream would not reply.

Let us know about the news about deprecation of WAPI 2.0. I think holding status-quo for now is the best thing to do rather than breaking current workflows.

I checked back with my co-worker and he told me that statement was more of “gut feeling” based on how Infoblox has handled these things in the past. He would expect them to deprecate WAPI 2.0 with the next release, but there is no official info on that.

Alright, I probably just misunderstood the intention of your proposal. On a first thought, I really like the idea of this proposal.

This would be great. I have no insight of how many people actually use the Infoblox integration, but since they are the biggest player in the IPAM business, I would assume maintaining this integration would be of benefit to a larger part of the community.

Infoblox is fully supported in Satellite, therefore at Red Hat we are fully committed to fix bugs and maintain whatever is needed to keep this going.

Take my words as a long-term wish how I would like to see Foreman in many years. I am currently doing a lot of recap of my 10 years experience with maintaining bare-metal provisioning components in Foreman and I have learned many lessons. Only if there was a chance to redesign everything from scratch :slight_smile:

2 Likes

I have a related question: I saw that foreman also wants/needs to maintain the networks itself and the VLANS.

Is there a change one could get those from the DDI-solution, too?

I’m not necessarily interested in doing DHCP in Foreman, I’d like Foreman to “request” an IP for a new host from the DDI-solution (and subsequently registering it there).

And for that, I assume, Foreman would need to know the subnet to request it from.

Foreman currently has two APIs:

A) DHCP API with implementations for ISC DHCP, MS DHCP, Infoblox and few others. This API is the “basic”, it has queries like “give me next free IP” or “create/delete IP/MAC reservation”.

When you create a Subnet in Foreman, you can set IPAM to DHCP meaning it will use the “next free IP” request to find next IP. If you set IPAM to DB/RandomDB, Foreman will maintain a pool of IPs in a SQL table. You can also set it to External IPAM:

B) External IPAM API which is new addition (still a “tech preview”) that has a request “suggest new IP” which will do the same, there are implementations for MyIPAM and Netbox. The API offers more rich requests (list subnets, groups) and there is a widget in development that can show the data on the dashboard. External API is not complete, we have merged basic support into Foreman core and there is a PR for Smart Proxy to bring the support in.

Very interesting thread, which i stumbled over first now.

I recently worked on the smart_proxy_dhcp_infoblox plugin (see this PR) and since i did not say “no” fast enough, ekohl now made me a maintainer of both infoblox plugins.

Here i described my first steps with getting deeply known the Foreman IP address selection process and issues and i already mentioned the external ipam approach there. Over there i told that it might be impossible with Infoblox as it (means external ipam core parts) lacks a MAC address needed for DHCP stuff, but reading @lzap longer text here now, i am changing my mind now. With proper booting it might indeed by possible to provision without DHCP records (and such without a MAC) at all as long as PXE/TFTP boot is properly configured. Maybe i will continue my approach to extend the external ipam smart proxy plugin with Infoblox in the future.

But for now i am happy to keep the two existing smart proxy infoblox plugins alive. Some users may have reasons still using them. Ours f.e. is lack of time/budget to change to a complete DHCP less setup right now.

I also saw that the git repo of the “infoblox” gem is archived and thus the gem not updated since 2019. But i am with @lzap that we should keep the status quo alive and maybe take actions (adopt/fork) once we stumble over bugs or vulnerabilities. So far i noticed no issues.