Development of IPAM Plugins

Development
#1

Hi,
i want to look in to the external ipam feature and extend it with an provider for NetBox

im not sure whats the right approach to contribute the code. should i extend the existing smart_proxy_ipam from @grizzthedj with the stuff for netbox?
Or should i create a new smart_proxy_ipam_netbox ?

thanks Matthias

3 Likes
#2

The right approach is to extend the existing smart_proxy_ipam, however, there needs to be a refactoring of that implementation before additional providers can be added effectively. What sort of timeline were you looking to implement Netbox as a provider?

2 Likes
#3

ok cool. i will do this in the next upcoming weeks. it doenst seem to be a lot of work, the netbox api looks quite easy and functional.

#4

hey @lzap & @grizzthedj
i have some questsions for you two :slight_smile:

  1. are we sure that this part in foreman itself is working?
    https://github.com/theforeman/foreman/blob/c4c097fab09588a886da113e0c598120644f5d34/lib/proxy_api/external_ipam.rb#L233

    when i try this out, the smart proxy correctly sends a 404 if a ip is not allocated in external ipam

    2020-07-31T16:05:00 0371b8ba [I] Finished GET /ipam/subnet/10.100.60.0/24/10.100.60.3 with 404 (413.5 ms)

    but foreman throws an error

    Creating IPv4 in External IPAM for norma-dolbin.whatever.internal task failed with the following error: ERF12-7024 [ProxyAPI::ProxyException]: Unable to obtain IP address for subnet_id 10.100.60.0/24 from External IPAM. ([RestClient::NotFound]: 404 Not Found) for proxy https://tfm-katello-t01.whatever.de:9090/ipam

  2. it would be good to have the possibility to enrich the ipam system with more Information than it an ip is in use and its mac address.
    the hosts fqdn or a link to the hosts page in foreman would be good.

  3. if i got this correctly the ip cache feature relies on mac addresses
    when creating a foreman host with vmware, the mac address is not available till the vm is deployed.
    right now, if i create two host parallel without a mac address filled in the race condition still applies

thanks Matthias

#5

The error handling needs a major overhaul, both on the Foreman and Smart Proxy side. It is full of layer violations. Quite often the error from the backend system is sent all the way to Foreman. The point you found is a good example of that.

You already found my PR (https://github.com/grizzthedj/smart_proxy_ipam/pull/36) but in the end I came to the point where I wanted to do a complete rewrite from scratch and didnā€™t have the time to do so.

The downside is that itā€™s already been merged to Foreman, so making changes to the API is painful now.

#6

This is very recent, there might be dragons. The plugin is not official yet.

Sure, work with @grizzthedj on extending it. Although Iā€™d slightly prefer making everything more robust, creating 2nd implementation, settling down on the final API and moving this into theforeman github organization.

Interesting, yeah. Itā€™s a bug. The API probably needs to be extended with some UUID flag and VMs would use VM UUID instead.

#7

Hi, was any progress made?

Managed to smart_proxy_ipam working as a plugin after some fiddling, only to find out the Netbox API has changed so the interaction is currently broken.

#8

@dmgeurts Iā€™m also interested in this, but as youā€™ve already gone through the process of getting it up and running, maybe this plugin will do the trick:

Also, could you share your notes on how to make it actually work? I didnā€™t come further then the following:

yum install -y make gcc openssl-devel ruby-devel
gem install foreman_netbox
echo "gem 'foreman_netbox'" > /usr/share/foreman/bundler.d/foreman_netbox.rb
systemctl restart foreman

I attempted it on a Satellite 6.11 system in my lab, which uses Foreman 3.1 underneath, but Foreman doesnā€™t want to start afterwards because (it seems) it canā€™t find certain files.

#9

Did you use EL 7 or 8? From the use of yum it suggests 7 and then you may have installed it using the system version rather than the SCL.

#10

Iā€™m running it on EL8 :slight_smile:

#11

@dmgeurts Could you elaborate on what is broken or what has changed in the Netbox API?

#12

As FYI, External IPAM features(with support for Netbox and phpIPAM) are in this PR awaiting to be merged into Smart Proxy core.

1 Like
#13

I now think the wrong API is used as when I look at the code the Netbox API looks right to me, yet the error shows the other API used.

As far as I can tell my configuration is correct as I see Netbox mentioned in the logs, Yet the API calls are not compatible with Netbox.

Iā€™m away from my computer at the moment so canā€™t give more specific details until a week from now.

#14

Back at work now so can give faster responses. Iā€™m on EL8 (CentOS Stream r8) and am finding that the queries made to Netbox are valid for phpIPAM, but not for Netbox. My config is set for Netbox and some of the logs show this as well, see Should installation of this plugin work on Foreman 3.3? Ā· Issue #56 Ā· grizzthedj/smart_proxy_ipam Ā· GitHub

My prior suspicions about the Netbox API having changed are probably wrong. I now think the IPAM plugin wrongly tries to use the phpIPAM API when talking to Netbox.

Aug 15 00:04:46 fm smart-proxy[1273044]: 2022-08-15 00:04:46 - JSON::ParserError - 783: unexpected token at '<!DOCTYPE HTML>
[...]
Aug 15 00:04:47 fm smart-proxy[1273044]: #011/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:48:in `get_ipam_subnet_by_cidr'
Aug 15 00:04:47 fm smart-proxy[1273044]: #011/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/netbox/netbox_client.rb:29:in `get_ipam_subnet'
Aug 15 00:04:47 fm smart-proxy[1273044]: #011/usr/local/share/gems/gems/smart_proxy_ipam-0.1.4/lib/smart_proxy_ipam/ipam_api.rb:100:in `block in <class:Api>'
[...]
Aug 15 00:04:47 fm smart-proxy[1273044]: 10.0.0.10 - - [15/Aug/2022:00:04:45 CEST] "GET /ipam/subnet/10.0.1.0/24?group= HTTP/1.1" 500 590509
Aug 15 00:04:47 fm smart-proxy[1273044]: - -> /ipam/subnet/10.0.1.0/24?group=
#15

All Iā€™ve been trying to do is add subnets known in Netbox to Foreman so that IPAM details can be kept in sync. We donā€™t register our hosts in Netbox but if Foreman can automate this then we may start doing so, but this is not an immediate need for us.

That said, itā€™s good to know that thereā€™s work on integrating the CMDB aspect of Netbox into Foreman.

#16

Thanks for the details @dmgeurts - I have posted a response to get more specific info in the github issue you opened. We can continue the conversation there, then report the fix back here once resolved.

1 Like
#17

Has any thought been put into pushing the hostname of the machine being built to the IPAM solution? I have been working on adding support for Bluecat IPAM, but it requires the hostname to add a record.

Thanks

#18

Some other users have also requested this. I think there is some value in having hostname, and potentially some other fields passed over as well.

Since the IPAM plugin integration is on its way into Smart Proxy core(the PR is above in this thread), I think it would make sense to wait until this is merged before doing anything.

#19

The PR has been stale since July. Is there any help needed? My Org would like to see extended IPAM support as well, especially with NetBox.

#20

I have attempted this again after the release of Satellite 6.12 on EL8

The steps above work like a charm, so I guess it was some support missing on the side of the gem or in Foreman.

One additional step I found was that you need to run these commands in addition:

foreman-rake db:migrate
systemctl restart foreman

Afterwards it looks like this in Satellite :slight_smile:

image
image1794Ɨ506 64.4 KB

I still need to look into it further, but for those that are looking around, it might be interesing :slight_smile: