New Plugin: foreman_phpipam

grizzthedj · April 22, 2019, 4:43pm

I am going to start development shortly on building a new Foreman new plugin that will integrate with the open source tool phpIPAM(https://phpipam.net/) for IP Address Management.

Based on my research, it seems that such a plugin does not exist, and there is not anything currently in development at this time. Please advise if this is not the case.

Our primary use case is to have phpIPAM determine what the next available IP Address will be when provisioning new hosts.

Subnets in Foreman will be created with the option of selecting “phpIPAM” for IPAM. Host NICs that use a subnet with the IPAM type “phpIPAM” will auto suggest the next available address from phpIPAM via API call.

I will start with the links below, but would welcome any recommendations or suggestions that anyone might have for getting started.

https://projects.theforeman.org/projects/foreman/wiki/How_to_Create_a_Plugin
https://guides.rubyonrails.org/engines.html

Cheers,

Chris

Ale · April 23, 2019, 5:45am

what about https://github.com/digitalocean/netbox ?

Marek_Hulan · April 23, 2019, 5:49am

Hello,

these may be useful too:
https://www.theforeman.org/contribute.html
https://www.theforeman.org/handbook.html
Custom IPAM provider - someone did the same thing for different IPAM, sadly not much info was shared

note that you’ll likely need a plugin on smart proxy side, which will talk to phpIPAM instance. The foreman plugin should probably just add new IPAM type, which is quite easy

List of Smart-Proxy Plugins - Foreman - you can take a look at proxy dhcp plugin if you want to go further than just to determine the free IP

let us know how it goes

Hope this helps

grizzthedj · April 23, 2019, 1:22pm

Thanks @Marek_Hulan for the links. I will have a look at these as well.

Good point about the smart proxy. My first glance assumption would be that everything might go in the Foreman plugin, but this makes sense.

This is really helpful - Thanks for responding!

grizzthedj · May 21, 2019, 4:36pm

Hello Community,

I have pushed up a first cut at new Foreman and Smart Proxy plugins for phpIPAM integration, and also new branches for Foreman and Smart Proxy patches(branch is plugin/foreman_phpipam for both). This mainly for a preliminary review and is still a WIP.

Here is what as been done at a high level(so far):

1). New Foreman Plugin - Repo: https://github.com/grizzthedj/foreman_phpipam

Provides a simple view in the Foreman UI(Infrastructure => phpIPAM Dashboard) to look at phpIPAM sections, subnets. More functionality will be added here. See the TOOD section in https://github.com/grizzthedj/foreman_phpipam/blob/master/README.md for more info.
Adds new phpIPAM feature to Features table via seed file.
Queries phpIPAM API via the Smart Proxy interface

2). New Smart Proxy Plugin - Repo: https://github.com/grizzthedj/smart_proxy_phpipam

Provides interface to phpIPAM API
- Get all sections
- Get all subnets for a given section
- Get next available IP for a given subnet
- More features will be integrated here

3). Patch to Foreman core:
https://github.com/grizzthedj/foreman/tree/plugin/foreman_phpipam

Added new IPAM provider phpipam (NOTE: I have not made this “pluggable”(yet), but this will be done)
Persists suggested IP back to phpIPAM on Host create

4). Patch to Smart Proxy core: https://github.com/grizzthedj/smart-proxy/tree/plugin/smart_proxy_phpipam

Simply added config/settings.d/phpipam.yml.example for phpIPAM URL and credentials

I have not created any PRs yet as it is not complete!(still need to write tests, finish translations, release to rubygems.org etc). I was hoping to get a preliminary code review, and also get a few questions answered(below). Would also welcome any other feedback here as well.

Questions:

1). In terms of getting the Smart Proxy base URL, I’m not quite sure how to do that. To get this working locally, I just used the Smart Proxy URL of the first SmartProxy from the DB(which is obviously not correct).

github.com

grizzthedj/foreman/blob/plugin/foreman_phpipam/app/models/subnet.rb#L204




def template_proxy(attrs = {})
  @template_proxy ||= ProxyAPI::Template.new({:url => template.url}.merge(attrs)) if template?
end


def phpipam?
  supports_ipam_mode?(:phpipam) 
end


def phpipam_proxy(attrs = {})
  proxy_url = SmartProxy.first[:url]
  @phpipam_proxy ||= ProxyAPI::Phpipam.new({:url => proxy_url}.merge(attrs)) if phpipam?
end


def ipam?
  self.ipam != IPAM::MODES[:none]
end


def ipam_needs_range?
  ipam? && self.ipam != IPAM::MODES[:eui64]
end

I tried following the way DHCP was implemented, but I didn’t really follow. Any guidance here on how this part should be implemented would be appreciated.

2). I have made modifications to some HTML that exists in the following translation: “You can select one of the IPAM modes supported by the selected IP protocol: …”. This has already been translated in the following .pot files:

es, fr, ja, pt_BR, zh_CN

Should I reset the “msgstr” to blank string so that it will be translated again? How are updates to existing translations handled?

Thanks in advance!

Chris

Dirk · May 22, 2019, 9:44am

For the code I can not help much, but two small suggestions.

Most times it is a random one nowadays so you can not guess next IP used. Perhaps is is also an option for you to implement.

No need to add this in smart-proxy, keep the config in the Smart-Proxy plugin and in packaging we will handle it. (As an example: smart_proxy_monitoring/settings.d at master · theforeman/smart_proxy_monitoring · GitHub and foreman-packaging/packages/plugins/rubygem-smart_proxy_monitoring/rubygem-smart_proxy_monitoring.spec at rpm/develop · theforeman/foreman-packaging · GitHub)

grizzthedj · May 24, 2019, 4:37pm

Thanks @Dirk

I will ensure that the smart proxy plugin(only) has the example settings file. In terms of the next available IP for a subnet, phpIPAM supports this. See /api/my_app/subnets/{id}/first_free/ at https://phpipam.net/api/api_documentation/

Thanks,

Chris

lzap · May 28, 2019, 2:04pm

Hey, looks good so far. The smart-proxy API could be generalized to IPAM in a similar way as it’s done in DHCP or DNS and implementation could by myIpam. In that case, you could easily commit this into smart-proxy core as it does not bring any new rubygem dependencies so we could distribute this with core. Optionally you can only add IPAM API to core and keep your myIpam implementation in a separate plugin if you want.

You need to add a new AciveRecord association for your new feature:

github.com

grizzthedj/foreman/blob/plugin/foreman_phpipam/app/models/subnet.rb#L60-L64


belongs_to_proxy :dns,
  :feature => N_('DNS'),
  :label => N_('Reverse DNS Proxy'),
  :api_description => N_('DNS Proxy ID to use within this subnet'),
  :description => N_('DNS Proxy to use within this subnet for managing PTR records, note that A and AAAA records are managed via Domain DNS proxy')

Just copy and paste and modify and then you can use this so get the url the similar way as other features do.

Just change all strings and don’t worry, translators have tools which will find fuzzy strings and very likely show how it was updated.

grizzthedj · May 29, 2019, 6:07pm

Thanks @lzap.

I added this block of code, and am getting this error undefined method 'url' for nil:NilClass when trying to get the url here. This makes sense since I haven’t set this url anywhere.

  def phpipam_proxy(attrs = {})
    @phpipam_proxy ||= ProxyAPI::Phpipam.new({:url => phpipam.url}.merge(attrs)) if phpipam?
  end

Where/when does this url get set for DNS / DHCP and other features? Does this happen when they are installed via foreman-installer? Are the URL’s stored in the database?

Cheers,

Chris

lzap · June 3, 2019, 7:18am

Have you added belongs_to_proxy :phpipam? If so, has the subnet associated phpipam proxy?

This is a simple DB association, users set it in UI/API in Subnets - Edit - Proxies - DNS Proxy DHCP Proxy etc. You can only set those proxies which reports the features. It is a bit complicated, lemme try again:

Start a proxy with phpipam module (feature) so when you curl proxy/features you see this reported.
Add this proxy into Foreman using UI, existing one can refresh features with Refresh features button
Make sure the feature phpipam is detected
Now visit Subnet - Edit and associate the subnet to phpipam proxy
Now the association should be set and not nil

grizzthedj · June 3, 2019, 3:06pm

That was the missing piece. I did not associate the subnet with the phpipam proxy. Once I did this, everything started working. Thanks @lzap!

lzap · June 4, 2019, 7:29am

Great, if you want to invest little bit more time into this, you can name the feature IPAM instead of phpIpam so it’s more generic. The same for proxy feature and API, so in the future the code can be extracted into module with multiple implementations.

grizzthedj · June 4, 2019, 2:46pm

Makes sense. I can definitely find some more time to make this more pluggable/extendable.

For what you are envisioning, the plugins themselves would be renamed to foreman_ipam and smart_proxy_ipam, and they would contain the implementations for phpIPAM, Netbox and other IPAM providers? We would then introduce 2 new models on the Foreman side:

Ipam which would replace the IPAM::MODES constant(i.e. DHCP, Internal DB, Random DB etc.)
IpamProvider, a one to many with Ipam. Providers would be phpIPAM, Netbox etc. This would be populated in a new dropdown on the create Subnet page, when the user selects IPAM(for IPAM).

The providers could then be enabled via config(or some other way?) on the plugin side, and the proxy plugin would provide an endpoint to Foreman to expose enabled IPAM providers?

Is what you are thinking from an implementation perspective? Just want to make sure we are on the same page.

lzap · June 6, 2019, 1:47pm

I suggest you to carry on with what you are doing now, this was an idea for the future. If you would like to go this path, this would be probably best to merge into core. I suggest to go with a separate plugins, myipam only for now. After some time in full deployment feel free to get back to us and I will help you with merging the stuff into core.

grizzthedj · June 6, 2019, 6:48pm

Sounds good. I have renamed the plugins to IPAM, but the IPAM::MODES remains a constant until we revisit later. Just need to fix/write some tests, the add some docs, then will submit the PR.

lzap · June 7, 2019, 11:31am

Thanks, just keep the (most) functionality as plugins which is a great way for incubating new code. Let’s revisit pushing this into core after some time. Ping me if you need any assistance or review.

grizzthedj · June 10, 2019, 4:27pm

Hi @lzap,

I am running the Foreman test suite and am seeing 131 errors and 4 failed tests.

Below is an example error. All these errors are almost exactly the same(i.e. they contain Error: Component not found: ...), and are related to javascript errors. Perhaps I am maybe missing some gem/library etc. in my local dev environment? I have phantomjs, poltergeist, capybara-webkit and qt installed on a MAC running High Sierra.

For the failures, it doesn’t seem like the 4 failures below come as a result of my implementation, but of course I could be wrong. Also not sure what the state of the build is.

Any light you could shed on this would be great.

Errors:

Error:
PtableJSTest#test_0003_make sure that ptable names with slashes and dots work:
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

Error: Component not found: ReactApp among SearchBar, AutoComplete, DonutChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, DateTime, ModelsTable, AuditsPage, TemplateGenerator
Error: Component not found: ReactApp among SearchBar, AutoComplete, DonutChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, DateTime, ModelsTable, AuditsPage, TemplateGenerator
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in getComponent
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in markup
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in mount
    test/integration/ptable_js_test.rb:24:in `block in <class:PtableJSTest>'

Failures:

Failure #1:
=========================================
Api::V2::RolesControllerTest#test_0013_org admin should not create roles by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/roles_controller_test.rb:252]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_roles"
+"Missing one of the required permissions: create_roles, create_roles"


Failure #2:
=========================================
Api::V2::LocationsControllerTest#test_0036_org admin should not create locations by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/locations_controller_test.rb:407]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_locations"
+"Missing one of the required permissions: create_locations, create_locations"


Failure #3:
=========================================
TemplateRenderJobTest::processing#test_0002_render report and delivers it to mail [/Users/christopher.smith/Documents/development/opensource/foreman/app/jobs/template_render_job.rb:13]:
unexpected invocation: ReportMailer.report({"foo" => "bar", "send_mail" => true, "mail_to" => "this@email.cz", "gzip" => true}, "result", {:start => 2019-06-10 11:16:02 -0400 (1560179762.908043 secs), :end => 2019-06-10 11:16:02 -0400 (1560179762.9080782 secs)})
unsatisfied expectations:
- expected exactly once, not yet invoked: #<Mock:mailer>.deliver_now(any_parameters)
- expected exactly once, not yet invoked: #<AnyInstance:ReportComposer>.report_filename(any_parameters)
- expected exactly once, not yet invoked: ReportMailer.report("this@email.cz", "report.gz", "result")
satisfied expectations:
- expected exactly once, invoked once: #<AnyInstance:ReportComposer>.render(any_parameters)


Failure #4:
=========================================
AccessPermissionsTest#test_0780_route api/v2/fake_with_filename/index should have a permission that grants access [/Users/christopher.smith/Documents/development/opensource/foreman/test/unit/shared/access_permissions_test_base.rb:29]:
permission for api/v2/fake_with_filename/index not found, check access_permissions.rb.
Expected [] to not be empty.

lzap · June 12, 2019, 3:35pm

I think I’ve seen them on CI, @tbrisker or @Marek_Hulan? The Capybara that one I haven’t seen yet, @ezr-ondrej?

tbrisker · June 12, 2019, 6:33pm

not sure about the permissions one, but for the other one perhaps try deleting public/webpack/* and running rake webpack:compile. if that doesn’t help, it might also be some MacOS weirdness, perhaps @gilad215 ran into something similar?

grizzthedj · June 12, 2019, 7:23pm

That cleared up the 131 Capybara errors! Only seeing the 4 failed tests now.

Failure:
Api::V2::RolesControllerTest#test_0013_org admin should not create roles by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/roles_controller_test.rb:252]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_roles"
+"Missing one of the required permissions: create_roles, create_roles"

Failure:
Api::V2::LocationsControllerTest#test_0036_org admin should not create locations by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/locations_controller_test.rb:407]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_locations"
+"Missing one of the required permissions: create_locations, create_locations"

Failure:
TemplateRenderJobTest::processing#test_0002_render report and delivers it to mail [/Users/christopher.smith/Documents/development/opensource/foreman/app/jobs/template_render_job.rb:13]:
unexpected invocation: ReportMailer.report({"foo" => "bar", "send_mail" => true, "mail_to" => "this@email.cz", "gzip" => true}, "result", {:start => 2019-06-11 11:55:20 -0400 (1560268520.4714398 secs), :end => 2019-06-11 11:55:20 -0400 (1560268520.471474 secs)})
unsatisfied expectations:
- expected exactly once, not yet invoked: #<Mock:mailer>.deliver_now(any_parameters)
- expected exactly once, not yet invoked: #<AnyInstance:ReportComposer>.report_filename(any_parameters)
- expected exactly once, not yet invoked: ReportMailer.report("this@email.cz", "report.gz", "result")
satisfied expectations:
- expected exactly once, invoked once: #<AnyInstance:ReportComposer>.render(any_parameters)

Failure:
AccessPermissionsTest#test_0780_route api/v2/fake_with_filename/index should have a permission that grants access [/Users/christopher.smith/Documents/development/opensource/foreman/test/unit/shared/access_permissions_test_base.rb:29]:
permission for api/v2/fake_with_filename/index not found, check access_permissions.rb.
Expected [] to not be empty.

Thanks @tbrisker!