New Plugin: foreman_phpipam

lzap · June 3, 2019, 7:18am

Have you added belongs_to_proxy :phpipam? If so, has the subnet associated phpipam proxy?

This is a simple DB association, users set it in UI/API in Subnets - Edit - Proxies - DNS Proxy DHCP Proxy etc. You can only set those proxies which reports the features. It is a bit complicated, lemme try again:

Start a proxy with phpipam module (feature) so when you curl proxy/features you see this reported.
Add this proxy into Foreman using UI, existing one can refresh features with Refresh features button
Make sure the feature phpipam is detected
Now visit Subnet - Edit and associate the subnet to phpipam proxy
Now the association should be set and not nil

grizzthedj · June 3, 2019, 3:06pm

That was the missing piece. I did not associate the subnet with the phpipam proxy. Once I did this, everything started working. Thanks @lzap!

lzap · June 4, 2019, 7:29am

Great, if you want to invest little bit more time into this, you can name the feature IPAM instead of phpIpam so it’s more generic. The same for proxy feature and API, so in the future the code can be extracted into module with multiple implementations.

grizzthedj · June 4, 2019, 2:46pm

Makes sense. I can definitely find some more time to make this more pluggable/extendable.

For what you are envisioning, the plugins themselves would be renamed to foreman_ipam and smart_proxy_ipam, and they would contain the implementations for phpIPAM, Netbox and other IPAM providers? We would then introduce 2 new models on the Foreman side:

Ipam which would replace the IPAM::MODES constant(i.e. DHCP, Internal DB, Random DB etc.)
IpamProvider, a one to many with Ipam. Providers would be phpIPAM, Netbox etc. This would be populated in a new dropdown on the create Subnet page, when the user selects IPAM(for IPAM).

The providers could then be enabled via config(or some other way?) on the plugin side, and the proxy plugin would provide an endpoint to Foreman to expose enabled IPAM providers?

Is what you are thinking from an implementation perspective? Just want to make sure we are on the same page.

lzap · June 6, 2019, 1:47pm

I suggest you to carry on with what you are doing now, this was an idea for the future. If you would like to go this path, this would be probably best to merge into core. I suggest to go with a separate plugins, myipam only for now. After some time in full deployment feel free to get back to us and I will help you with merging the stuff into core.

grizzthedj · June 6, 2019, 6:48pm

Sounds good. I have renamed the plugins to IPAM, but the IPAM::MODES remains a constant until we revisit later. Just need to fix/write some tests, the add some docs, then will submit the PR.

lzap · June 7, 2019, 11:31am

Thanks, just keep the (most) functionality as plugins which is a great way for incubating new code. Let’s revisit pushing this into core after some time. Ping me if you need any assistance or review.

grizzthedj · June 10, 2019, 4:27pm

Hi @lzap,

I am running the Foreman test suite and am seeing 131 errors and 4 failed tests.

Below is an example error. All these errors are almost exactly the same(i.e. they contain Error: Component not found: ...), and are related to javascript errors. Perhaps I am maybe missing some gem/library etc. in my local dev environment? I have phantomjs, poltergeist, capybara-webkit and qt installed on a MAC running High Sierra.

For the failures, it doesn’t seem like the 4 failures below come as a result of my implementation, but of course I could be wrong. Also not sure what the state of the build is.

Any light you could shed on this would be great.

Errors:

Error:
PtableJSTest#test_0003_make sure that ptable names with slashes and dots work:
Capybara::Poltergeist::JavascriptError: One or more errors were raised in the Javascript code on the page. If you don't care about these errors, you can ignore them by setting js_errors: false in your Poltergeist configuration (see documentation for details).

Error: Component not found: ReactApp among SearchBar, AutoComplete, DonutChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, DateTime, ModelsTable, AuditsPage, TemplateGenerator
Error: Component not found: ReactApp among SearchBar, AutoComplete, DonutChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart, Pagination, Layout, EmptyState, BarChart, ChartBox, ComponentWrapper, ConfigReports, DiffModal, TemplateInput, RelativeDateTime, LongDateTime, ShortDateTime, IsoDate, DateTime, ModelsTable, AuditsPage, TemplateGenerator
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in getComponent
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in markup
    at http://127.0.0.1:51527/webpack/bundle-ac35546fdea74fb3c243.js:1 in mount
    test/integration/ptable_js_test.rb:24:in `block in <class:PtableJSTest>'

Failures:

Failure #1:
=========================================
Api::V2::RolesControllerTest#test_0013_org admin should not create roles by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/roles_controller_test.rb:252]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_roles"
+"Missing one of the required permissions: create_roles, create_roles"


Failure #2:
=========================================
Api::V2::LocationsControllerTest#test_0036_org admin should not create locations by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/locations_controller_test.rb:407]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_locations"
+"Missing one of the required permissions: create_locations, create_locations"


Failure #3:
=========================================
TemplateRenderJobTest::processing#test_0002_render report and delivers it to mail [/Users/christopher.smith/Documents/development/opensource/foreman/app/jobs/template_render_job.rb:13]:
unexpected invocation: ReportMailer.report({"foo" => "bar", "send_mail" => true, "mail_to" => "this@email.cz", "gzip" => true}, "result", {:start => 2019-06-10 11:16:02 -0400 (1560179762.908043 secs), :end => 2019-06-10 11:16:02 -0400 (1560179762.9080782 secs)})
unsatisfied expectations:
- expected exactly once, not yet invoked: #<Mock:mailer>.deliver_now(any_parameters)
- expected exactly once, not yet invoked: #<AnyInstance:ReportComposer>.report_filename(any_parameters)
- expected exactly once, not yet invoked: ReportMailer.report("this@email.cz", "report.gz", "result")
satisfied expectations:
- expected exactly once, invoked once: #<AnyInstance:ReportComposer>.render(any_parameters)


Failure #4:
=========================================
AccessPermissionsTest#test_0780_route api/v2/fake_with_filename/index should have a permission that grants access [/Users/christopher.smith/Documents/development/opensource/foreman/test/unit/shared/access_permissions_test_base.rb:29]:
permission for api/v2/fake_with_filename/index not found, check access_permissions.rb.
Expected [] to not be empty.

lzap · June 12, 2019, 3:35pm

I think I’ve seen them on CI, @tbrisker or @Marek_Hulan? The Capybara that one I haven’t seen yet, @ezr-ondrej?

tbrisker · June 12, 2019, 6:33pm

not sure about the permissions one, but for the other one perhaps try deleting public/webpack/* and running rake webpack:compile. if that doesn’t help, it might also be some MacOS weirdness, perhaps @gilad215 ran into something similar?

grizzthedj · June 12, 2019, 7:23pm

That cleared up the 131 Capybara errors! Only seeing the 4 failed tests now.

Failure:
Api::V2::RolesControllerTest#test_0013_org admin should not create roles by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/roles_controller_test.rb:252]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_roles"
+"Missing one of the required permissions: create_roles, create_roles"

Failure:
Api::V2::LocationsControllerTest#test_0036_org admin should not create locations by default [/Users/christopher.smith/Documents/development/opensource/foreman/test/controllers/api/v2/locations_controller_test.rb:407]:
--- expected
+++ actual
@@ -1 +1 @@
-"Missing one of the required permissions: create_locations"
+"Missing one of the required permissions: create_locations, create_locations"

Failure:
TemplateRenderJobTest::processing#test_0002_render report and delivers it to mail [/Users/christopher.smith/Documents/development/opensource/foreman/app/jobs/template_render_job.rb:13]:
unexpected invocation: ReportMailer.report({"foo" => "bar", "send_mail" => true, "mail_to" => "this@email.cz", "gzip" => true}, "result", {:start => 2019-06-11 11:55:20 -0400 (1560268520.4714398 secs), :end => 2019-06-11 11:55:20 -0400 (1560268520.471474 secs)})
unsatisfied expectations:
- expected exactly once, not yet invoked: #<Mock:mailer>.deliver_now(any_parameters)
- expected exactly once, not yet invoked: #<AnyInstance:ReportComposer>.report_filename(any_parameters)
- expected exactly once, not yet invoked: ReportMailer.report("this@email.cz", "report.gz", "result")
satisfied expectations:
- expected exactly once, invoked once: #<AnyInstance:ReportComposer>.render(any_parameters)

Failure:
AccessPermissionsTest#test_0780_route api/v2/fake_with_filename/index should have a permission that grants access [/Users/christopher.smith/Documents/development/opensource/foreman/test/unit/shared/access_permissions_test_base.rb:29]:
permission for api/v2/fake_with_filename/index not found, check access_permissions.rb.
Expected [] to not be empty.

Thanks @tbrisker!

lzap · June 14, 2019, 9:28am

Frankly I am surprised that Foreman starts on MacOS. I know either smart proxy or foreman was not starting last time I tried.

You can either try start digging or just ignore these tests - your work will very unlikely affect them.

One more idea: try to migrate and reseed the database, admin is being assigned with some permissions which you might run twice?

grizzthedj · June 14, 2019, 3:24pm

I tried re-migrating and seeding the database, and still getting the same 4 failures.

Funny thing is that when I run the two roles and permissions test files individually, they are green, but fail when running the whole test suite.

bundle exec rails test test/controllers/api/v2/locations_controller_test.rb
bundle exec rails test test/controllers/api/v2/roles_controller_test.rb

Perhaps there are some tests that don’t clean up properly prior to these running?

I will try to dig a little more.

grizzthedj · June 17, 2019, 6:39pm

@lzap @tbrisker

I checked out the latest develop branch, and it seems the same 4 tests are failing there as well. I’m not sure if this has to do with my local setup or not, but I have proceeded with the PR anyway.

I have pushed the latest changes and submitted the PR for the phpIPAM Integration with Foreman in a new branch called plugin/foreman_ipam.

github.com/theforeman/foreman

Fixes #27067 - IPAM Integration with phpIPAM

theforeman:develop ← grizzthedj:plugin/foreman_ipam

opened 06:29PM - 17 Jun 19 UTC

grizzthedj

+542 -4

This PR is a patch to support of the use of plugins for external IPAM use, with …opensource provider phpIPAM(https://phpipam.net). The primary use case is to provide the next available IPv4 address from phpIPAM, for a given phpIPAM subnet. The functionality that Foreman will use for external IPAM is encompassed in the below plugins(which are both new). **foreman_ipam(v0.0.7)** **Git:** https://github.com/grizzthedj/foreman_ipam **Gem:** https://rubygems.org/gems/foreman_ipam **smart_proxy_ipam(v0.0.13)** **Git:** https://github.com/grizzthedj/smart_proxy_ipam **Gem:** https://rubygems.org/gems/smart_proxy_ipam The `foreman_ipam` plugin provides a very basic dashboard to display sections and subnets from phpIPAM, and the `smart_proxy_ipam` plugin communicates with the phpIPAM API. Features and functionality * Simple IPAM Dashboard page(Infrastructure => IPAM Dashboard) to view sections and subnets for phpIPAM. More features will be added here based on internal user feedback and from the community. * When creating a phpIPAM subnet(IPAM type = External IPAM) in Foreman, subnet/mask must exist in phpIPAM, or error will be thrown * When creating a new host in Foreman, auto suggested IP's from phpIPAM will be persisted back to phpIPAM on Host submit. * When updating the IP address on a NIC that uses a phpIPAM subnet, the old IP will be deleted from phpIPAM, and new one will be added. * Deleting a host in Foreman will delete all NIC ip's that used a phpIPAM subnet * Deleting a NIC from a host will also delete the IP from phpIPAM * Deleting a subnet in Foreman will NOT delete the associated subnet in phpIPAM. This could easily be added later if it makes sense to the community, however initial thoughts are to not delete it. NOTE: When creating a host in Foreman and adding multiple nics/interfaces, the same IP will be suggested from phpIPAM since it is not actually reserved in phpIPAM until you submit the host. The workaround for multiple nics is to use the suggested ip for the first interface(e.g. 10.0.0.1), then manually increment the IP for the second interface(e.g. 10.0.0.2). Errors will be thrown however if IP's are already taken. Prerequisites for testing: * Working Foreman environment. * A phpIPAM instance that the Forman Smart Proxy has network connectivity to. * The API must be enabled in the phpIPAM settings, and `Prettify Links` also enabled. * An API user in phpIPAM must be created. The credentials for this user are to be put in the `config/settings.d/external_ipam.yml` file in the Smart Proxy core

Here are the links to the Foreman Plugin repos, and published gems:

foreman_ipam(v0.0.3)
Git: GitHub - grizzthedj/foreman_ipam: Foreman plugin for integration with External IPAM providers
Gem: foreman_ipam | RubyGems.org | your community gem host

smart_proxy_ipam(v0.0.4)
Git: GitHub - grizzthedj/smart_proxy_ipam: Foreman Smart Proxy plugin for integration with External IPAM providers
Gem: smart_proxy_ipam | RubyGems.org | your community gem host

There are more details on the features and functionality in the PR itself, but had a few questions regarding next steps.

Questions:

How does one integrate the install/setup of these plugins with the foreman-installer?
How can we get this added to the list of supported Foreman/Proxy plugins?
In terms of integration testing, is there a project that has integration tests for Foreman? There are some tests, but none that test the actual phpIPAM integration.
Is there any available infrastructure(in the cloud or otherwise) where we can setup a small phpIPAM instance for integration testing purposes?

Feel free to reach out if I have missed anything here!

Cheers,

Chris

tbrisker · June 18, 2019, 8:06am

This is a two step process:

adding the plugins to the puppet-foreman and puppet-foreman_proxy modules. Examples can be seen here for foreman and here for proxy.
once the modules are updated, adding them as installer options requires adding them to the answers file and creating a migration. example here

List of Plugins - Foreman is a wiki page that can be edited to add it, if you don’t have permission to do that send me a private message with your redmine username and I’ll fix it.
If you wish to transfer the plugin to the official theforeman organization under github, please reach out to me or one of the other org admins (preferably @ekohl (ewoud on irc) or @ehelms, though @ohadlevy and @Gwmngilfen may also be able to help if they have time) on IRC to facilitate it. We can also open a project on redmine for tracking issues if you prefer that to github.

We have a jenkins instance that is used to test many projects. The configuration for it is maintained under https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder and should be hopefully not be too complicated to add jobs for the plugin based on examples from other plugins. @infra team should be able to help guide you here towards the best practices.

I’m not sure what it entails to spin up an instance, but we do have some tests that run vms as part of the testing process on our infra. If it’s something that can be automatically spun up in a vm or container as part of the testing process that’s likely possible. Deffering to @infra team again for further guidance.

lzap · June 18, 2019, 12:05pm

Well then it smells, if you checkout develop locally do you see the same four tests failing?

I’ve reviewed your initial patch, keep it going. You are defining brand new Smart Proxy API (feature) here which is great, we like this design as it really paied off so far.

grizzthedj · June 18, 2019, 1:36pm

I checked out the latest develop branch locally, and the same 4 tests are failing there as well. Not sure if it is related to my local environment or not.

Thanks @lzap and @tbrisker for the feedback on the PR, and for answering the other questions. I will get cracking on those changes!

lzap · June 18, 2019, 1:47pm

Probably a new thread with better subject could help, there are lot more devs here who might know.

grizzthedj · June 18, 2019, 5:50pm

lzap · March 20, 2020, 9:44am

For the record, the Foreman core part has been merged and will be in Foreman 2.1: