Foreman - Multi vCenter and Multi Network Configuration?

[fdo_ng]

Hi Everyone, I am currently running Foreman 1.21.3 quickstart setup. Have it configured to deploy Centos/ESXI vms succesfully. I am trying to add a second vCenter as Compute Resource as well as the new subnet and I am getting the following error:
“Create DHCP Settings for bruce-arens.test.ops task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://centpx02.test.ops:8443/dhcp”

• Do I need to deploy a new smart proxy vm on the second vCenter?
• All the network I am testing with are L3 they can resolve and reach foreman server.

[cintrix84]

Hi @fdo_ng,

One option is to deploy a smart proxy vm, but if they call all reach each other, I would look at adding a 2nd dhcp zone with this example:

Foreman :: Plugin Manuals - Multiple subnets and domains

and see if that helps resolve the issue for you.

[fdo_ng]

I forgot to mention my second vCenter is on a complete different vlan/subnet. I added the new subnet in Foreman pointing to the only Foreman i had.

My environment only have 1 domain no DHCP, I was trying to configure PXE Kickstart to assign a static IP instead of using DHCP.

Thanks @cintrix84 will check that out.

[areyus]

You do not need any additional smartrpoxies for a new VCenter or new Subnet as long as

1. Foreman can reach the new VCenter
2. Foreman is reachable from the new Subnet (and, if applicable, the same DHCP and DNS servers are used as with your other subnets)

Can you clarify on which step exactly you get the mentioned error? I would assume subnet creation (i.e. clicking submit on the new subnet form)?
If so, can you confirm that you have
a) left IPAM and boot mode on the subnet as something different than DHCP
b) have not set a DHCP proxy?
Also, pleas post the related portion of /var/log/foreman/producion.log and /var/log/foreman-proxy/proxy.log.

Regards

[fdo_ng]

Hi areyus,

1. Foreman can reach the new VCenter - YES
2. Foreman is reachable from the new Subnet (and, if applicable, the same DHCP and DNS servers are used as with your other subnets) - YES all network I am using are L3, and vCenters can reach foreman.

In Foreman, I have the new subnet proxies pointing to the only foreman smart proxy i have:

• Infrastructure/Subnet/Proxies/[DHCP,TFTP,REverDNS,Discovery] all pointing at the foreman server.
  This work when I provision vms to the vCenter where foreman resides. However when I deploy to the new vCenter (compute resource) with new subnet, I get that error message during provision.
  
  

If so, can you confirm that you have
a) left IPAM and boot mode on the subnet as something different than DHCP
b) have not set a DHCP proxy?

pulling the logs…now…

[fdo_ng]

Here are the logs:

foreman-proxy/proxy.log:::
2019-06-12T09:55:02 be1fbbac [I] Finished POST / with 400 (28.43 ms)
2019-06-12T09:55:02 be1fbbac [I] Started DELETE /10.158.47.0/mac/00:50:56:90:71:43
2019-06-12T09:55:02 be1fbbac [I] Finished DELETE /10.158.47.0/mac/00:50:56:90:71:43 with 200 (17.79 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /serverName
2019-06-12T09:55:34 be1fbbac [I] Finished GET /serverName with 200 (0.43 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /10.158.47.0/mac/00:50:56:90:71:43
2019-06-12T09:55:34 be1fbbac [E] No DHCP record for MAC 10.158.47.0/00:50:56:90:71:43 found
2019-06-12T09:55:34 be1fbbac [I] Finished GET /10.158.47.0/mac/00:50:56:90:71:43 with 404 (0.81 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /10.158.47.0/ip/10.158.47.151
2019-06-12T09:55:34 be1fbbac [E] No DHCP records for IP 10.158.47.0/10.158.47.151 found
2019-06-12T09:55:34 be1fbbac [I] Finished GET /10.158.47.0/ip/10.158.47.151 with 404 (0.65 ms)
2019-06-12T09:55:38 be1fbbac [I] Started GET /serverName
2019-06-12T09:55:38 be1fbbac [I] Finished GET /serverName with 200 (0.36 ms)
2019-06-12T09:55:38 be1fbbac [I] Started POST /10.158.47.0
2019-06-12T09:55:38 be1fbbac [I] Finished POST /10.158.47.0 with 200 (19.8 ms)
2019-06-12T09:55:38 be1fbbac [I] Started POST /
2019-06-12T09:55:38 be1fbbac [E] Update errors: Answer:

foreman/production.log::
Completed 200 OK in 4420ms (Views: 300.2ms | ActiveRecord: 58.5ms)
2019-06-12T09:56:33 [I|app|9e8eb0cd] Started GET “/tasks/970fe84d-3afe-495a-b5ae-885a8d2fc6c0” for 10.163.253.201 at 2019-06-12 09:56:33 -0400
2019-06-12T09:56:33 [I|app|9e8eb0cd] Processing by TasksController#show as /
2019-06-12T09:56:33 [I|app|9e8eb0cd] Parameters: {“id”=>“970fe84d-3afe-495a-b5ae-885a8d2fc6c0”}
2019-06-12T09:56:33 [I|app|9e8eb0cd] Current user set to admin (admin)
2019-06-12T09:56:33 [I|app|9e8eb0cd] Rendered tasks/_list.html.erb (0.6ms)
2019-06-12T09:56:33 [I|app|9e8eb0cd] Completed 200 OK in 7ms (Views: 1.0ms | ActiveRecord: 1.4ms)
2019-06-12T09:56:36 [I|app|78b1d6cc] Started GET “/notification_recipients” for 10.163.253.201 at 2019-06-12 09:56:36 -0400
2019-06-12T09:56:36 [I|app|78b1d6cc] Processing by NotificationRecipientsController#index as JSON
2019-06-12T09:56:36 [I|app|78b1d6cc] Current user set to admin (admin)
2019-06-12T09:56:36 [I|app|78b1d6cc] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 1.4ms)

Might be related to this:
https://theforeman.org/2017/07/adding-new-subnet-for-provisioning.html

How does Foreman import a subnet via the smart proxy

When installing foreman, I've used the installer to setup a dhcp service on the same host. I define a network, a subnet range, name server, router, dns search domain, etc everything you'd expect in a dhcp service, and it works just fine. When going into the smart proxy and clicking "import IPV4 subnets" the dhcp configuration I've defined by the installer is detected and displayed in the menu ready for import. However there are fields / parameters missing from the foreman page that are d…

This is my dhcpd.conf – added the new subnet toward the end…

dhcpd.conf

omapi-port 7911;

default-lease-time 43200;
max-lease-time 86400;

ddns-update-style none;

option domain-name “ashlab.ops”;
option domain-name-servers 10.159.18.168;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update on; # set the “O” and “S” flag bits
option fqdn.rcode2 255;
option pxegrub code 150 = text ;

Bootfile Handoff

next-server 10.159.18.168;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
filename “grub2/shim.efi”;
} elsif option architecture = 00:07 {
filename “grub2/shim.efi”;
} elsif option architecture = 00:09 {
filename “grub2/shim.efi”;
} else {
filename “pxelinux.0”;
}

log-facility local7;

include “/etc/dhcp/dhcpd.hosts”;

ashlab.ops

subnet 10.159.18.0 netmask 255.255.255.0 {

option subnet-mask 255.255.255.0;
}

subnet 10.158.47.0 netmask 255.255.255.0 {

option subnet-mask 255.255.255.0;
option routers 10.158.47.1;
}

[areyus]

Here are the logs:

foreman-proxy/proxy.log:::
2019-06-12T09:55:02 be1fbbac [I] Finished POST / with 400 (28.43 ms)

Hi,

could you provide the log entries before this particular line? I would asume there might be interesting stuff just before that.
Also, I think the production.log entries are from to late in time since there are no errors that should be there.

[fdo_ng]

Here is a bigger chunk of the proxy.log

2019-06-12T09:53:42 be1fbbac [I] Started POST /
2019-06-12T09:53:42 be1fbbac [E] Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 19727

;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;10.in-addr.arpa. IN SOA

;; TSIG PSEUDOSECTION:

rndc-key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1560347622 300 16 /MdSt/Oun4dvRwiR5FI6aw== 19727 NOERROR 0

2019-06-12T09:53:42 be1fbbac [I] Finished POST / with 400 (31.28 ms)
2019-06-12T09:53:42 be1fbbac [I] Started DELETE /10.158.47.0/mac/00:50:56:90:66:a7
2019-06-12T09:53:42 be1fbbac [I] Finished DELETE /10.158.47.0/mac/00:50:56:90:66:a7 with 200 (20.37 ms)
2019-06-12T09:54:58 be1fbbac [I] Started GET /serverName
2019-06-12T09:54:58 be1fbbac [I] Finished GET /serverName with 200 (0.47 ms)
2019-06-12T09:54:58 be1fbbac [I] Started GET /10.158.47.0/mac/00:50:56:90:66:a7
2019-06-12T09:54:58 be1fbbac [E] No DHCP record for MAC 10.158.47.0/00:50:56:90:66:a7 found
2019-06-12T09:54:58 be1fbbac [I] Finished GET /10.158.47.0/mac/00:50:56:90:66:a7 with 404 (0.8 ms)
2019-06-12T09:54:58 be1fbbac [I] Started GET /10.158.47.0/ip/10.158.47.151
2019-06-12T09:54:58 be1fbbac [E] No DHCP records for IP 10.158.47.0/10.158.47.151 found
2019-06-12T09:54:58 be1fbbac [I] Finished GET /10.158.47.0/ip/10.158.47.151 with 404 (0.66 ms)
2019-06-12T09:55:01 be1fbbac [I] Started GET /serverName
2019-06-12T09:55:01 be1fbbac [I] Finished GET /serverName with 200 (0.4 ms)
2019-06-12T09:55:01 be1fbbac [I] Started POST /10.158.47.0
2019-06-12T09:55:01 be1fbbac [I] Finished POST /10.158.47.0 with 200 (20.08 ms)
2019-06-12T09:55:02 be1fbbac [I] Started POST /
2019-06-12T09:55:02 be1fbbac [E] Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 22766

;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;10.in-addr.arpa. IN SOA

;; TSIG PSEUDOSECTION:

rndc-key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1560347702 300 16 6P1KSOQB40uikKrRkMSnVw== 22766 NOERROR 0

2019-06-12T09:55:02 be1fbbac [I] Finished POST / with 400 (28.43 ms)
2019-06-12T09:55:02 be1fbbac [I] Started DELETE /10.158.47.0/mac/00:50:56:90:71:43
2019-06-12T09:55:02 be1fbbac [I] Finished DELETE /10.158.47.0/mac/00:50:56:90:71:43 with 200 (17.79 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /serverName
2019-06-12T09:55:34 be1fbbac [I] Finished GET /serverName with 200 (0.43 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /10.158.47.0/mac/00:50:56:90:71:43
2019-06-12T09:55:34 be1fbbac [E] No DHCP record for MAC 10.158.47.0/00:50:56:90:71:43 found
2019-06-12T09:55:34 be1fbbac [I] Finished GET /10.158.47.0/mac/00:50:56:90:71:43 with 404 (0.81 ms)
2019-06-12T09:55:34 be1fbbac [I] Started GET /10.158.47.0/ip/10.158.47.151
2019-06-12T09:55:34 be1fbbac [E] No DHCP records for IP 10.158.47.0/10.158.47.151 found
2019-06-12T09:55:34 be1fbbac [I] Finished GET /10.158.47.0/ip/10.158.47.151 with 404 (0.65 ms)
2019-06-12T09:55:38 be1fbbac [I] Started GET /serverName
2019-06-12T09:55:38 be1fbbac [I] Finished GET /serverName with 200 (0.36 ms)
2019-06-12T09:55:38 be1fbbac [I] Started POST /10.158.47.0
2019-06-12T09:55:38 be1fbbac [I] Finished POST /10.158.47.0 with 200 (19.8 ms)
2019-06-12T09:55:38 be1fbbac [I] Started POST /
2019-06-12T09:55:38 be1fbbac [E] Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 37686

;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;10.in-addr.arpa. IN SOA

;; TSIG PSEUDOSECTION:

rndc-key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1560347738 300 16 vDwuGTrXv75bV6soywoeIA== 37686 NOERROR 0

2019-06-12T09:55:38 be1fbbac [I] Finished POST / with 400 (29.03 ms)
2019-06-12T09:55:38 be1fbbac [I] Started DELETE /10.158.47.0/mac/00:50:56:90:70:b4
2019-06-12T09:55:38 be1fbbac [I] Finished DELETE /10.158.47.0/mac/00:50:56:90:70:b4 with 200 (39.71 ms)
2019-06-12T09:56:28 be1fbbac [I] Started GET /serverName
2019-06-12T09:56:28 be1fbbac [I] Finished GET /serverName with 200 (0.46 ms)
2019-06-12T09:56:28 be1fbbac [I] Started GET /10.158.47.0/mac/00:50:56:90:70:b4
2019-06-12T09:56:28 be1fbbac [E] No DHCP record for MAC 10.158.47.0/00:50:56:90:70:b4 found
2019-06-12T09:56:28 be1fbbac [I] Finished GET /10.158.47.0/mac/00:50:56:90:70:b4 with 404 (0.76 ms)
2019-06-12T09:56:28 be1fbbac [I] Started GET /10.158.47.0/ip/10.158.47.151
2019-06-12T09:56:28 be1fbbac [E] No DHCP records for IP 10.158.47.0/10.158.47.151 found
2019-06-12T09:56:28 be1fbbac [I] Finished GET /10.158.47.0/ip/10.158.47.151 with 404 (0.6 ms)
2019-06-12T09:56:31 be1fbbac [I] Started GET /serverName
2019-06-12T09:56:31 be1fbbac [I] Finished GET /serverName with 200 (0.4 ms)
2019-06-12T09:56:31 be1fbbac [I] Started POST /10.158.47.0
2019-06-12T09:56:31 be1fbbac [I] Finished POST /10.158.47.0 with 200 (20.08 ms)
2019-06-12T09:56:31 be1fbbac [I] Started POST /
2019-06-12T09:56:31 be1fbbac [E] Update errors: Answer:

;; ->>HEADER<<- opcode: UPDATE, status: REFUSED, id: 13362

;; flags: qr ra; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1

;; ZONE SECTION:

;10.in-addr.arpa. IN SOA

;; TSIG PSEUDOSECTION:

rndc-key. 0 ANY TSIG hmac-md5.sig-alg.reg.int. 1560347791 300 16 j+jDhz251a975sHWZ/M6tw== 13362 NOERROR 0

2019-06-12T09:56:31 be1fbbac [I] Finished POST / with 400 (28.18 ms)
2019-06-12T09:56:32 be1fbbac [I] Started DELETE /10.158.47.0/mac/00:50:56:90:29:32
2019-06-12T09:56:32 be1fbbac [I] Finished DELETE /10.158.47.0/mac/00:50:56:90:29:32 with 200 (17.51 ms)
[root@centpx02 log]#

[areyus]

Thank you for providing more information.
Yet, I do fear I am unable to help you. It looks like a problem with the dhcpd, but I don’t know that at all.
I have only used Foreman with Infoblox IPAM.
Maybe someone with more dhcpd experience can help.

[fdo_ng]

No… Thanks for assisting… I am trying to understand how/where the reverse dns records been added into Foreman…

[fdo_ng]

At this point i am pretty certain I did not add the new subnet properly. I am trying to follow this document to Foreman :: Adding new subnet for provisioning (Does this apply on the latest version?)

1- Checked the SubNets in Foreman UI

2 - Add the ReverseDNS Zones under /var/named/dynamic:
Under /var/named/dynamic, uses original subnet as reference to create new zone and restarted the bind. Noticed there was no jnl file created for the new subnet I added.

[root@centpx02 dynamic]# ls -lsa
total 40
0 drwxr-x— 2 named named 126 Jun 12 13:31 .
0 drwxrwx–T 5 root named 120 Jun 12 13:31 …
4 -rw-r–r-- 1 named named 486 Jun 11 11:49 db.18.159.10.in-addr.arpa
28 -rw-r–r-- 1 named named 26457 Jun 11 11:37 db.18.159.10.in-addr.arpa.jnl
4 -rw-r–r-- 1 named named 353 Jun 12 13:31 db.47.158.10.in-addr.arpa
4 -rw-r–r-- 1 named named 218 May 24 14:25 db.ashlab.ops

Content of the file:
[root@centpx02 dynamic]# cat db.47.158.10.in-addr.arpa
$ORIGIN .
$TTL 10800 ; 3 hours
47.158.10.in-addr.arpa IN SOA centpx02.ashlab.ops. root.47.158.10.in-addr.arpa. (
89 ; serial
86400 ; refresh (1 day)
3600 ; retry (1 hour)
604800 ; expire (1 week)
3600 ; minimum (1 hour)
)
NS centpx02.ashlab.ops.
$ORIGIN 47.158.10.in-addr.arpa.
$TTL 86400 ; 1 day
[root@centpx02 dynamic]#

Restarted the bind and still not seeing the new zone in the logs
Jun 12 13:43:34 centpx02 named[22887]: command channel listening on 127.0.0.1#953
Jun 12 13:43:34 centpx02 named[22887]: managed-keys-zone: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone 0.in-addr.arpa/IN: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone 18.159.10.in-addr.arpa/IN: loaded serial 89
Jun 12 13:43:34 centpx02 named[22887]: zone 1.0.0.127.in-addr.arpa/IN: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone localhost.localdomain/IN: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone localhost/IN: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa/IN: loaded serial 0
Jun 12 13:43:34 centpx02 named[22887]: zone ashlab.ops/IN: loaded serial 1
Jun 12 13:43:34 centpx02 named[22887]: all zones loaded
Jun 12 13:43:34 centpx02 named[22887]: running
Jun 12 13:43:34 centpx02 systemd: Started Berkeley Internet Name Domain (DNS).

[fdo_ng]

Ok followed the doc on " Adding new subnet for provisioning". Made sure I followed steps properly.

Still getting the error message

Unable to save * Create Reverse IPv4 DNS record for bob-tille.as.ops task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://centpx02.as.ops:8443/dns"

Checked the /var/log/messgae:
“12 14:16:56 centpx02 dynflowd: /opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/daemonize.rb:75: warning: conflicting chdir during another chdir block
Jun 12 14:16:56 centpx02 dynflowd: /opt/theforeman/tfm/root/usr/share/gems/gems/daemons-1.2.3/lib/daemons/daemonize.rb:108: warning: conflicting chdir during another chdir block
Jun 12 14:16:56 centpx02 dynflowd: dynflow_executor: process with pid 26286 started.
Jun 12 14:16:56 centpx02 systemd: Started Foreman jobs daemon.
Jun 12 14:17:53 centpx02 smart-proxy: centpx02.ashlab.ops - - [12/Jun/2019:14:17:53 EDT] “GET /tftp/serverName HTTP/1.1” 200 17
Jun 12 14:17:53 centpx02 smart-proxy: - -> /tftp/serverName
Jun 12 14:17:54 centpx02 smart-proxy: centpx02.ashlab.ops - - [12/Jun/2019:14:17:54 EDT] “POST /dhcp/10.158.47.0 HTTP/1.1” 200 0
Jun 12 14:17:54 centpx02 smart-proxy: - -> /dhcp/10.158.47.0
Jun 12 14:17:54 centpx02 named[25169]: client 127.0.0.1#47526/key rndc-key: update ‘10.IN-ADDR.ARPA/IN’ denied
Jun 12 14:17:54 centpx02 smart-proxy: update failed: REFUSED
Jun 12 14:17:54 centpx02 smart-proxy: centpx02.as.ops - - [12/Jun/2019:14:17:54 EDT] “POST /dns/ HTTP/1.1” 400 329
Jun 12 14:17:54 centpx02 smart-proxy: - -> /dns/
Jun 12 14:17:54 centpx02 smart-proxy: centpx02.as.ops - - [12/Jun/2019:14:17:54 EDT] “DELETE /dhcp/10.158.47.0/mac/00:50:56:90:5d:e6 HTTP/1.1” 200 0
Jun 12 14:17:54 centpx02 smart-proxy: - -> /dhcp/10.158.47.0/mac/00:50:56:90:5d:e6
Jun 12 14:29:17 centpx02 named[25169]: client 127.0.0.1#54566/key rndc-key: update ‘10.IN-ADDR.ARPA/IN’ denied
Jun 12 14:29:17 centpx02 smart-proxy: update failed: REFUSED
Jun 12 14:29:17 centpx02 smart-proxy: centpx02.asops - - [12/Jun/2019:14:29:17 EDT] “POST /dns/ HTTP/1.1” 400 328
Jun 12 14:29:17 centpx02 smart-proxy: - -> /dns/”

Anyone know why i am getting denied???

[fdo_ng]

Alright - I was able to get past that error create DNS Record error message. I skipped the step to edit the /etc/zones.conf because file did not exist. Then noticed it was located /etc/named/zones.conf once that was updated the error message was gone.

But now during pxe boot it can’t get the IP Address…

Question is how does will the vm pickup the pxe-ipAddress if Foreman is not physically on the same network?/

[areyus]

Nice to hear you managed to get the DHCPD problem out of the way.
Afaik, most L3 switches (or at least the more professional ones) have an option to foreward DHCP broadcasts to your DHCP server. That needs to be configured if your servers are seperated on L3 from your foreman server. How to do that and how it’s called probably depends on your network manufacturer (our network department calls those “IP helper” entries).

Regards

[fdo_ng]

Awesome, I that was the missing piece of my puzzle. Thanks everyone.