Hi all,
We are using foreman 1.15.6 and recently noticed an issue with the “Rebuild Configs” workflow. We wanted to recreate the default “local boot” pxeconfig for a group of machines. We selected many nodes in foreman hosts UI, selected “rebuild configs”. The pxeconfig files were updated correctly, but we then noticed the dns entries for all of these hosts had vanished!
Watching logs on the dns smart proxy, it definitely indicated it had deleted the dns records but did not re-add.
We did a “rebuild configs” on 1 host at a time and the dns records were added back. If we ran rebuild configs again on the same host, the dns records would be deleted.
Has anyone else seen or experienced something similar to this?
We run BIND DNS, dhcpd DHCP, and tftpd all on linux (Centos 7)
Example of bad dns proxy (notice all DELETES)
D, [2018-02-05T12:13:56.118188 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:13:56.118977 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:13:56.119328 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:13:56.128431 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:13:56.128647 ] DEBUG -- : nsupdate: executed - update delete mr-hadp-data-bis-prod-192.cloud.prod A
I, [2018-02-05T12:13:56.145565 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:13:56 -0600] "DELETE /dns/mr-hadp-data-bis-prod-192.cloud.prod/A HTTP/1.1" 200 - 0.0268
D, [2018-02-05T12:13:56.146558 ] DEBUG -- : close: 172.28.63.13:60281
D, [2018-02-05T12:13:56.157195 ] DEBUG -- : accept: 172.28.63.13:60282
D, [2018-02-05T12:13:56.159092 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:13:56.159592 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:13:56.159829 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:13:56.168462 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:13:56.168661 ] DEBUG -- : nsupdate: executed - update delete 192.58.28.172.in-addr.arpa PTR
I, [2018-02-05T12:13:56.183773 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:13:56 -0600] "DELETE /dns/192.58.28.172.in-addr.arpa HTTP/1.1" 200 - 0.0243
D, [2018-02-05T12:13:56.184685 ] DEBUG -- : close: 172.28.63.13:60282
D, [2018-02-05T12:13:56.197565 ] DEBUG -- : accept: 172.28.63.13:60283
D, [2018-02-05T12:13:56.199722 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:13:56.200312 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:13:56.200579 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:13:56.208609 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:13:56.208804 ] DEBUG -- : nsupdate: executed - update delete mr-hadp-data-bis-prod-192bmc.cloud.prod A
I, [2018-02-05T12:13:56.224427 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:13:56 -0600] "DELETE /dns/mr-hadp-data-bis-prod-192bmc.cloud.prod/A HTTP/1.1" 200 - 0.0242
D, [2018-02-05T12:13:56.225247 ] DEBUG -- : close: 172.28.63.13:60283
D, [2018-02-05T12:13:56.235357 ] DEBUG -- : accept: 172.28.63.13:60284
D, [2018-02-05T12:13:56.237172 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:13:56.237742 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:13:56.237986 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:13:56.246137 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:13:56.246313 ] DEBUG -- : nsupdate: executed - update delete 239.28.28.172.in-addr.arpa PTR
I, [2018-02-05T12:13:56.261952 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:13:56 -0600] "DELETE /dns/239.28.28.172.in-addr.arpa HTTP/1.1" 200 - 0.0243
D, [2018-02-05T12:13:56.262952 ] DEBUG -- : close: 172.28.63.13:60284
And a “good” run a few seconds later on the same host
D, [2018-02-05T12:14:20.835944 ] DEBUG -- : accept: 172.28.63.13:60293
D, [2018-02-05T12:14:20.838267 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:20.838951 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:20.839192 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:20.847561 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:20.847742 ] DEBUG -- : nsupdate: executed - update delete mr-hadp-data-bis-prod-192.cloud.prod A
I, [2018-02-05T12:14:20.861110 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:20 -0600] "DELETE /dns/mr-hadp-data-bis-prod-192.cloud.prod/A HTTP/1.1" 200 - 0.0223
D, [2018-02-05T12:14:20.862045 ] DEBUG -- : close: 172.28.63.13:60293
D, [2018-02-05T12:14:20.873891 ] DEBUG -- : accept: 172.28.63.13:60294
D, [2018-02-05T12:14:20.875509 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:20.876236 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:20.878007 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:20.885363 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:20.885540 ] DEBUG -- : nsupdate: executed - update add mr-hadp-data-bis-prod-192.cloud.prod. 360 A 172.28.58.192
I, [2018-02-05T12:14:20.899821 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:20 -0600] "POST /dns/ HTTP/1.1" 200 - 0.0239
D, [2018-02-05T12:14:20.900736 ] DEBUG -- : close: 172.28.63.13:60294
D, [2018-02-05T12:14:20.911745 ] DEBUG -- : accept: 172.28.63.13:60295
D, [2018-02-05T12:14:20.913359 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:20.913853 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:20.914070 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:20.921242 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:20.921403 ] DEBUG -- : nsupdate: executed - update delete 192.58.28.172.in-addr.arpa PTR
I, [2018-02-05T12:14:20.933163 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:20 -0600] "DELETE /dns/192.58.28.172.in-addr.arpa HTTP/1.1" 200 - 0.0194
D, [2018-02-05T12:14:20.933883 ] DEBUG -- : close: 172.28.63.13:60295
D, [2018-02-05T12:14:20.944310 ] DEBUG -- : accept: 172.28.63.13:60296
D, [2018-02-05T12:14:20.945911 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:20.946562 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:20.948101 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:20.955868 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:20.956045 ] DEBUG -- : nsupdate: executed - update add 192.58.28.172.in-addr.arpa. 360 PTR mr-hadp-data-bis-prod-192.cloud.prod
I, [2018-02-05T12:14:20.971622 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:20 -0600] "POST /dns/ HTTP/1.1" 200 - 0.0252
D, [2018-02-05T12:14:20.972564 ] DEBUG -- : close: 172.28.63.13:60296
D, [2018-02-05T12:14:20.985029 ] DEBUG -- : accept: 172.28.63.13:60297
D, [2018-02-05T12:14:20.987139 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:20.987794 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:20.988048 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:20.997252 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:20.997483 ] DEBUG -- : nsupdate: executed - update delete mr-hadp-data-bis-prod-192bmc.cloud.prod A
I, [2018-02-05T12:14:21.012760 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:21 -0600] "DELETE /dns/mr-hadp-data-bis-prod-192bmc.cloud.prod/A HTTP/1.1" 200 - 0.0251
D, [2018-02-05T12:14:21.013807 ] DEBUG -- : close: 172.28.63.13:60297
D, [2018-02-05T12:14:21.026999 ] DEBUG -- : accept: 172.28.63.13:60298
D, [2018-02-05T12:14:21.028712 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:21.029331 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:21.030833 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:21.038137 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:21.038309 ] DEBUG -- : nsupdate: executed - update add mr-hadp-data-bis-prod-192bmc.cloud.prod. 360 A 172.28.28.239
I, [2018-02-05T12:14:21.052558 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:21 -0600] "POST /dns/ HTTP/1.1" 200 - 0.0234
D, [2018-02-05T12:14:21.053849 ] DEBUG -- : close: 172.28.63.13:60298
D, [2018-02-05T12:14:21.063972 ] DEBUG -- : accept: 172.28.63.13:60299
D, [2018-02-05T12:14:21.065862 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:21.066548 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:21.066753 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:21.073894 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:21.074060 ] DEBUG -- : nsupdate: executed - update delete 239.28.28.172.in-addr.arpa PTR
I, [2018-02-05T12:14:21.086965 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:21 -0600] "DELETE /dns/239.28.28.172.in-addr.arpa HTTP/1.1" 200 - 0.0205
D, [2018-02-05T12:14:21.087905 ] DEBUG -- : close: 172.28.63.13:60299
D, [2018-02-05T12:14:21.098345 ] DEBUG -- : accept: 172.28.63.13:60300
D, [2018-02-05T12:14:21.100487 ] DEBUG -- : Rack::Handler::WEBrick is invoked.
D, [2018-02-05T12:14:21.101316 ] DEBUG -- : require_ssl_client_verification: skipping, non-HTTPS request
D, [2018-02-05T12:14:21.103062 ] DEBUG -- : running /usr/bin/nsupdate
D, [2018-02-05T12:14:21.111573 ] DEBUG -- : nsupdate: executed - server localhost
D, [2018-02-05T12:14:21.111787 ] DEBUG -- : nsupdate: executed - update add 239.28.28.172.in-addr.arpa. 360 PTR mr-hadp-data-bis-prod-192bmc.cloud.prod
I, [2018-02-05T12:14:21.125823 ] INFO -- : 172.28.63.13 - - [05/Feb/2018:12:14:21 -0600] "POST /dns/ HTTP/1.1" 200 - 0.0248
D, [2018-02-05T12:14:21.126565 ] DEBUG -- : close: 172.28.63.13:60300
Thanks!
Andrew