Problem:
I have a single foreman installation performing DHCP-proxying (to a remote DHCP server) and IPA proxying (to a remote FreeIPA server). When attempting to add/remove a host, smart-proxy is crashing when attempting to interact with the freeipa server. See error in /var/log/messages below.
Expected outcome:
Host is added/removed from FreeIPA server.
Foreman and Proxy versions:
foreman-3.4.1-1.el8.noarch
foreman-proxy-3.4.1-1.el8.noarch
Foreman and Proxy plugin versions:
rubygem-foreman_ansible-9.0.1-1.fm3_4.el8.noarch
rubygem-foreman_host_reports-1.0.2-3.fm3_3.el8.noarch
rubygem-foreman_maintain-1.2.1-1.el8.noarch
rubygem-foreman_remote_execution-8.0.0-2.fm3_4.el8.noarch
rubygem-foreman-tasks-7.0.0-1.fm3_4.el8.noarch
rubygem-foreman_webhooks-3.0.5-1.fm3_4.el8.noarch
rubygem-hammer_cli_foreman-3.4.0-1.el8.noarch
rubygem-hammer_cli_foreman_ansible-0.4.0-1.fm3_4.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
rubygem-hammer_cli_foreman_webhooks-0.0.3-1.fm3_4.el8.noarch
Distribution and version:
Other relevant data:
Dec 1 13:15:57 foremanserver smart-proxy[1054]: /usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:46: warning: already initialized constant XMLRPC::Config::ENABLE_NIL_PARSER
Dec 1 13:15:57 foremanserver smart-proxy[1054]: /usr/share/gems/gems/xmlrpc-0.3.0/lib/xmlrpc/config.rb:22: warning: previous definition of ENABLE_NIL_PARSER was here
Dec 1 13:15:57 foremanserver smart-proxy[1054]: fd00::ffff:c0a8:3b - - [01/Dec/2022:13:15:57 PST] "DELETE /realm/DOMAIN.COM/host.DOMAIN.COM HTTP/1.1" 404 46
Dec 1 13:15:57 foremanserver smart-proxy[1054]: - -> /realm/DOMAIN.COM/host.DOMAIN.COM
Dec 1 13:16:23 foremanserver smart-proxy[1054]: malloc_consolidate(): unaligned fastbin chunk detected
Dec 1 13:16:23 foremanserver systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Dec 1 13:16:23 foremanserver systemd[1]: Started Process Core Dump (PID 95436/UID 0).
Dec 1 13:16:23 foremanserver systemd-coredump[95437]: Resource limits disable core dumping for process 1054 (smart-proxy).
Dec 1 13:16:23 foremanserver systemd-coredump[95437]: Process 1054 (smart-proxy) of user 992 dumped core.
Dec 1 13:16:23 foremanserver systemd[1]: foreman-proxy.service: Main process exited, code=killed, status=6/ABRT
Dec 1 13:16:23 foremanserver systemd[1]: foreman-proxy.service: Failed with result 'signal'.
The FreeIPA angle is probably a red herring. Here’s a later attempt to add a host that failed, but the IPA server was updated successfully.
Dec 1 14:30:08 foremanserver smart-proxy[10005]: /usr/share/foreman-proxy/modules/realm_freeipa/provider.rb:46: warning: already initialized constant XMLRPC::Config::ENABLE_NIL_PARSER
Dec 1 14:30:08 foremanserver smart-proxy[10005]: /usr/share/gems/gems/xmlrpc-0.3.0/lib/xmlrpc/config.rb:22: warning: previous definition of ENABLE_NIL_PARSER was here
Dec 1 14:30:08 foremanserver smart-proxy[10005]: fd00::ffff:c0a8:3b - - [01/Dec/2022:14:30:08 PST] "POST /realm/DOMAIN.COM/ HTTP/1.1" 200 630
Dec 1 14:30:08 foremanserver smart-proxy[10005]: - -> /realm/DOMAIN.COM/
Dec 1 14:30:13 foremanserver systemd[1]: session-5.scope: Succeeded.
Dec 1 14:30:22 foremanserver smart-proxy[10005]: fd00::ffff:c0a8:3b - - [01/Dec/2022:14:30:22 PST] "GET /tftp/serverName HTTP/1.1" 200 29
Dec 1 14:30:22 foremanserver smart-proxy[10005]: - -> /tftp/serverName
Dec 1 14:30:22 foremanserver smart-proxy[10005]: fd00::ffff:c0a8:3b - - [01/Dec/2022:14:30:22 PST] "GET /tftp/serverName HTTP/1.1" 200 29
Dec 1 14:30:22 foremanserver smart-proxy[10005]: - -> /tftp/serverName
Dec 1 14:30:22 foremanserver smart-proxy[10005]: fd00::ffff:c0a8:3b - - [01/Dec/2022:14:30:22 PST] "GET /tftp/serverName HTTP/1.1" 200 29
Dec 1 14:30:22 foremanserver smart-proxy[10005]: - -> /tftp/serverName
Dec 1 14:30:22 foremanserver smart-proxy[10005]: fd00::ffff:c0a8:3b - - [01/Dec/2022:14:30:22 PST] "GET /tftp/serverName HTTP/1.1" 200 29
Dec 1 14:30:22 foremanserver smart-proxy[10005]: - -> /tftp/serverName
Dec 1 14:30:22 foremanserver smart-proxy[10005]: malloc(): unaligned fastbin chunk detected
Troubleshooting a bit further, the last line in the proxy log indicates it’s an issue with the dhcp proxy:
/var/log/foreman-proxy/proxy.log:
...
2022-12-05T15:53:59 77847195 [I] Started GET /dhcp/192.168.0.0/unused_ip from=192.168.0.50&to=192.168.0.100
2022-12-05T15:54:01 77847195 [I] Finished GET /dhcp/192.168.0.0/unused_ip with 200 (2072.7 ms)
2022-12-05T15:54:16 feec9ec8 [I] Started POST /realm/REALM.NET/
2022-12-05T15:54:16 feec9ec8 [I] Finished POST /realm/REALM.NET/ with 200 (201.69 ms)
2022-12-05T15:54:20 feec9ec8 [I] Started GET /tftp/serverName
2022-12-05T15:54:20 feec9ec8 [I] Finished GET /tftp/serverName with 200 (0.55 ms)
2022-12-05T15:54:20 feec9ec8 [I] Started GET /tftp/serverName
2022-12-05T15:54:20 feec9ec8 [I] Finished GET /tftp/serverName with 200 (0.44 ms)
2022-12-05T15:54:21 feec9ec8 [I] Started GET /tftp/serverName
2022-12-05T15:54:21 feec9ec8 [I] Finished GET /tftp/serverName with 200 (0.57 ms)
2022-12-05T15:54:21 feec9ec8 [I] Started GET /tftp/serverName
2022-12-05T15:54:21 feec9ec8 [I] Finished GET /tftp/serverName with 200 (0.78 ms)
2022-12-05T15:54:21 feec9ec8 [I] Started POST /dhcp/192.168.0.0
Here’s the error when creating the host on the web page:
Create DHCP Settings for anewhost.realm.net task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::ServerBrokeConnection]: Server broke connection) for proxy https://foreman.realm.net:9090/dhcp
I’m using a remote ISC DHCP server. I tried “re-installing” the DHCP feature and restarting the foreman services to no avail:
# foreman-installer --foreman-proxy-dhcp=true \
> --foreman-proxy-dhcp-provider=remote_isc \
> --foreman-proxy-plugin-dhcp-remote-isc-dhcp-config /mnt/nfs/etc/dhcp/dhcpd.conf \
> --foreman-proxy-plugin-dhcp-remote-isc-dhcp-leases /mnt/nfs/var/lib/dhcpd/dhcpd.leases \
> --foreman-proxy-plugin-dhcp-remote-isc-key-name=omapi_key \
> --foreman-proxy-plugin-dhcp-remote-isc-key-secret=<key> \
> --foreman-proxy-plugin-dhcp-remote-isc-omapi-port=7911 \
> --enable-foreman-proxy-plugin-dhcp-remote-isc \
> --foreman-proxy-dhcp-server=dhcpserver.realm.net
There is a core dump after smart-proxy crashes. Would that be of help? If so, does the Foreman team have a secure way of sharing core dumps?
If anyone else runs across this, upgrading to Foreman 3.5.1 and Katello 4.7 (and puppet7) appears to have resolved the issue on my end. Not sure what the root cause was, however.
1 Like
Could it be that this change has something to do with the improved behavior in Foreman 3.5.1 ?
I would be curious to know why the issue happens in the first place in < Foreman 3.5.1 and hopefully someone have some answers.
