Foreman all hosts out of sync

Problem: All existing hosts in foreman are suddenly not syncing anymore. No changes to host groups , such as setting Smart Variables and Class Parameters are working. By not working I mean, those changes are not applied to the hosts infrastructure when running puppet agent -t, but are seen in the web gui.

Expected outcome: Apply catalog states to puppet agents as usual.

Foreman and Proxy versions: 1.22.0-1

Foreman and Proxy plugin versions:

Distribution and version: Distributor ID: Ubuntu
Description: Ubuntu 18.04.2 LTS
Release: 18.04
Codename: bionic

Other relevant data: The issue occurred when we restarted the machine. After the reboot , hosts are out of sync. Strange thing is that logs are silent on the matter and activity is as if the service is working.


2021-06-15T09:08:33 [D|dyn|] Executor heartbeat
2021-06-15T09:08:48 [D|dyn|] Executor heartbeat
2021-06-15T09:09:03 [D|dyn|] Executor heartbeat
2021-06-15T09:28:08 [I|app|10dbae07] Started GET "/notification_recipients" for 
1.2.3.4 at 2021-06-15 09:28:08 +0000
2021-06-15T09:28:08 [I|app|10dbae07] Processing by NotificationRecipientsController#index as JSON
2021-06-15T09:28:08 [I|app|10dbae07] Current user set to user.1(admin)
2021-06-15T09:28:08 [D|app|10dbae07] Current location set to none
2021-06-15T09:28:08 [D|app|10dbae07] Current organization set to none
2021-06-15T09:28:08 [D|not|10dbae07] Cache Miss: notification, writing cache for notification-8
2021-06-15T09:28:08 [D|not|10dbae07] Expiring notification cache notification-8 in 3600 seconds
2021-06-15T09:28:08 [D|app|10dbae07] Body: {"notifications":[]}
2021-06-15T09:28:08 [I|app|10dbae07] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 3.7ms)
2021-06-15T09:30:06 [I|dyn|] start terminating throttle_limiter...
2021-06-15T09:30:06 [I|dyn|] start terminating client dispatcher...
2021-06-15T09:30:06 [I|dyn|] stop listening for new events...
2021-06-15T09:30:06 [I|dyn|] start terminating clock...
2021-06-15T09:30:06 [D|app|] Current user set to foreman_admin (admin)
2021-06-15T09:30:06 [I|dyn|] start terminating throttle_limiter...
2021-06-15T09:30:06 [I|dyn|] start terminating client dispatcher...
2021-06-15T09:30:06 [I|dyn|] stop listening for new events...
2021-06-15T09:30:06 [I|dyn|] start terminating clock...
2021-06-15T09:40:05 [I|app|76b96192] Started GET "/notification_recipients" for 1.2.3.5 at 2021-06-15 09:40:05 +0000
2021-06-15T09:40:05 [I|app|76b96192] Processing by NotificationRecipientsController#index as JSON
2021-06-15T09:40:05 [I|app|76b96192] Current user set to ivan.iliev (admin)
2021-06-15T09:40:05 [D|app|76b96192] Current location set to none
2021-06-15T09:40:05 [D|app|76b96192] Current organization set to none
2021-06-15T09:40:05 [D|not|76b96192] Cache Miss: notification, writing cache for notification-5
2021-06-15T09:40:05 [D|not|76b96192] Expiring notification cache notification-5 in 3600 seconds
2021-06-15T09:40:05 [D|app|76b96192] Body: {"notifications":[]}
2021-06-15T09:40:05 [I|app|76b96192] Completed 200 OK in 11ms (Views: 0.1ms | ActiveRecord: 3.8ms)
2021-06-15T09:40:06 [I|app|e2841c42] Started GET "/config_reports?search=eventful+%3D+true" for 1.2.3.4 at 2021-06-15 09:40:06 +0000
2021-06-15T09:40:06 [I|app|e2841c42] Processing by ConfigReportsController#index as HTML
2021-06-15T09:40:06 [I|app|e2841c42]   Parameters: {"search"=>"eventful = true"}
2021-06-15T09:40:06 [I|app|e2841c42] Current user set to ivan.iliev (admin)
2021-06-15T09:40:06 [D|app|e2841c42] Current location set to none
2021-06-15T09:40:06 [D|app|e2841c42] Current organization set to none
2021-06-15T09:40:06 [I|app|e2841c42]   Rendering config_reports/index.html.erb within layouts/application
2021-06-15T09:40:06 [I|app|e2841c42]   Rendered common/_pagination.html.erb (0.2ms)
2021-06-15T09:40:06 [I|app|e2841c42]   Rendered config_reports/_list.html.erb (8.8ms)
2021-06-15T09:40:06 [I|app|e2841c42]   Rendered config_reports/index.html.erb within layouts/application (9.4ms)
2021-06-15T09:40:06 [I|app|e2841c42]   Rendered common/_searchbar.html.erb (0.2ms)

Hello,

that feels similar to some other reports we’ve seen recently. Perhaps the node.rb script has some issues with SSL. Check logs at /var/log/puppetlabs/puppetserver/ and see if there’s something interesting. It coud be that your certificates expired. Also, Foreman 1.22 is very old and generally not supported, is there a chance you could update it?

Hi @Marek_Hulan , thanks for your reply.

We actually found that the problem was caused by a custom module we have that overwrites the master and main sections in the puppet.conf on the master server.

That in turn messed up the master config and agents were not getting the right stuff.

On a side note, can I ask for permission to edit my initial post of this thread to edit out some of the internal content that was pasted ?

Thanks !

1 Like