GUI shows out of sync, but agent is running

Support
1

Hi,
I just installed foreman/puppet and everything works. Except 1 thing: 3 hosts are out of sync for more than 24 hours (the other 18 hosts are “green”), but the agent is running. I can start “puppet agent -t” and all changes are applied. But there is no report.
When starting with --debug this is the output at the end:

Notice: Applied catalog in 0.37 seconds
Debug: Resolving service 'report' using Puppet::HTTP::Resolver::Settings
Debug: Using cached connection for https://puppet.domain.com:8140
Debug: Caching connection for https://puppet.domain.com:8140
Debug: Resolved service 'report' to https://puppet.domain.com:8140/puppet/v3
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: Puppet::Network::Format[rich_data_msgpack]: feature msgpack is missing
Debug: report supports formats: json pson yaml
Debug: Using cached connection for https://puppet.domain.com:8140
Debug: HTTP PUT https://puppet.domain.com:8140/puppet/v3/report/webgaby.domain.com?environment=production returned 200 OK
Debug: Caching connection for https://puppet.domain.com:8140
Debug: Closing connection for https://puppet.domain.com:8140

on the puppet-server I can see this in /var/log/foreman/production.log:

2023-02-02T11:38:38 [I|app|e3860694] Started POST "/api/config_reports" for 192.168.0.9 at 2023-02-02 11:38:38 +0100
2023-02-02T11:38:38 [I|app|e3860694] Processing by Api::V2::ConfigReportsController#create as JSON
2023-02-02T11:38:38 [I|app|e3860694]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-02-02T11:38:38 [W|app|e3860694] Action failed
2023-02-02T11:38:38 [I|app|e3860694] Backtrace for 'Action failed' error (ActiveRecord::AssociationTypeMismatch): Host::Managed(#60120) expected, got #<Host::Base id: 7, name: "webgaby.domain.com", last_comp
ile: nil, last_report: nil, updated_at: "2023-01-31 14:46:08.787971000 +0000", created_at: "2023-01-31 14:46:06.429227000 +0000", root_pass: nil, architecture_id: 1, operatingsystem_id: 3, ptable_id: nil, m
edium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 2, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: 1, managed: false, use_image: nil,
 image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: "webgaby.domain.com", image_id: nil, organization_id: 1, location_id: 2, type: nil, otp: nil, realm_id: nil, compute_profi
le_id: nil, provision_method: nil, grub_pass: nil, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil> which is an instance of Host::Base(#60040)
 e3860694 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/associations/association.rb:283:in `raise_on_type_mismatch!'
 e3860694 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/associations/belongs_to_association.rb:77:in `replace'
 e3860694 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/associations/singular_association.rb:17:in `writer'
 e3860694 | /usr/share/gems/gems/activerecord-6.1.7/lib/active_record/associations/builder/association.rb:111:in `host='

192.168.0.9 is my puppet-host,
172.16.0.9 is one of the 3 agents

Foreman and Proxy versions:
foreman-release-3.5.1-1.el8.noarch
puppetserver-7.9.3-1.el8.noarch

Agent Version:
puppet-agent-7.21.0-1.el8.x86_64

Agent OS-Releases:
Oracle Linux Server release 8.7
Oracle Linux Server release 7.9

Distribution and version:
Oracle Linux Server release 8.7

2

From the logs you provided, it looks like the hosts might be in the “unmanaged” state.
When you go to the “all hosts” page and search for managed = false, are the affected hosts listed there? If so, the hosts are disabled for management and Foreman cannot import the reports the hosts for that reason.
In case this is unintentional (which I assume), you can go to the edit page of each host and click “manage host”, which should solve that problem.

1 Like
3

Damn, that was too easy… now it works

thank your for your help :slight_smile:

1 Like
4

I don’t think that’s really necessary. Almost all of our hosts are “managed = false”, except those few which I have provisioned recently via foreman. 250 “unmanaged” servers send puppet reports without problems…

If it fixed the problem to set them managed by guess would be that it will continue to work if you “unmanaged” them again. The change of manage state may have set something else to make it work…

1 Like
5

It worked for a moment, and after some time it was “out of sync” again. I digged deeper into the postgres-database and found the problem: The 3 hosts with the problem were 2 times in the hosts-table. I deleted the host in the webgui and cleaned up the database. After this it was possible to readd them again:

DELETE FROM fact_values WHERE  host_id='7';
DELETE FROM host_puppet_facets WHERE  host_id='7';
DELETE FROM host_facets_reported_data_facets WHERE  host_id='7';
DELETE FROM nics WHERE  host_id='7';
DELETE FROM reports WHERE  host_id='7';
DELETE FROM host_status WHERE  host_id='7';
DELETE FROM hosts WHERE id='7';

This topic helped me: Reports Processing Failure (AssociationTypeMismatch)