Virtual machines showing as Out of sync in foreman console

We have around 800 Azure VMs hosted in foreman (v 2.5.0), out of which only few machines are showing as out of sync in foreman console. On further check, it turns out that VMs are correctly reporting at puppet server ( updated logs are there in /var/log/puppetlabs/puppetserver/puppetserver.log corresponding to each out of sync VM ). But some how foreman failed to pickup the status. When I did a grep for out of sync VM in /var/log/foreman/production.log, I managed to pull below error corresponding to each out of sync VM

2021-06-14T00:37:57 [I|app|a1038b97] Backtrace for ‘Action failed’ error (ActiveRecord::AssociationTypeMismatch): Host::Managed(#81661260) expected, got #<Host::Base id: 193, name: “xxx.domain.com”, last_compile: nil, last_report: nil, updated_at: “2021-05-17 12:50:28”, created_at: “2021-05-17 12:50:25”, root_pass: nil, architecture_id: 1, operatingsystem_id: 2, environment_id: 1, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 1, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: 1, managed: false, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: “xxx.domain.com”, image_id: nil, organization_id: 1, location_id: 2, type: nil, otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: “”, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil> which is an instance of Host::Base(#77610880)

Any idea why only few machines are showing this behavior while majority are reporting correctly.

It would be interesting how such hosts were created. It seems your DB lists them as Host::Base which is an abstract class, that should never be instantiated. Could you take a look at such host audits? What were the last operations with them? How do you generally create hosts?

The fix is to update the hosts SQL table, there’s a type column. Everywhere you see the Host::Base, modify that to Host::Managed. That should make it work again, but of course doesn’t resolve the cause.