Reports Processing Failure (AssociationTypeMismatch)

Problem:

The ‘Last Report’ field is blank for a number of hosts. Upon further investigation the debug logs show the following (see other relevant data’). The hosts have been deleted within Foreman, the certificates revoked/cleaned via ‘pupperserver ca’ and the Puppet Agent reinstalled on the offending machines to no available. Your help would be appreciated.

Expected outcome:
The report(s) are processed and the ‘Last Report’ field is updated.

Foreman and Proxy versions:

  • foreman-dynflow-sidekiq-2.1.4-1.el8.noarch
  • foreman-release-2.1.4-1.el8.noarch
  • rubygem-foreman_maintain-0.6.9-1.el8.noarch
  • foreman-2.1.4-1.el8.noarch
  • foreman-postgresql-2.1.4-1.el8.noarch
  • foreman-debug-2.1.4-1.el8.noarch
  • foreman-selinux-2.1.4-1.el8.noarch
  • foreman-installer-2.1.4-1.el8.noarch
  • foreman-service-2.1.4-1.el8.noarch
  • foreman-proxy-2.1.4-1.el8.noarch
  • foreman-cli-2.1.4-1.el8.noarch

Distribution and version:

  • CentOS Linux release 8.2.2004 (Core)

Other relevant data:

2020-11-20T12:21:17 [D|app|4902a0b8] Backtrace for 'Action failed' error (ActiveRecord::AssociationTypeMismatch): Host::Managed(#47222304019520) expected, got #<Host::Base id: 49, name: "myserver.hostname.tld", last_compile: nil, last_report: nil, updated_at: "2020-11-03 16:45:54", created_at: "2020-11-03 16:45:41", root_pass: nil, architecture_id: 1, operatingsystem_id: 7, environment_id: 13, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 10, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: 2, managed: false, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 2, certname: "myserver.hostname.tld", image_id: nil, organization_id: nil, location_id: nil, type: nil, otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: "", global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil> which is an instance of Host::Base(#47222306019100)
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/associations/association.rb:283:in `raise_on_type_mismatch!'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/associations/belongs_to_association.rb:66:in `replace'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/associations/singular_association.rb:17:in `writer'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/associations/builder/association.rb:108:in `host='
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:50:in `public_send'
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:50:in `_assign_attribute'
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:43:in `block in _assign_attributes'
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:42:in `each'
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:42:in `_assign_attributes'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/attribute_assignment.rb:21:in `_assign_attributes'
/usr/share/gems/gems/activemodel-6.0.3.1/lib/active_model/attribute_assignment.rb:35:in `assign_attributes'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/core.rb:325:in `initialize'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/inheritance.rb:70:in `new'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/inheritance.rb:70:in `new'
/usr/share/foreman/app/models/concerns/foreman/sti.rb:19:in `new'
/usr/share/foreman/app/services/report_importer.rb:154:in `create_report_and_logs'
/usr/share/foreman/app/services/config_report_importer.rb:13:in `create_report_and_logs'
/usr/share/foreman/app/services/report_importer.rb:41:in `block in import'
/usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
/usr/share/foreman/app/services/report_importer.rb:40:in `import'
/usr/share/foreman/app/services/report_importer.rb:22:in `import'
/usr/share/foreman/app/models/config_report.rb:27:in `import'
/usr/share/foreman/app/controllers/api/v2/config_reports_controller.rb:42:in `create'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `block in instrument'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in `instrument'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in `process_action'
/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in `process'
/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in `process'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in `dispatch'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in `dispatch'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in `block in serve'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `each'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in `serve'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in `call'
/usr/share/gems/gems/apipie-dsl-2.2.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/tempfile_reaper.rb:15:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/etag.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/conditional_get.rb:40:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:266:in `context'
/usr/share/gems/gems/rack-2.2.2/lib/rack/session/abstract/id.rb:260:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in `run_callbacks'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in `call_app'
/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in `call'
/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/method_override.rb:24:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/runtime.rb:22:in `call'
/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in `call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/sendfile.rb:110:in `call'
/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in `call'
/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in `call'
/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `public_send'
/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.2/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in `call'
/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in `handle_request'
/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in `process_client'
/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in `block in run'
/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in `block in spawn_thread'
/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-11-20T12:21:17 [I|app|4902a0b8]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2020-11-20T12:21:17 [I|app|4902a0b8]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 0.3ms | Allocations: 67)
2020-11-20T12:21:17 [I|app|4902a0b8] Completed 500 Internal Server Error in 94ms (Views: 1.0ms | ActiveRecord: 16.1ms | Allocations: 81665)
1 Like

Hello, we have the same issue. Did you find a solution ?

Thanks

Unfortunately not.

Did this ever work for you? Was it after an upgrade?

It was a brand new deployment and the issue only impacted approximately 10% of the estate. I spent a couple of days on it but no matter what I did, I couldn’t even identify the cause never-mind a solution.

For some reason the object is created as Host::Base (abstract class) instead of Host::Managed. I have no idea what the case may be, however, is there a chance you’re using foreman-discovery plugin? (it’s not listed in the packages but still…) Also Foreman 2.1 seems quite old, could you update to the more recent version first? 2.4 was just released. Looking at the code, this could only happen (at least with 2.4) if an existing Host::Base object is found in the DB when the report arives. Perhaps you can try running foreman-rake console and try Host::Base.find_by_name('the.fqdn.of.the.host'). Then put the result here, if it says Host::Base and not Host::Managed, that’s the problem.

I only found a 6 years old report of the similar problem when used with salt plugin - Bug #10243: Error in processing salt reports - Salt - Foreman

1 Like

Yep, the machines that I’m having issues with are showing Host::Base. Upgraded to the latest version and confirmed that we don’t have any plugins. No joy.

=> #<Host::Base id: 32, name: “redacted.host.tld”, last_compile: nil, last_report: nil, updated_at: “2020-11-03 16:24:12”, created_at: “2020-11-03 16:24:05”, root_pass: nil, architecture_id: 1, operatingsystem_id: 7, environment_id: 13, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 9, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: 2, managed: false, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 2, certname: “redacted.host.tld”, image_id: nil, organization_id: nil, location_id: nil, type: nil, otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: “”, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil>

Ok, it would be interesting to find out from logs, how such record got created. The easiest way to fix it is to delete them. For that, use the following command Host::Base.find_by_name('the.fqdn.of.the.host').destroy

if you want to keep these hosts, let me know. When I’m at the computer, I’d test a command to convert all Base hosts to Managed.

1 Like

I’ve got the same problem with a newly built foreman 2.4. About 20 of the 300 hosts are not showing reports and are getting the same error as above, but when I try to delete the host via foreman-rake, I get the following error.

irb(main):002:0> Host::Base.find_by_name(‘deviprtweb001.sdlc.nzrb.local’).destroy
Traceback (most recent call last):
2: from lib/tasks/console.rake:5:in `block in <top (required)>’
1: from (irb):2
ActiveRecord::InvalidForeignKey (PG::ForeignKeyViolation: ERROR: update or delete on table “hosts” violates foreign key constraint “fk_rails_a82ac722f1” on table “host_facets_reported_data_facets”)
DETAIL: Key (id)=(64) is still referenced from table “host_facets_reported_data_facets”.
irb(main):003:0>

What is interesting is I would expect Foreman to accept reports for any (unmanaged) hosts, not just managed hosts. Yet the association is only for “Host::Managed” type. This is weird.

Anyway, we need to figure out what is causing these abstract hosts to be created.

There’s a big difference between Host::Managed#managed that can be true/false, which is what defines whether host is managed or unamanged versus Host::Base and Host::Managed classes. The first is abstract class that should never be initiated and clearly we have some bug that allows it to be created.

Oh, this means there’s a relation that’s not automatically cleaned up upon deletion (again because it’s the wrong class). You need to first delete that. Try this

HostFacets::ReportedDataFacet.find_by_host_id(123).destroy where 123 is the ID of the host you’re trying to delete.

I don’t know if this will help anyone in the future, but I found this post with the same “AssociationTypeMismatch” error. (Used search feature in this site.)

I have a fresh Foreman 3.1 on Debian 11, installed on Christmas 2021. (It’s my home lab, not for work.)
I have one Ubuntu 14.04 server that is connected to that Foreman instance and it was not registering any reports and throwing this error in the logs:

Backtrace for ‘Action failed’ error (ActiveRecord::AssociationTypeMismatch): Host::Managed(#67160) expected, got #<Host::Base id: 6, name: “hostname.domain.tld”, last_compile: nil, last_report: nil, updated_at: “2021-12-25 16:07:00”, created_at: “2021-12-25 16:06:57”, root_pass: nil, architecture_id: 1, operatingsystem_id: 3, ptable_id: nil, medium_id: nil, build: false, comment: nil, disk: nil, installed_at: nil, model_id: 2, hostgroup_id: nil, owner_id: nil, owner_type: nil, enabled: true, puppet_ca_proxy_id: 1, managed: false, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: “hostname.domain.tld”, image_id: nil, organization_id: nil, location_id: nil, type: nil, otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil, grub_pass: “”, global_status: 0, lookup_value_matcher: [FILTERED], pxe_loader: nil, initiated_at: nil, build_errors: nil> which is an instance of Host::Base(#67080)

Of particular note, I saw “organization_id: nil, location_id: nil”, even though I usually assign all my hosts to an organization and a location in Foreman. I checked, and apparently I’d missed that step this time.

Once I assigned an organization and a location to the host in Foreman, I ran “puppet apply --agent” on the Ubuntu 14.04 machine and it started reporting hosts and the error went away.

I’m not knowledgeable enough to understand why, but that was enough for me and maybe this will help someone in the future(?)

Edit: Put the error in a blockquote

Hi!

Did anyone in the meantime figure out why this happens or if there is a quicker fix than to delete the affected VM?

The suggestion above about organization and location does not work for us, both are assigned and still the hosts are not reporting.

The issue also seems to be re-occurring for some hosts. We fixed some a couple of weeks ago, but today, after we migrated a bunch of additional VMs to the Foreman, some of the existing ones suddenly stopped reporting.

We are running version 3.1.1 currently.

Cheers,
Sarah

Hi again!

We have done some investigation in the meantime. We noticed that the affected hosts appear twice in the hosts database, once with type Host::Managed and once with empty type.

When we delete the host from the Foreman GUI, only the entry with Host::Managed type gets removed, but the one with empty type stays. This probably explains why the issue reappears for some hosts after a while.

I tried deleting the entries with empty type like described above, but run into another foreign key contraint:

irb(main):004:0> Host::Base.where(id: [354]).destroy_all
Traceback (most recent call last):
        2: from lib/tasks/console.rake:5:in `block in <top (required)>'
        1: from (irb):4
ActiveRecord::InvalidForeignKey (PG::ForeignKeyViolation: ERROR:  update or delete on table "hosts" violates foreign key constraint "fk_rails_cadab260b9" on table "host_puppet_facets")
DETAIL:  Key (id)=(354) is still referenced from table "host_puppet_facets".

@Marek_Hulan can you provide the good command to solve this?

1 Like