Puppet fact parser is failing with String does not have #dig method

Support
1

Problem:

We recently upgraded Foreman from 1.18 to 1.24 and the fact parsing seems to be partially broken. I’m able to clear the facts from a host a force a puppet run to ingest them back into the DB but getting this error:

2021-04-13T16:01:46 [W|app|c92c0a3d] Action failed
2021-04-13T16:01:46 [D|app|c92c0a3d] Backtrace for 'Action failed' error (TypeError): String does not have #dig method
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/hash_with_indifferent_access.rb:192:in `dig'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/hash_with_indifferent_access.rb:192:in `dig'

This came to my attention when I was updating the default hostgroup plugin. It seems that this plugin will fail if certain facts are not parsed properly.

My concern is that I’m still using an older version of puppet (3.8.6) and facter (2.4.6) for both clients and servers. We have 1000s of systems and it will be awhile before we can migrate all the systems over to a new puppet version.

Expected outcome:

would like the puppet facter to parse properly

Foreman and Proxy versions:

Foreman: 1.24

Foreman and Proxy plugin versions:

Default Host Group 5.0.0

Distribution and version:

CentOS 7.9

Other relevant data:

2021-04-13T16:01:46 [W|app|c92c0a3d] Action failed
2021-04-13T16:01:46 [D|app|c92c0a3d] Backtrace for 'Action failed' error (TypeError): String does not have #dig method
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/hash_with_indifferent_access.rb:192:in `dig'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/hash_with_indifferent_access.rb:192:in `dig'
 | /usr/share/foreman/app/services/puppet_fact_parser.rb:189:in `os_name'
 | /usr/share/foreman/app/services/puppet_fact_parser.rb:51:in `architecture'
 | /usr/share/foreman/app/models/host/base.rb:199:in `block in set_non_empty_values'
 | /usr/share/foreman/app/models/host/base.rb:198:in `each'
 | /usr/share/foreman/app/models/host/base.rb:198:in `set_non_empty_values'
 | /usr/share/foreman/app/models/host/base.rb:193:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/host/managed.rb:416:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/host/base.rb:168:in `block in parse_facts'
 | /usr/share/foreman/app/services/foreman/telemetry_helper.rb:27:in `telemetry_duration_histogram'
 | /usr/share/foreman/app/models/host/base.rb:167:in `parse_facts'
 | /usr/share/foreman/app/models/host/base.rb:153:in `import_facts'
 | /usr/share/foreman/app/models/host/managed.rb:330:in `import_facts'
 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:308:in `facts'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
 | /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
 | /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-04-13T16:01:46 [I|app|c92c0a3d]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2021-04-13T16:01:46 [I|app|c92c0a3d]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.3ms)
2021-04-13T16:01:46 [I|app|c92c0a3d] Completed 500 Internal Server Error in 204ms (Views: 0.7ms | ActiveRecord: 54.9ms)
2

That is a bug, our unit tests are poor in this regard but @dmatoulek greatly improved them in Fixes #28099 - Adding total_disk_size to host facets by domitea · Pull Request #8401 · theforeman/foreman · GitHub few days ago and I am wondering if he could also add 2.x version of facter fixture and ensure it works and never breaks from now on :slight_smile:

Would you mind creating an issue for this problem? Thanks

3

TIL we are considering dropping facter 2.X: Refactor #27906: Use modern Facter 3 facts - Foreman

I am not sure if we communicated this properly tho. @ekohl @Marek_Hulan

4

That weird. It looks like it’s this line:

Do you by any chance have a custom os fact that is a string? What makes me wonder this is:

[1] pry(main)> {os: 'bla'}.dig(:os, :name)
TypeError: String does not have #dig method
from (pry):1:in `dig'

Note that Facter 2.4 already has a structured os fact so I really think your custom fact is overlapping with a built in fact.

Now it is probably safe to say we shouldn’t fail with such an exception so that’s at least worth a bug report.

5

We have several custom facts but none that specifically targets the OS. I actually removed all the custom facts as a test but was still able to reproduce the issue so I don’t think there is an overlap. Is there something you would like me to try?

@lzap are you wanting me to submit something else or was that request for someone else?

6

Could you check the facts it tries to upload? I don’t recall the exact path, but it should store those as YAML files on the server. You can also upload them by calling node.rb --push-facts [CERTNAME] to reproduce fact pushing.

7

this is the fact value for os:

os: "{\x22name\x22=>\x22CentOS\x22, \x22family\x22=>\x22RedHat\x22, \x22release\x22=>{\x22major\x22=>\x227\x22, \x22minor\x22=>\x226\x22, \x22full\x22=>\x227.6.1810\x22}, \x22lsb\x22=>{\x22distcodename\x22=>\x22Core\x22, \x22distid\x22=>\x22CentOS\x22, \x22distdescription\x22=>\x22CentOS Linux release 7.6.1810 (Core) \x22, \x22release\x22=>\x22:core-4.1-amd64:core-4.1-noarch\x22, \x22distrelease\x22=>\x227.6.1810\x22, \x22majdistrelease\x22=>\x227\x22, \x22minordistrelease\x22=>\x226\x22}}"
8

@ekohl is there more information you needed? Did you need more facter data? Let me know and I will upload whatever you need.

9

That does looks like it stringifies the fact and that’s why it’s failing to parse. There was a setting stringify_facts back in 3.x. IIRC in 3.x it defaults to true and perhaps you can set that to false to resolve it.

10

I’m still using 2.x so how could I set that? where would I make that change?

11

It’s a setting in puppet.conf

12

@ekohl that seems to have done the trick. I just updated the puppet.conf on all the hosts and it was able to parse the facts again. Thanks guys.

13

Allright so it is not an issue in Foreman, however I suggest if we would add a fixture for puppet 2.x to the test suite so we know it works until we are ready to deprecate it.

14

I think we have fixtures for Facter 2.x but none with stringify_facts=true. However, that only exists in Puppet < 4 and we’ve officially dropped support for Puppet < 5. We haven’t removed code so on the Foreman side it’ll still work, you just need an older Foreman Proxy for Puppet 3. The question is how far we really want to go with this. It’s understandable that it’s hard to migrate a large Puppet infrastructure with 100s or 1000s of hosts, but Puppet 3 has been EOL for 4 years.

Right now I’d recommend @jgray to consider upgrading their Foreman itself to 2.4 and use a separate server only the Puppet 3 server and a Foreman Proxy that still supports that version. The API between Foreman and Foreman Proxy is unchanged (at least registration + Puppet) so it’ll still work and at least the Foreman itself would be on a supported version.

15

Okay, I am more than fine with dropping Puppet 2.X facts completely. Discovery was the last blocker and it was upgraded in 2.4 so we are good.

1 Like