Puppet agents failing after Foreman 3 upgrade

Problem:
Puppet run on any client after Foreman 3 upgrade results in

Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node slurm-login01.svi.edu.au: Failed to find slurm-login01.svi.edu.au via exec: Execution of '/etc/puppetlabs/puppet/node.rb slurm-login01.svi.edu.au' returned 1:

Error logs from Forman production log

ActionController::RoutingError (No route matches [GET] "/node/<node-name>"):

Expected outcome:
Puppet runs to succeed

Foreman and Proxy versions:
3.0

Foreman and Proxy plugin versions:
3.0

Distribution and version:
Ubuntu 18
Puppet agent 7.11
Puppet server 7.4

I took a snapshot and rolling back was fine. Happy to do any testing suggested.

How did you upgrade? Did you not install the foreman_puppet plugin on upgrade?

Thanks for the message. I installed it after the fact when I noticed things not working. Wrong order obviously. I’ve installed it now and puppet runs are working.

I encouter the same error with all my puppet agents after upgrading to 3.0.1, using 2 servers with foreman role only and 2 servers with puppetca/puppet proxy roles.
I added --enable-foreman-plugin-puppet to my foreman-installer parameters after rpms upgrade to upgrade the stack.
rubygem-foreman_puppet-1.0.4-1.fm3_0.el8.noarch rpm is installed.
It looks like my problem comes from generating host yaml data, puppet yaml button on each host gives me this : Unable to generate output, Check log files.
In the logs i can see this :
2021-11-04T13:42:28 [W|app|e5187b55] Failed to generate external nodes for xxxx.xxxx.fr
2021-11-04T13:42:28 [I|app|e5187b55] Backtrace for ‘Failed to generate external nodes for xxxx.xxxx.fr’ error (TypeError): no implicit conversion of nil into Array
e5187b55 | /usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in +' e5187b55 | /usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/relation/delegation.rb:87:in +’
e5187b55 | /usr/share/gems/gems/foreman_puppet-1.0.4/app/services/foreman_puppet/host_info_providers/config_groups_info.rb:7:in host_info' e5187b55 | /usr/share/foreman/app/models/concerns/host_info_extensions.rb:27:in block in info’
e5187b55 | /usr/share/foreman/app/models/concerns/host_info_extensions.rb:25:in each' e5187b55 | /usr/share/foreman/app/models/concerns/host_info_extensions.rb:25:in info’
e5187b55 | /usr/share/gems/gems/foreman_puppet-1.0.4/app/controllers/concerns/foreman_puppet/extensions/hosts_controller_extensions.rb:100:in block in externalNodes' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/mime_responds.rb:205:in respond_to’
e5187b55 | /usr/share/gems/gems/foreman_puppet-1.0.4/app/controllers/concerns/foreman_puppet/extensions/hosts_controller_extensions.rb:96:in externalNodes' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:195:in process_action' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rendering.rb:30:in process_action’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:42:in block in process_action' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:112:in block in run_callbacks’
e5187b55 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks’
e5187b55 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks’
e5187b55 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks’
e5187b55 | /usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks’
e5187b55 | /usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:121:in block in run_callbacks’
e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:139:in run_callbacks' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/callbacks.rb:41:in process_action’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/rescue.rb:22:in process_action' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:33:in block in process_action’
e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in block in instrument' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in instrument’
e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications.rb:180:in instrument' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/instrumentation.rb:32:in process_action’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal/params_wrapper.rb:245:in process_action' e5187b55 | /usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/railties/controller_runtime.rb:27:in process_action’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/abstract_controller/base.rb:136:in process' e5187b55 | /usr/share/gems/gems/actionview-6.0.3.7/lib/action_view/rendering.rb:39:in process’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:190:in dispatch' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_controller/metal.rb:254:in dispatch’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:50:in dispatch' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:33:in serve’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in block in serve' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in each’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in serve' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in call’
e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in call' e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in public_send’
e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in method_missing' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:19:in block in class:Constraints
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/mapper.rb:48:in serve' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:49:in block in serve’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in each' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/journey/router.rb:32:in serve’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/routing/route_set.rb:834:in call' e5187b55 | /usr/share/gems/gems/apipie-dsl-2.4.0/lib/apipie_dsl/static_dispatcher.rb:67:in call’
e5187b55 | /usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call' e5187b55 | /usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call’
e5187b55 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call' e5187b55 | /usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call’
e5187b55 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call' e5187b55 | /usr/share/gems/gems/prometheus-client-1.0.0/lib/prometheus/middleware/exporter.rb:32:in call’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call' e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in call' e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/http/content_security_policy.rb:18:in call' e5187b55 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context' e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/cookies.rb:648:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’
e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:101:in run_callbacks' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/callbacks.rb:26:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/show_exceptions.rb:33:in call' e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:37:in call_app’
e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/rack/logger.rb:28:in call' e5187b55 | /usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call’
e5187b55 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/remote_ip.rb:81:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/request_id.rb:27:in call' e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call' e5187b55 | /usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call’
e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/executor.rb:14:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/static.rb:126:in call’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call' e5187b55 | /usr/share/gems/gems/actionpack-6.0.3.7/lib/action_dispatch/middleware/host_authorization.rb:76:in call’
e5187b55 | /usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call' e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/engine.rb:527:in call’
e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in public_send' e5187b55 | /usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in method_missing’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call' e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each’
e5187b55 | /usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call' e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/configuration.rb:249:in call’
e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:77:in block in handle_request' e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:338:in with_force_shutdown’
e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/request.rb:76:in handle_request' e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/server.rb:438:in process_client’
e5187b55 | /usr/share/gems/gems/puma-5.3.2/lib/puma/thread_pool.rb:145:in block in spawn_thread' e5187b55 | /usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’