Using Template-Variables in Smart-Class Parameters

jdelker · May 8, 2020, 1:25pm

Problem:
I’m following the documentation about how to use template variables in parameter default values.
In particular, I’m using the Puppet-Module-Classes of “puppetlabs/apt” with a override on the “source” attribute to set a default value.
To adjust to the particular distribution, I would like to utilize the “lsbdistcodename” fact in that value.
According to the docs, it should work by using template expressions like “<%= … %>”.
For the above fact, that should then be “<%= @host.facts[‘lsbdistcodename’] %>”.
However, as soon I save that smart-parameter definition, the host config fails to render.
The UI shows “Unable to generate output, Check log files” when selection the host’s YAML Button.

Note: I also tried other - more simple - template expression like “<%= @host.name %>” to rule out any other effects - to no avail.

Expected outcome:

Properly resolved fact value for “<%= … %>”

Foreman and Proxy versions:
1.24.2

Foreman and Proxy plugin versions:
1.24.2

Distribution and version:
Debian Buster
(using packages from https://deb.theforeman.org/)

tbrisker · May 10, 2020, 9:17am

What do you see in /var/log/foreman/production.log when you attempt to render the YAML for the said host? that might give a hint as to what failed.

jdelker · May 24, 2020, 6:10pm

Not very much. This app appears upon rendering the host’s YAML:

2020-05-24T20:06:58 [I|app|79301d33] Started GET "/hosts/my.host/externalNodes?name=my.host" for 10.8.0.30 at 2020-05-24 20:06:58 +0200
2020-05-24T20:06:58 [I|app|79301d33] Processing by HostsController#externalNodes as HTML
2020-05-24T20:06:58 [I|app|79301d33]   Parameters: {"name"=>"my.host", "id"=>"my.host"}
2020-05-24T20:06:58 [W|app|79301d33] No SSL cert with CN supplied - request from 10.8.0.30,
2020-05-24T20:06:58 [W|app|79301d33] Failed to generate external nodes for my.host
2020-05-24T20:06:58 [I|app|79301d33]   Rendering text template
2020-05-24T20:06:58 [I|app|79301d33]   Rendered text template (0.0ms)
2020-05-24T20:06:58 [I|app|79301d33] Completed 412 Precondition Failed in 260ms (Views: 0.7ms | ActiveRecord: 27.4ms)

tbrisker · May 25, 2020, 6:48am

can you change the log level to debug (https://theforeman.org/manuals/latest/index.html#7.2Debugging) and try again?

jdelker · May 25, 2020, 7:44pm

Enabling the debug logging, shows this:

2020-05-25T21:31:36 [I|app|edea31d3] Started GET "/hosts/my.host/externalNodes?name=my.host" for 10.8.0.30 at 2020-05-25 21:31:36 +0200
2020-05-25T21:31:36 [I|app|edea31d3] Processing by HostsController#externalNodes as HTML
2020-05-25T21:31:36 [I|app|edea31d3]   Parameters: {"name"=>"my.host", "id"=>"my.host"}
2020-05-25T21:31:36 [W|app|edea31d3] No SSL cert with CN supplied - request from 10.8.0.30,
2020-05-25T21:31:36 [W|app|edea31d3] Failed to generate external nodes for my.host
2020-05-25T21:31:36 [D|app|edea31d3] Backtrace for 'Failed to generate external nodes for my.host' error (TypeError): no implicit conversion of String into Hash
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/core_ext/hash/deep_merge.rb:24:in `merge!'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/core_ext/hash/deep_merge.rb:24:in `deep_merge!'
/usr/share/foreman/app/services/classification/values_hash_query.rb:160:in `block in update_hash_matcher'
/usr/share/foreman/app/services/classification/values_hash_query.rb:155:in `reverse_each'
/usr/share/foreman/app/services/classification/values_hash_query.rb:155:in `update_hash_matcher'
/usr/share/foreman/app/services/classification/values_hash_query.rb:52:in `merged_value'
/usr/share/foreman/app/services/classification/values_hash_query.rb:38:in `calculate_value'
/usr/share/foreman/app/services/classification/values_hash_query.rb:21:in `block in values_hash'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
/usr/share/foreman/app/services/classification/values_hash_query.rb:20:in `values_hash'
/usr/share/foreman/app/models/concerns/classification.rb:5:in `block (2 levels) in <module:Classification>'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation.rb:288:in `instance_exec'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation.rb:288:in `_exec_scope'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/scoping/named.rb:186:in `block in scope'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:67:in `block in values_hash'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation.rb:281:in `scoping'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:67:in `values_hash'
/usr/share/foreman/app/models/host_info_providers/puppet_info.rb:21:in `puppetclass_parameters'
/usr/share/foreman/app/models/host_info_providers/puppet_info.rb:87:in `classes_info_hash'
/usr/share/foreman/app/models/host_info_providers/puppet_info.rb:11:in `host_info'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:29:in `block in info'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:27:in `each'
/usr/share/foreman/app/models/concerns/host_info_extensions.rb:27:in `info'
/usr/share/foreman/app/controllers/hosts_controller.rb:199:in `block in externalNodes'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/mime_responds.rb:197:in `respond_to'
/usr/share/foreman/app/controllers/hosts_controller.rb:195:in `externalNodes'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/tempfile_reaper.rb:17:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/etag.rb:27:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/conditional_get.rb:27:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/head.rb:14:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/session/abstract/id.rb:277:in `context'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/session/abstract/id.rb:271:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/runtime.rb:24:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/sendfile.rb:113:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/secure_headers-6.2.0/lib/secure_headers/middleware.rb:11:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/usr/share/foreman/vendor/ruby/2.5.0/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/urlmap.rb:77:in `block in call'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/urlmap.rb:61:in `each'
/usr/share/foreman/vendor/ruby/2.5.0/gems/rack-2.1.1/lib/rack/urlmap.rb:61:in `call'
/usr/lib/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:152:in `accept_and_process_next_request'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:113:in `main_loop'
/usr/lib/ruby/vendor_ruby/phusion_passenger/request_handler.rb:416:in `block (3 levels) in start_threads'
/usr/lib/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
/usr/share/foreman/vendor/ruby/2.5.0/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-05-25T21:31:36 [I|app|edea31d3]   Rendering text template
2020-05-25T21:31:36 [I|app|edea31d3]   Rendered text template (0.0ms)
2020-05-25T21:31:36 [I|app|edea31d3] Completed 412 Precondition Failed in 208ms (Views: 0.6ms | ActiveRecord: 40.6ms)

I’m currently using a value of

repo-<%= @host.name %>:
  location: http://my.repo/repos
  repos: main

for an override of the “sources” param of puppetlabs/apt class.

As stated above, I actually would rather like to use a facts value like “lsbdistcodename” or “os:distro:codename”, but as nothing of that worked out so far, I started with using “@host.name” to proof the general functionality of variable replacement.

tbrisker · May 26, 2020, 7:40am

This looks like you’re hitting Bug #19980: Error in ENC YAML generation when merge enabled and variable or macro is present in parameters - Foreman - it seems erb in parameters doesn’t play well with the “merge overrides” option.

jdelker · May 26, 2020, 2:56pm

Hmm… I’m somewhat discuraged when looking at the bug history and accompanied pull request.
That issue seems to exist for already more than 2 years, and it doesn’t look like this will be fixed any time soon.
Which is quite strange, because it is declared to be “easy” and a working pull request seems to be available…

Is it just me doing something really strange in foreman, or is there an obvious workaround available, which I haven’t found out yet? Otherwise, I would expext that many would be affected by this…!?

tbrisker · May 27, 2020, 9:51am

Thanks for the feedback and bringing it to our attention!

I believe the reason this wasn’t fixed is a combination of two causes:

While the original reporter thought this was an easy fix, there were some concerns raised during the PR review that indicated it might be more complicated to fix properly for all use cases. That is also the reason the PR was never merged. I also tried taking a quick jab at it yesterday, and it looks like properly fixing might require some more significant refactoring of the classification service’s code.
Considering there have only been two upvotes for the issue in 3 years, it appears that the combination of using ERB in parameters and the “merge override” option is not a very common use-case. As there weren’t many users complaining about the issue, it did not get highly prioritized for fixing.

I will try to find some time to dig into fixing this issue properly, but I can’t promise how long it would take.

jdelker · May 28, 2020, 8:45pm

Thanks for giving some insight and looking into this.I appreciate it.

My feedback on the few upvotes:
Although I would consider myself quite engaged to analyse and research such issues, I actually failed to find that bug you referenced myself, even after searching for a resolution intensively.
Now, that may be afterall my personal inability, but maybe others fail with this also.
I wouldn’t give to much weight on the votes for this.