foreman-nightly-rpm-pipeline 1124 failed

Foreman RPM nightly pipeline failed:

https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/1124/

foreman-pipeline-foreman-nightly-centos7-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-stream-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-install (failed) (remote job)
foreman-pipeline-foreman-nightly-centos7-upgrade (failed) (remote job)
foreman-pipeline-foreman-nightly-centos8-upgrade (failed) (remote job)

It looks like Puppet doesn’t really work. It appears to assign the class (at least no error is returned) and Puppet runs, but then it doesn’t apply the class so /etc/motd isn’t updated.

It seems the puppet server couldn’t get the ENC from Foreman

puppetserver.log:

2021-07-19T22:16:43.649Z WARN  [qtp2135865030-35] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: Error retrieving node pipe-foreman-server-nightly-centos7.n62.example.com: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.

2021-07-19T22:16:43.711Z ERROR [qtp2135865030-35] [puppetserver] Puppet Server Error: Failed to find pipe-foreman-server-nightly-centos7.n62.example.com via exec: Execution of '/etc/puppetlabs/puppet/node.rb pipe-foreman-server-nightly-centos7.n62.example.com' returned 1: 
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:222:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:138:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:54:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1809:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:65:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'
2021-07-19T22:16:54.328Z WARN  [qtp2135865030-35] [puppetserver] Puppet The node parameter 'fqdn' for node 'pipe-foreman-server-nightly-centos7.n62.example.com' was already set to 'pipe-foreman-server-nightly-centos7.n62.example.com'. It could not be set to 'pipe-foreman-server-nightly-centos7.n62.example.com'
2021-07-19T22:16:54.329Z WARN  [qtp2135865030-35] [puppetserver] Puppet The node parameter 'hostname' for node 'pipe-foreman-server-nightly-centos7.n62.example.com' was already set to 'pipe-foreman-server-nightly-centos7'. It could not be set to 'pipe-foreman-server-nightly-centos7'

I don’t see the foreman.log in the sosreport, are we bundling it too somewhere? Is there a way to access that if it’s not in sosreport?

ignore me, I see the Foreman production.log now, investigating

From production.log:

2021-07-19T22:09:11 [W|app|] ForemanPuppet: skipping engine hook (PG::UndefinedTable: ERROR:  relation "hosts" does not exist
 | LINE 8:  WHERE a.attrelid = '"hosts"'::regclass
 |                             ^
 | )
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:65:in `async_exec'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:65:in `block (2 levels) in query'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:64:in `block in query'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:722:in `block (2 levels) in log'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:721:in `block in log'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract_adapter.rb:712:in `log'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql/database_statements.rb:63:in `query'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:820:in `column_definitions'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/abstract/schema_statements.rb:114:in `columns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/schema_cache.rb:76:in `columns'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/schema_cache.rb:82:in `columns_hash'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/model_schema.rb:488:in `load_schema!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/attributes.rb:247:in `load_schema!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/attribute_decorators.rb:50:in `load_schema!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/model_schema.rb:478:in `block in load_schema'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/model_schema.rb:475:in `synchronize'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/model_schema.rb:475:in `load_schema'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.7/lib/active_record/model_schema.rb:352:in `columns_hash'
 | /usr/share/foreman/app/services/graphql_attribute.rb:15:in `required?'
 | /usr/share/foreman/app/graphql/types/base_object.rb:54:in `nullable?'
 | /usr/share/foreman/app/graphql/types/base_object.rb:9:in `field'
 | /usr/share/foreman/app/graphql/types/base_object.rb:14:in `timestamps'
 | /usr/share/foreman/app/graphql/types/host.rb:8:in `<class:Host>'
 | /usr/share/foreman/app/graphql/types/host.rb:2:in `<module:Types>'
 | /usr/share/foreman/app/graphql/types/host.rb:1:in `<top (required)>'
 | /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
 | /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:411:in `block in require_or_load'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `block in load_interlock'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:14:in `block in loading'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/concurrency/share_lock.rb:151:in `exclusive'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies/interlock.rb:13:in `loading'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:40:in `load_interlock'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:389:in `require_or_load'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:544:in `load_missing_constant'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:214:in `const_missing'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_puppet-1.0.0.rc.1/lib/foreman_puppet/engine.rb:96:in `block in <class:Engine>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `instance_exec'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:428:in `block in make_lambda'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:605:in `block (2 levels) in default_terminator'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `catch'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:604:in `block in default_terminator'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:201:in `block in halting'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `block in invoke_before'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:513:in `invoke_before'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/callbacks.rb:134:in `run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/reloader.rb:88:in `prepare!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/application/finisher.rb:113:in `block in <module:Finisher>'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `instance_exec'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:32:in `run'
 | /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
 | /usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
 | /usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:61:in `block in run_initializers'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:228:in `block in tsort_each'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:431:in `each_strongly_connected_component_from'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:349:in `block in each_strongly_connected_component'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:347:in `each'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:347:in `call'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:347:in `each_strongly_connected_component'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:226:in `tsort_each'
 | /opt/rh/rh-ruby27/root/usr/share/ruby/tsort.rb:205:in `tsort_each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/initializable.rb:60:in `run_initializers'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/application.rb:363:in `initialize!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `public_send'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/railtie.rb:190:in `method_missing'
 | /usr/share/foreman/config/environment.rb:5:in `<top (required)>'
 | /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
 | /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in `require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `block in require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in `load_dependency'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in `require'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/application.rb:339:in `require_environment!'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.7/lib/rails/application.rb:523:in `block in run_tasks_blocks'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `block in execute'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `each'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:281:in `execute'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:241:in `each'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:241:in `invoke_prerequisites'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:243:in `block in invoke_prerequisites'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:241:in `each'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:241:in `invoke_prerequisites'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:218:in `block in invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `synchronize'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:199:in `invoke_with_call_chain'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/task.rb:188:in `invoke'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:160:in `invoke_task'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `each'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:116:in `block in top_level'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:125:in `run_with_threads'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:110:in `top_level'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:83:in `block in run'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in `standard_exception_handling'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in `run'
 | /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
 | /opt/rh/rh-ruby27/root/usr/bin/rake:23:in `load'
 | /opt/rh/rh-ruby27/root/usr/bin/rake:23:in `<main>'

I wouldn’t expect to see that. It looks like it needs DB access somewhere. I’m not sure which rake task that is.

Is it triggering on this line? 96 is empty

Another thing:

2021-07-19T22:19:14 [W|api|b3e3e842] /api/v2/environments API endpoints are deprecated, please use /foreman_puppet/api/v2/environments instead

That’s probably from Hammer.

So I spun up a box here, and could reproduce it.

The problem seems to be that while our tests execute hammer host update --puppet-class-ids $id --name $(hostname -f), the values are correct and hammer answers Host updated., the puppet class is not actually assigned.

Assigning the class in the WebUI or via the API (curl -XPUT -d'{"host":{"puppetclass_ids":[1]}}' -H 'Content-Type: application/json' -u admin:changeme -k https://localhost/api/hosts/1) seems to work fine, and a Puppet run afterwards does apply the motd module.

And indeed, if you run hammer --debug with the above parameters, the resulting update call is actually empty:

[ INFO 2021-07-22T08:41:35 HammerCLIForeman::Host::UpdateCommand] Called with options: {"option_puppetclass_ids"=>[1], "option_volume_list"=>[], "option_interface_list"=>[], "option_name"=>"pipe-foreman-server-nightly-centos7.yatsu.example.com", "option_id"=>1}
[DEBUG 2021-07-22T08:41:35 HammerCLIForeman::CommandExtensions::UpdateCommon] Called block for HammerCLIForeman::UpdateCommand request params:
        #<Proc:0x00000000027d84b0 /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-2.6.0.pre.develop/lib/hammer_cli_foreman/command_extensions/update_common.rb:8>
[ INFO 2021-07-22T08:41:35 API] Server: https://pipe-foreman-server-nightly-centos7.yatsu.example.com
[ INFO 2021-07-22T08:41:35 API] PUT /api/hosts/1
[DEBUG 2021-07-22T08:41:35 API] Params: {
    "host" => {
        "name" => "pipe-foreman-server-nightly-centos7.yatsu.example.com"
    }
}

I don’t know hammer well enough to understand why it stopped translating this, but maybe someone wiser than me can chime in here.

FWIW, I did find GitHub - theforeman/hammer-cli-foreman-puppet but didn’t manage to install it on a nightly box, as the commands clash with the ones from hammer_cli_foreman and I really have no clue how to make them not clash.

FWIW2:

on a working 2.5 install, the same hammer call generates:

[ INFO 2021-07-22T12:09:04 HammerCLIForeman::Host::UpdateCommand] Called with options: {"option_puppetclass_ids"=>[1], "option_volume_list"=>[], "option_interface_list"=>[], "option_name"=>"centos7-foreman-2-5.yatsu.example.com", "option_id"=>1}
[DEBUG 2021-07-22T12:09:04 HammerCLIForeman::CommandExtensions::UpdateCommon] Called block for HammerCLIForeman::UpdateCommand request params:
	#<Proc:0x000000000281b6c0 /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-2.5.1/lib/hammer_cli_foreman/command_extensions/update_common.rb:8>
[ INFO 2021-07-22T12:09:04 API] Server: https://centos7-foreman-2-5.yatsu.example.com
[ INFO 2021-07-22T12:09:04 API] PUT /api/hosts/1
[DEBUG 2021-07-22T12:09:04 API] Params: {
    "host" => {
                   "name" => "centos7-foreman-2-5.yatsu.example.com",
        "puppetclass_ids" => [
            [0] 1
        ]
    }
}

The winner gets 3.0 points, and ows both @Marek_Hulan and me 3.0 pints of beer each.

foreman_puppet has a nice check to only be overriding things when it’s actually needed: ForemanPuppet.extracted_from_core?:

As you can see there, the checked version is “3.0”, but our develop branch is still versioned 2.6. That means that this method returns false. And then the API override does not get applied in

Which means the apidoc is missing the puppetclasses_ids parameter when creating/updating hosts (and hostgroups). And well, if something is not in the apidoc, hammer ignores it. And so would FAM.

So we either need to revert the extraction, or bump to 3.0 with red nightlies and hope it works. (Given it’s the only failing test, I do expect it to become green then)

Let’s bump to 3.0:

Bump!

It’s been bumped and foreman-nightly-rpm-pipeline #1127 [Jenkins] was green – besides the fact that web01 ran out of diskspace (I’m fixing this now)

And foreman-nightly-rpm-pipeline #1128 [Jenkins] was fully green now.

Debian still needs building, and Katello needs an adjustment of its deps (Update Katello requirement on Foreman by evgeni · Pull Request #6934 · theforeman/foreman-packaging · GitHub), but then we’re good