Problem:
On puppetserver, compilation and fact uploads seem to sort of work BUT I see the following in my logs for all catalog compilations :
018-07-18 10:35:30,718 INFO [qtp771881950-61] [puppetserver] Puppet 'replace_catalog' command for XXXXX submitted to PuppetDB with UUID c6767455-5420-4711-b8cf-e2596704074c
2018-07-18 10:36:34,136 WARN [qtp771881950-92] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During the fact upload the server responded with: 500 Internal Server Error. Error is ignored and the execution continues.
{
"error": {"message":"Validation failed: Password hash is not included in the list"}
}
In the foreman server, production logs, I see a whole stack trace :
2018-07-18 10:41:08 537fcc1f [app] [I] Import facts for 'wn268.datagrid.cea.fr' completed. Added: 0, Updated: 23, Deleted 0 facts
2018-07-18 10:41:08 537fcc1f [app] [W] Action failed
| ActiveRecord::RecordInvalid: Validation failed: Password hash is not included in the list
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:78:in `raise_validation_error'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:50:in `save!'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/attribute_methods/dirty.rb:43:in `save!'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `block in save!'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
| /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `save!'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/suppressor.rb:46:in `save!'
| /usr/share/foreman/app/services/puppet_fact_parser.rb:57:in `operatingsystem'
| /usr/share/foreman/app/models/host/base.rb:183:in `block in set_non_empty_values'
| /usr/share/foreman/app/models/host/base.rb:182:in `each'
| /usr/share/foreman/app/models/host/base.rb:182:in `set_non_empty_values'
| /usr/share/foreman/app/models/host/base.rb:177:in `populate_fields_from_facts'
| /usr/share/foreman/app/models/host/managed.rb:372:in `populate_fields_from_facts'
| /usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts'
| /usr/share/foreman/app/models/host/base.rb:157:in `parse_facts'
| /usr/share/foreman/app/models/host/base.rb:146:in `import_facts'
| /usr/share/foreman/app/models/host/managed.rb:298:in `import_facts'
| /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:286:in `facts'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/api/v2/base_controller.rb:152:in `disable_json_root'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:16:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:46:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/extractor/recorder.rb:136:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
| /usr/share/foreman/lib/middleware/session_safe_logging.rb:17:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /usr/share/foreman/lib/middleware/tagged_logging.rb:18:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-3.4.1/lib/secure_headers/middleware.rb:12:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
| /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
| /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
| /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
| /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2018-07-18 10:41:08 537fcc1f [app] [I] Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2018-07-18 10:41:08 537fcc1f [app] [I] Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.6ms)
2018-07-18 10:41:08 537fcc1f [app] [I] Completed 500 Internal Server Error in 1232ms (Views: 2.0ms | ActiveRecord: 104.4ms)
and … I can’t find how to fix that. Looks like there’s a password hash missing somewhere, but where ?
Any idea ?
Expected outcome:
fact upload without stack traces everywhere
Foreman and Proxy versions:
1.17
Foreman and Proxy plugin versions:
Other relevant data:
Using foreman puppet modules, foreman 1.17 ruby ENC script.