Fact uploads succeed, but throw error 500 "Password hash is not included in the list"

Problem:

On puppetserver, compilation and fact uploads seem to sort of work BUT I see the following in my logs for all catalog compilations :

018-07-18 10:35:30,718 INFO  [qtp771881950-61] [puppetserver] Puppet 'replace_catalog' command for XXXXX submitted to PuppetDB with UUID c6767455-5420-4711-b8cf-e2596704074c
2018-07-18 10:36:34,136 WARN  [qtp771881950-92] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During the fact upload the server responded with: 500 Internal Server Error. Error is ignored and the execution continues.
{
  "error": {"message":"Validation failed: Password hash is not included in the list"}
}

In the foreman server, production logs, I see a whole stack trace :

2018-07-18 10:41:08 537fcc1f [app] [I] Import facts for 'wn268.datagrid.cea.fr' completed. Added: 0, Updated: 23, Deleted 0 facts
2018-07-18 10:41:08 537fcc1f [app] [W] Action failed
 | ActiveRecord::RecordInvalid: Validation failed: Password hash is not included in the list
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:78:in `raise_validation_error'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/validations.rb:50:in `save!'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/attribute_methods/dirty.rb:43:in `save!'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `block in save!'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:384:in `block in with_transaction_returning_status'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `block in transaction'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:194:in `block in within_new_transaction'
 | /opt/rh/rh-ruby24/root/usr/share/ruby/monitor.rb:214:in `mon_synchronize'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/transaction.rb:191:in `within_new_transaction'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/connection_adapters/abstract/database_statements.rb:235:in `transaction'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:210:in `transaction'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:381:in `with_transaction_returning_status'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/transactions.rb:313:in `save!'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/suppressor.rb:46:in `save!'
 | /usr/share/foreman/app/services/puppet_fact_parser.rb:57:in `operatingsystem'
 | /usr/share/foreman/app/models/host/base.rb:183:in `block in set_non_empty_values'
 | /usr/share/foreman/app/models/host/base.rb:182:in `each'
 | /usr/share/foreman/app/models/host/base.rb:182:in `set_non_empty_values'
 | /usr/share/foreman/app/models/host/base.rb:177:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/host/managed.rb:372:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/concerns/puppet_host_extensions.rb:3:in `populate_fields_from_facts'
 | /usr/share/foreman/app/models/host/base.rb:157:in `parse_facts'
 | /usr/share/foreman/app/models/host/base.rb:146:in `import_facts'
 | /usr/share/foreman/app/models/host/managed.rb:298:in `import_facts'
 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:286:in `facts'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/api/v2/base_controller.rb:152:in `disable_json_root'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.4.1/lib/audited/sweeper.rb:14:in `around'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:16:in `block in <class:Constraints>'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:46:in `serve'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/static_dispatcher.rb:65:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/extractor/recorder.rb:136:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 | /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/conditional_get.rb:38:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
 | /usr/share/foreman/lib/middleware/session_safe_logging.rb:17:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 | /usr/share/foreman/lib/middleware/tagged_logging.rb:18:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-3.4.1/lib/secure_headers/middleware.rb:12:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
 | /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop'
 | /usr/share/gems/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context'
2018-07-18 10:41:08 537fcc1f [app] [I]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2018-07-18 10:41:08 537fcc1f [app] [I]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (0.6ms)
2018-07-18 10:41:08 537fcc1f [app] [I] Completed 500 Internal Server Error in 1232ms (Views: 2.0ms | ActiveRecord: 104.4ms)

and … I can’t find how to fix that. Looks like there’s a password hash missing somewhere, but where ?
Any idea ?

Expected outcome:

fact upload without stack traces everywhere

Foreman and Proxy versions:

1.17

Foreman and Proxy plugin versions:

Other relevant data:

Using foreman puppet modules, foreman 1.17 ruby ENC script.

Can you check the facter output on the host and what value it has for the operating system password_hash? Operating systems must use one of the following for the password hash: "SHA256", "SHA512", "Base64", "MD5".

Hi,

Welll… it’s empty :

# facter -p password_hash

#

Note sure where this should come from… ?
I will check what I can find in the mean time about that fact…

Regards

hmm, that’s interesting, checking locally it seems i also don’t have this fact, which I think should be fine. Perhaps you can try manually creating the operating system in Foreman first and then it will work correctly?

Hi,

Well, I won’t be able to re-create the OSes as I have numerous machines using them, and I believe foreman will refuse to delete an OS which is used by at least one host.

It is true that the OS (CentOS 7.5) was auto-created after hosts were upgraded, and I checked the foreman password hash settings for all my OSes and they are set to SHA256.

I’ll try to play with that setting and will report here if I find anything interesting, but I’m not even sure the error comes from something missing in the DB or in the uploaded facts, so I’m afraid I could search a solution for long :confused:

This doesn’t seem to be blocking, fortunately :]

Mmmh… Interesting.

Editing the OSes, changing the password hash to SHA512, then going back and setting back SHA256 seems to fix the issue.