Unable to provision hosts due to FIPS ssh key error in Govcloud after 3.4.1 upgrade

Support
#1

Just updated our instance of foreman in a govcloud environment that enforces FIPS mode and I’m seeing an error:

2023-02-06T14:58:22 [I|app|d80f6231] negotiating protocol version
2023-02-06T14:58:22 [I|app|d80f6231] sending KEXINIT
2023-02-06T14:58:22 [I|app|d80f6231] got KEXINIT from server
2023-02-06T14:58:22 [I|app|d80f6231] negotiating algorithms
2023-02-06T14:58:22 [W|app|d80f6231] SSH error
2023-02-06T14:58:22 [I|app|d80f6231] Backtrace for 'SSH error' error (OpenSSL::PKey::DHError): Failed to generate key: non FIPS method

Has anyone ran into this? I saw some earlier threads that talked about this for katello installs, but we are not running that and 3.4.1 looks to have the patches mentioned already applied (and the error format is different)

#2

Full stack:

2023-02-06T14:58:22 [I|app|d80f6231] SSH connection established to 10.99.28.168 - executing template
2023-02-06T14:58:22 [I|app|d80f6231] negotiating protocol version
2023-02-06T14:58:22 [I|app|d80f6231] sending KEXINIT
2023-02-06T14:58:22 [I|app|d80f6231] got KEXINIT from server
2023-02-06T14:58:22 [I|app|d80f6231] negotiating algorithms
2023-02-06T14:58:22 [W|app|d80f6231] Failed to launch script on empty-test-box36-app.gov1.qprod.net: Failed to generate key: non FIPS method
2023-02-06T14:58:22 [I|app|d80f6231] Backtrace for 'Failed to launch script on empty-test-box36-app.gov1.qprod.net: Failed to generate key: non FIPS method' error (OpenSSL::PKey::DHError): Failed to generate key: non FIPS method
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:125:in `generate_key!'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:125:in `generate_key'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/kex/diffie_hellman_group1_sha1.rb:51:in `initialize'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/algorithms.rb:373:in `new'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/algorithms.rb:373:in `exchange_keys'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/algorithms.rb:201:in `proceed!'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/algorithms.rb:149:in `accept_kexinit'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:210:in `block in poll_message'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:188:in `loop'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:188:in `poll_message'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:225:in `block in wait'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:223:in `loop'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:223:in `wait'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh/transport/session.rb:88:in `initialize'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `new'
 d80f6231 | /usr/share/gems/gems/net-ssh-4.2.0/lib/net/ssh.rb:237:in `start'
 d80f6231 | /usr/share/gems/gems/net-scp-3.0.0/lib/net/scp.rb:202:in `start'
 d80f6231 | /usr/share/gems/gems/fog-core-2.2.4/lib/fog/core/scp.rb:73:in `upload'
 d80f6231 | /usr/share/foreman/app/services/foreman/provision/ssh.rb:18:in `deploy!'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration/ssh_provision.rb:64:in `setSSHProvision'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration.rb:227:in `execute'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration.rb:152:in `block in process'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `each'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration.rb:144:in `process'
 d80f6231 | /usr/share/foreman/app/models/concerns/orchestration.rb:56:in `post_commit'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:427:in `block in make_lambda'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:270:in `block in simple'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:516:in `block in invoke_after'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:516:in `each'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:516:in `invoke_after'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:107:in `run_callbacks'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:824:in `_run_commit_callbacks'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:321:in `committed!'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:155:in `commit_records'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:304:in `block in commit_transaction'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:294:in `commit_transaction'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:351:in `block in within_new_transaction'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/transactions.rb:298:in `save'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/suppressor.rb:44:in `save'
 d80f6231 | /usr/share/foreman/app/models/concerns/foreman/sti.rb:26:in `save'
 d80f6231 | /usr/share/foreman/app/controllers/api/v2/hosts_controller.rb:147:in `create'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/base.rb:228:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 d80f6231 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 d80f6231 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 d80f6231 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 d80f6231 | /usr/share/gems/gems/audited-4.10.0/lib/audited/sweeper.rb:14:in `around'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 d80f6231 | /usr/share/gems/gems/audited-4.10.0/lib/audited/sweeper.rb:14:in `around'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:137:in `run_callbacks'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications.rb:203:in `block in instrument'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications.rb:203:in `instrument'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 d80f6231 | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/base.rb:165:in `process'
 d80f6231 | /usr/share/gems/gems/actionview-6.1.6.1/lib/action_view/rendering.rb:39:in `process'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal.rb:190:in `dispatch'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal.rb:254:in `dispatch'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:32:in `each'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:32:in `serve'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:842:in `call'
 d80f6231 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 d80f6231 | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/static_dispatcher.rb:66:in `call'
 d80f6231 | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/extractor/recorder.rb:137:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 d80f6231 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 d80f6231 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 d80f6231 | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 d80f6231 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 d80f6231 | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/rack/logger.rb:37:in `call_app'
 d80f6231 | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/rack/logger.rb:28:in `call'
 d80f6231 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 d80f6231 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 d80f6231 | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 d80f6231 | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 d80f6231 | /usr/share/gems/gems/secure_headers-6.3.4/lib/secure_headers/middleware.rb:11:in `call'
 d80f6231 | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/engine.rb:539:in `call'
 d80f6231 | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `public_send'
 d80f6231 | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `method_missing'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 d80f6231 | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/configuration.rb:252:in `call'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/request.rb:77:in `block in handle_request'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/request.rb:76:in `handle_request'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/server.rb:441:in `process_client'
 d80f6231 | /usr/share/gems/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 d80f6231 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Perhaps it’s because it’s trying to use the deffie_hellman_group1_sha1.rb instead of the sha256.rb ? Not really sure whats causing the issue here.

#3

Some extra bits here. I extracted the private key out of the db (due to a UI bug that will not present the key) verified it’s the same key-pair that is showing up in the AWS UI and used that to ssh into the host without any issue. So it feels like some config / setting / file somewhere is not informing the running foreman process to not pick the worst possible transport key to try and connect, but I cannot what is driving the decision to use the diffie_hellman_group1_sha1 instead of the better sha files.

#4

It seems you are using image based provisioning without user-data (cloud init). That way Foreman uses net/ssh which we know is out of date. That was the reason why REX plugin migrated to pure openssh client. I’m afraid there was no easy way around that (unless @aruzicka knows some trick).

#5

I don’t think thats the case, as I did upgrade to 3.5 in an attempt to see if that made any difference (and due to a note about crypto settings for apache) and that has been working fine since (just finished testing this yesterday). So there is something broken in 3.4, that was not broken way back in 2.4, that got fixed in 3.5. Perhaps it doesn’t matter as there is a path forward, but if this is the cause of 3.4 not working, folks probably have no way to provision using FIPS in that scenario.