Problem:
on foreman 3.16/katello 4.18 the deployment of new vms via vmware image build is working fine (at least if proper firmware is chosen, unfortunately its not saved/read from compute profile and defaults to BIOS) , some of the customization is also applied like root pwd change. But its e.g. not registering the systems to the Katello for content itself and some other scripts are missing
Expected outcome:
It should work like it did prior to the upgrade to foreman 3.16/katello 4.18
There the cloud init did work and continue with the customization.
Foreman and Proxy versions:
foreman 3.16/katello 4.18
Foreman and Proxy plugin versions:
foreman 3.16/katello 4.18
Distribution and version:
RHEL 9.6
Other relevant data:
Logs on deployed system
2025-10-16 15:43:52,415 - util.py[WARNING]: Failed to post phone home data to http://foreman-proxy.fqdn.local:8000/unattended/built?token=6be61449-a8a8-470a-90cc-25c99dcdd69d in 10 tries
2025-10-16 15:43:52,416 - util.py[DEBUG]: Failed to post phone home data to http://foreman-proxy.fqdn.local:8000/unattended/built?token=6be61449-a8a8-470a-90cc-25c99dcdd69d in 10 tries
Traceback (most recent call last):
File "/usr/lib/python3.9/site-packages/cloudinit/config/cc_phone_home.py", line 195, in handle
url_helper.read_file_or_url(
File "/usr/lib/python3.9/site-packages/cloudinit/url_helper.py", line 87, in read_file_or_url
return readurl(url, **kwargs)
File "/usr/lib/python3.9/site-packages/cloudinit/url_helper.py", line 370, in readurl
raise excps[-1]
cloudinit.url_helper.UrlError: 500 Server Error: Internal Server Error for url: http://foreman-proxy.fqdn.local:8000/unattended/built?token=6be61449-a8a8-470a-90cc-25c99dcdd69d
2025-10-16 15:43:52,418 - handlers.py[DEBUG]: finish: modules-final/config-phone-home: SUCCESS: config-phone-home ran successfully
Logs of main foreman
while I think most important are
025-10-17T09:23:34 [W|app|f6179538] undefined method `[]' for nil:NilClass
2025-10-17T09:23:34 [I|app|f6179538] Backtrace for 'undefined method `[]' for nil:NilClass' error (NoMethodError): undefined method `[]' for nil:NilClass
f6179538 | /usr/share/foreman/app/models/concerns/hostext/operating_system.rb:41:in `template_kinds'
f6179538 | /usr/share/foreman/app/models/concerns/orchestration/templates.rb:43:in `kind'
f6179538 | /usr/share/foreman/app/models/concerns/orchestration/templates.rb:35:in `template'
f6179538 | /usr/share/foreman/app/models/concerns/orchestration/templates.rb:12:in `queue_render_checks'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:400:in `block in make_lambda'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:247:in `block in halting'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:599:in `block in invoke_after'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:599:in `each'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:599:in `invoke_after'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:108:in `run_callbacks'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:929:in `_run_validation_callbacks'
f6179538 | /usr/share/gems/gems/activemodel-7.0.8.7/lib/active_model/validations/callbacks.rb:115:in `run_validations!'
f6179538 | /usr/share/gems/gems/activemodel-7.0.8.7/lib/active_model/validations.rb:337:in `valid?'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/validations.rb:68:in `valid?'
f6179538 | /usr/share/foreman/app/models/concerns/orchestration.rb:83:in `valid?'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/validations.rb:84:in `perform_validations'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/validations.rb:47:in `save'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/transactions.rb:298:in `block in save'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/transactions.rb:354:in `block in with_transaction_returning_status'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/transactions.rb:350:in `with_transaction_returning_status'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/transactions.rb:298:in `save'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/suppressor.rb:50:in `save'
f6179538 | /usr/share/foreman/app/models/host/managed.rb:387:in `built'
f6179538 | /usr/share/foreman/app/models/concerns/hostext/ui_notifications.rb:9:in `built'
f6179538 | /usr/share/foreman/app/controllers/unattended_controller.rb:32:in `built'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/abstract_controller/base.rb:215:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/rendering.rb:165:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/abstract_controller/callbacks.rb:234:in `block in process_action'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
f6179538 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
f6179538 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
f6179538 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
f6179538 | /usr/share/gems/gems/audited-5.8.0/lib/audited/sweeper.rb:16:in `around'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
f6179538 | /usr/share/gems/gems/audited-5.8.0/lib/audited/sweeper.rb:16:in `around'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:127:in `block in run_callbacks'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:138:in `run_callbacks'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/abstract_controller/callbacks.rb:233:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/rescue.rb:23:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/instrumentation.rb:67:in `block in process_action'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/notifications.rb:206:in `block in instrument'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/notifications.rb:206:in `instrument'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/instrumentation.rb:66:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal/params_wrapper.rb:259:in `process_action'
f6179538 | /usr/share/gems/gems/activerecord-7.0.8.7/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/abstract_controller/base.rb:151:in `process'
f6179538 | /usr/share/gems/gems/actionview-7.0.8.7/lib/action_view/rendering.rb:39:in `process'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal.rb:188:in `dispatch'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_controller/metal.rb:251:in `dispatch'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/routing/route_set.rb:32:in `serve'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/journey/router.rb:50:in `block in serve'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/journey/router.rb:32:in `each'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/journey/router.rb:32:in `serve'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/routing/route_set.rb:852:in `call'
f6179538 | /usr/share/gems/gems/katello-4.18.0/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
f6179538 | /usr/share/gems/gems/katello-4.18.0/lib/katello/middleware/event_daemon.rb:10:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/gems/gems/apipie-dsl-2.6.2/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
f6179538 | /usr/share/gems/gems/apipie-rails-1.4.2/lib/apipie/static_dispatcher.rb:74:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/static.rb:23:in `call'
f6179538 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
f6179538 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
f6179538 | /usr/share/gems/gems/apipie-rails-1.4.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/tempfile_reaper.rb:15:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/etag.rb:27:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/conditional_get.rb:27:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/head.rb:12:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/http/permissions_policy.rb:38:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/http/content_security_policy.rb:39:in `call'
f6179538 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/session/abstract/id.rb:266:in `context'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/session/abstract/id.rb:260:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/cookies.rb:704:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/callbacks.rb:99:in `run_callbacks'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/debug_exceptions.rb:28:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/show_exceptions.rb:29:in `call'
f6179538 | /usr/share/gems/gems/railties-7.0.8.7/lib/rails/rack/logger.rb:40:in `call_app'
f6179538 | /usr/share/gems/gems/railties-7.0.8.7/lib/rails/rack/logger.rb:27:in `call'
f6179538 | /usr/share/gems/gems/sprockets-rails-3.5.2/lib/sprockets/rails/quiet_assets.rb:17:in `call'
f6179538 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/remote_ip.rb:93:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/request_id.rb:26:in `call'
f6179538 | /usr/share/gems/gems/katello-4.18.0/lib/katello/prevent_json_parsing.rb:12:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/method_override.rb:24:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/runtime.rb:22:in `call'
f6179538 | /usr/share/gems/gems/activesupport-7.0.8.7/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/executor.rb:14:in `call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/sendfile.rb:110:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/ssl.rb:77:in `call'
f6179538 | /usr/share/gems/gems/actionpack-7.0.8.7/lib/action_dispatch/middleware/host_authorization.rb:131:in `call'
f6179538 | /usr/share/gems/gems/secure_headers-7.1.0/lib/secure_headers/middleware.rb:11:in `call'
f6179538 | /usr/share/gems/gems/railties-7.0.8.7/lib/rails/engine.rb:530:in `call'
f6179538 | /usr/share/gems/gems/railties-7.0.8.7/lib/rails/railtie.rb:226:in `public_send'
f6179538 | /usr/share/gems/gems/railties-7.0.8.7/lib/rails/railtie.rb:226:in `method_missing'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/urlmap.rb:74:in `block in call'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/urlmap.rb:58:in `each'
f6179538 | /usr/share/gems/gems/rack-2.2.17/lib/rack/urlmap.rb:58:in `call'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/configuration.rb:279:in `call'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/request.rb:99:in `block in handle_request'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/thread_pool.rb:390:in `with_force_shutdown'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/request.rb:98:in `handle_request'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/server.rb:472:in `process_client'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/server.rb:254:in `block in run'
f6179538 | /usr/share/gems/gems/puma-6.6.1/lib/puma/thread_pool.rb:167:in `block in spawn_thread'
f6179538 | /usr/share/gems/gems/logging-2.4.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2025-10-17T09:23:34 [I|app|f6179538] Completed 500 Internal Server Error in 837ms (ActiveRecord: 19.5ms | Allocations: 62346)
2025-10-17T09:23:34 [F|app|f6179538]
f6179538 | ActionView::MissingTemplate (Missing template common/500 with {:locale=>[:en], :formats=>[:text], :variants=>[], :handlers=>[:raw, :erb, :html, :builder, :ruby, :rabl]}.
f6179538 |
f6179538 | Searched in:
f6179538 | * "/usr/share/foreman/app/views"
f6179538 | * "/usr/share/gems/gems/katello-4.18.0/app/views"
f6179538 | * "/usr/share/gems/gems/katello-4.18.0/engines/bastion/app/views"
f6179538 | * "/usr/share/gems/gems/foreman_ansible-17.0.1/app/views"
f6179538 | * "/usr/share/gems/gems/foreman_remote_execution-16.2.1/app/views"
f6179538 | * "/usr/share/gems/gems/foreman-tasks-11.0.4/app/views"
f6179538 | * "/usr/share/gems/gems/apipie-dsl-2.6.2/app/views"
f6179538 | * "/usr/share/gems/gems/apipie-rails-1.4.2/app/views"
f6179538 | ):
f6179538 |
f6179538 | app/controllers/application_controller.rb:347:in `generic_exception'
f6179538 | katello (4.18.0) lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
f6179538 | katello (4.18.0) lib/katello/middleware/event_daemon.rb:10:in `call'
f6179538 | lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
f6179538 | lib/foreman/middleware/telemetry.rb:10:in `call'
f6179538 | lib/foreman/middleware/logging_context_session.rb:22:in `call'
f6179538 | lib/foreman/middleware/logging_context_request.rb:11:in `call'
f6179538 | katello (4.18.0) lib/katello/prevent_json_parsing.rb:12:in `call'
After update my environment was also affected by Smart Proxy Sync failure - #10 by rfm33 which was fixed by
foreman-installer --foreman-proxy-content-enable-docker false