Unattended installation - internal server 500 error - HTTP Not found

Support
1

I’m attempting to PXE boot Rocky Linux using Foreman, but I’m encountering the following error during the boot process:

curl (22): The request URL returned error: 500 Internal server error
warning: anaconda: failed to fetch kickstart from http://foreman.example.com:8000/unattended/provision?static=1&token=dfbfe658-dc84-4530-ba39-b99bb92ecdc2

I see below errors recorded in the foreman-proxy log

/usr/share/foreman-proxy/lib/proxy/log.rb:105:in call' /usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in service' /opt/rh/rh-ruby27/root/usr/share/ruby/webrick/httpserver.rb:140:in service’
/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/httpserver.rb:96:in run' /opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:307:in block in start_thread’
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context' 2024-06-05T08:27:39 78d7e870 [W] Failed to proxy /provision for {"static"=>"1", "token"=>"dfbfe658-dc84-4530-ba39-b99bb92ecdc2", "kind"=>"provision"}: Error retrieving unattended/provision for {"static"=>"1", "token"=>"dfbfe658-dc84-4530-ba39-b99bb92ecdc2", "url"=>"http://foreman.example.com:8000"} from foreman.example.com: Net::HTTPNotFound: <RuntimeError>: Error retrieving unattended/provision for {"static"=>"1", "token"=>"dfbfe658-dc84-4530-ba39-b99bb92ecdc2", "url"=>"http://foreman.example.com:8000"} from foreman.example.com: Net::HTTPNotFound /usr/share/foreman-proxy/modules/templates/proxy_request.rb:51:in call_template’
/usr/share/foreman-proxy/modules/templates/proxy_request.rb:12:in get' /usr/share/foreman-proxy/modules/templates/template_proxy_request.rb:6:in get’
/usr/share/foreman-proxy/modules/templates/templates_unattended_api.rb:23:in block (2 levels) in <class:TemplatesUnattendedApi>' /usr/share/foreman-proxy/lib/proxy/helpers.rb:18:in log_halt’
/usr/share/foreman-proxy/modules/templates/templates_unattended_api.rb:22:in block in <class:TemplatesUnattendedApi>' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1674:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1674:in block in compile!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in block (3 levels) in route!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in route_eval' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in block (2 levels) in route!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in block in process_route' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in process_route' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in block in route!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in route!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in block in dispatch!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in block in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in invoke’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in dispatch!' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in block in call!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in call!’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in call' /usr/share/foreman-proxy/lib/proxy/log.rb:105:in call’
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in block in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in synchronize’
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in service’
/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/httpserver.rb:140:in service' /opt/rh/rh-ruby27/root/usr/share/ruby/webrick/httpserver.rb:96:in run’
/opt/rh/rh-ruby27/root/usr/share/ruby/webrick/server.rb:307:in block in start_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
2024-06-05T08:27:39 78d7e870 [I] Finished GET /unattended/provision with 500 (26.55 ms)

2

To me this looks like your main Foreman has trouble rendering the provisioning template and throws an internal server error on that. The smart-proxy only passes the error through to the client in this case. I am not super sure I read the logs correctly, since I don’t have much experience with the smart-proxy template feature, but this line looks super suspicious to me:

So you should:
a) check in your main Foreman server if the template renders there (e.g. via the template preview feature)
b) check the Foreman production.log for any errors related to rendering that template

3

Thanks for your response.
The problem was caused by an expired build token. Once I cancelled the expired build and initiated a new one in Foreman, the provisioning proceeded smoothly.