Response: 500 Internal Server Error after upgrade to 3.15

admin_hw · June 25, 2025, 3:10pm

Problem:
During the upgrade from 3.14 to 3.15 I’ve received many errors with response 500. Foreman service starts, but I can’t see host, collections, lifecycle etc. Everything with Internal error 500.
Expected outcome:
Upgrade from 3.14 to 3.15

Foreman and Proxy versions:
foreman-3.15.0-1.el9.noarch

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

[PROD] [root@mit1-foreman-2:~]# foreman-installer
2025-06-25 16:51:31 [NOTICE] [root] Loading installer configuration. This will take some time.
2025-06-25 16:51:34 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2025-06-25 16:51:34 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2025-06-25 16:51:35 [NOTICE] [checks] System checks passed
2025-06-25 16:51:47 [NOTICE] [configure] Starting system configuration.
2025-06-25 16:51:56 [NOTICE] [configure] 250 configuration steps out of 1469 steps complete.
2025-06-25 16:51:57 [NOTICE] [configure] 500 configuration steps out of 1472 steps complete.
2025-06-25 16:52:00 [NOTICE] [configure] 750 configuration steps out of 1474 steps complete.
2025-06-25 16:52:00 [NOTICE] [configure] 1000 configuration steps out of 1480 steps complete.
2025-06-25 16:53:03 [NOTICE] [configure] 1250 configuration steps out of 1480 steps complete.
2025-06-25 16:54:11 [ERROR ] [configure] Error making POST request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/hosts/facts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:11 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-mit1-foreman-2.mit.etn]/ensure: change from 'absent' to 'present' failed: Error making POST request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/hosts/facts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:50 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-mit1-foreman-2.mit.etn]: Could not evaluate: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:50 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[mit1-foreman-2.mit.etn]: Could not evaluate: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:51 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[mit1-foreman-2.mit.etn]: Failed to call refresh: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:51 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[mit1-foreman-2.mit.etn]: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
2025-06-25 16:54:54 [NOTICE] [configure] System configuration has finished.

Error 1: Puppet Foreman_host resource 'foreman-mit1-foreman-2.mit.etn' failed. Logs:
  /Stage[main]/Foreman::Register/Foreman_host[foreman-mit1-foreman-2.mit.etn]
    Adding autorequire relationship with Anchor[foreman::service]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1375 of 1480)
    Evaluated in 0.87 seconds
  Foreman_host[foreman-mit1-foreman-2.mit.etn](provider=rest_v3)
    Making post request to https://mit1-foreman-2.mit.etn/api/v2/hosts/facts
    Received response 500 from request to https://mit1-foreman-2.mit.etn/api/v2/hosts/facts
  /Stage[main]/Foreman::Register/Foreman_host[foreman-mit1-foreman-2.mit.etn]/ensure
    change from 'absent' to 'present' failed: Error making POST request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/hosts/facts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
Error 2: Puppet Foreman_host resource 'foreman-proxy-mit1-foreman-2.mit.etn' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-mit1-foreman-2.mit.etn]
    Adding autorequire relationship with Anchor[foreman::service]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1465 of 1480)
    Could not evaluate: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
    Evaluated in 0.51 seconds
  Foreman_host[foreman-proxy-mit1-foreman-2.mit.etn](provider=rest_v3)
    Making get request to https://mit1-foreman-2.mit.etn/api/v2/hosts?search=name%3D%22mit1-foreman-2.mit.etn%22
    Received response 500 from request to https://mit1-foreman-2.mit.etn/api/v2/hosts?search=name%3D%22mit1-foreman-2.mit.etn%22
Error 3: Puppet Foreman_smartproxy resource 'mit1-foreman-2.mit.etn' failed. Logs:
  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[mit1-foreman-2.mit.etn]
    Adding autorequire relationship with Anchor[foreman::service]
    Adding autorequire relationship with Anchor[foreman::providers::oauth]
    Starting to evaluate the resource (1467 of 1480)
    Could not evaluate: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
    Failed to call refresh: Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
    Error making GET request to Foreman at https://mit1-foreman-2.mit.etn/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on mit1-foreman-2.mit.etn for detailed information
    Evaluated in 1.02 seconds
  Foreman_smartproxy[mit1-foreman-2.mit.etn](provider=rest_v3)
    Making get request to https://mit1-foreman-2.mit.etn/api/v2/smart_proxies?search=name%3D%22mit1-foreman-2.mit.etn%22
    Received response 500 from request to https://mit1-foreman-2.mit.etn/api/v2/smart_proxies?search=name%3D%22mit1-foreman-2.mit.etn%22
    Making get request to https://mit1-foreman-2.mit.etn/api/v2/smart_proxies?search=name%3D%22mit1-foreman-2.mit.etn%22
    Received response 500 from request to https://mit1-foreman-2.mit.etn/api/v2/smart_proxies?search=name%3D%22mit1-foreman-2.mit.etn%22

3 errors were detected.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.

The full log is at /var/log/foreman-installer/katello.log

production.tar.gz (1.0 MB)

admin_hw · June 26, 2025, 4:53pm

Hey,
my collegue Zigi make a hotfix by editing of apipie.rb

[PROD] [root@mit1-foreman-2:/usr/share/foreman/config/initializers]# diff -u apipie.rb.orig apipie.rb 
--- apipie.rb.orig      2025-06-09 14:43:55.000000000 +0200
+++ apipie.rb   2025-06-26 17:53:44.432990407 +0200
@@ -62,6 +62,9 @@
       :default_nic_type => InterfaceTypeMapper::DEFAULT_TYPE.humanized_name.downcase,
       :template_kinds => -> { Rails.cache.fetch("template_kind_names", expires_in: 1.hour) { TemplateKind.pluck(:name).join(", ") } },
       :host_rebuild_steps => -> { Host::Managed.valid_rebuild_only_values.join(', ') },
+      :package => "",
+      :job_category => "",
+      :template_name => "",
     }
 
     config.translate = lambda do |str, loc|

Please make a fix. Thank you.
Best regards,
Ondrej