Sync status ends in 500 Internal Server error

Problem:
500 inetrnel server error
Expected outcome:
200
Foreman and Proxy versions:
foreman-cli-2.1.4-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.1.3-1.el7.noarch
foreman-release-2.1.4-1.el7.noarch
foreman-2.1.4-1.el7.noarch
foreman-service-2.1.4-1.el7.noarch
foreman-installer-katello-2.1.4-1.el7.noarch
foreman-debug-2.1.4-1.el7.noarch
foreman-installer-2.1.4-1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman_virt_who_configure-0.5.2-1.fm2_1.el7.noarch
foreman-dynflow-sidekiq-2.1.4-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
rubygem-foreman_maintain-0.6.13-1.el7.noarch
foreman-postgresql-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch

tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
Foreman and Proxy plugin versions:
foreman-cli-2.1.4-1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.1.3-1.el7.noarch
foreman-release-2.1.4-1.el7.noarch
foreman-2.1.4-1.el7.noarch
foreman-service-2.1.4-1.el7.noarch
foreman-installer-katello-2.1.4-1.el7.noarch
foreman-debug-2.1.4-1.el7.noarch
foreman-installer-2.1.4-1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman_virt_who_configure-0.5.2-1.fm2_1.el7.noarch
foreman-dynflow-sidekiq-2.1.4-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
rubygem-foreman_maintain-0.6.13-1.el7.noarch
foreman-postgresql-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch

tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
Distribution and version:
RHEL7.9
Other relevant data:
output from /var/log/foreman/production.log

500 Internal Server Error: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>500 Internal Server Error</title>
</head><body>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>
</body></html>

2020-11-09T00:16:21 [W|app|898ee467] Action failed
2020-11-09T00:16:21 [I|app|898ee467]   Rendering common/500.html.erb within katello/layouts/katello
2020-11-09T00:16:21 [I|app|898ee467]   Rendered common/500.html.erb within katello/layouts/katello (Duration: 1.5ms | Allocations: 630)
2020-11-09T00:16:21 [I|app|898ee467] Completed 500 Internal Server Error in 97ms (ActiveRecord: 19.0ms | Allocations: 19766)
2020-11-09T00:16:21 [F|app|898ee467]
ActionView::Template::Error (wrong number of arguments (given 1, expected 0)):
    1: <%= alert :header => _("Oops, we're sorry but something went wrong"), :class => 'alert-danger',
    2:   :actions => link_to(_('Back'), main_app.root_path, :class => 'btn btn-default'),
    3:   :text => h(exception.message),
    4:   :close => false %>

app/views/common/500.html.erb:1:in `_3b4be9bfb574faaf3274ff6d95e2a211'
app/controllers/application_controller.rb:352:in `generic_exception'
katello (3.16.1.2) lib/katello/middleware/event_daemon.rb:10:in `call'
lib/foreman/middleware/telemetry.rb:10:in `call'
lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
lib/foreman/middleware/logging_context_session.rb:22:in `call'
lib/foreman/middleware/logging_context_request.rb:11:in `call'
katello (3.16.1.2) lib/katello/prevent_json_parsing.rb:12:in `call'

Hi @Aleksandar76

From reading the stack I’m not quite sure what’s causing the internal server error. Perhaps there’s a more informative error before that one in the log?

What specific steps are you taking when this error occurs?

Have you found any solution? I am getting the same error , but on a newer version. By the same error I mean " Sync status ends in 500 Internal Server error"
The log shows:

/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2020-11-16T17:44:53 [I|app|289bb]   Rendering common/500.html.erb within katello/layouts/katello
2020-11-16T17:44:53 [I|app|289bb]   Rendered common/500.html.erb within katello/layouts/katello (4.9ms)
2020-11-16T17:44:53 [I|app|289bb] Completed 500 Internal Server Error in 50ms (ActiveRecord: 5.6ms)
2020-11-16T17:44:53 [F|app|]   
2020-11-16T17:44:53 [F|app|] ActionView::Template::Error (wrong number of arguments (given 1, expected 0)):
2020-11-16T17:44:53 [F|app|]     1: <%= alert :header => _("Oops, we're sorry but something went wrong"), :class => 'alert-danger',
    2:   :actions => link_to(_('Back'), main_app.root_path, :class => 'btn btn-default'),
    3:   :text => h(exception.message),
    4:   :close => false %>
2020-11-16T17:44:53 [F|app|]   
2020-11-16T17:44:53 [F|app|] app/views/common/500.html.erb:1:in `_4fd2abc175d50005f949d18eab2a9e4e'
app/controllers/application_controller.rb:329:in `generic_exception'
lib/middleware/telemetry.rb:9:in `call'
lib/middleware/catch_json_parse_errors.rb:8:in `call'
lib/middleware/logging_context.rb:23:in `call'
katello (3.10.1.1) lib/katello/prevent_json_parsing.rb:12:in `call'

In my case it was typo in yum.importer.json,but i switched back to 3.14, the only working version for me

Having a similar message, did anybody get anywhere with this?

2021-01-27T13:33:01 [I|app|09d28b48] Started GET “/katello/sync_management” for 131.185.41.82 at 2021-01-27 13:33:01 +1030
2021-01-27T13:33:01 [I|app|09d28b48] Processing by Katello::SyncManagementController#index as HTML
2021-01-27T13:33:01 [D|tax|09d28b48] Current location set to none
2021-01-27T13:33:01 [D|tax|09d28b48] Current organization set to ORG
2021-01-27T13:33:01 [E|kat|09d28b48] POST: https://foreman.org.au/pulp/api/v2/repositories/search/: {“criteria”:{“filters”:{“id”:{"$in":[“e1a67f64-93f2-40d6-ac31-2de8d084e945”,“26d89656-6665-41b7-a8fb-07261d05f1aa”,“e54c4dd5-3031-45cc-b1f2-0411d9637ccd”,“4fc0cd2f-ec8c-40a8-8842-1684374dbfea”,“a47afa7b-17bc-413d-bc43-33434c5f030a”,“1fb70347-b5ab-4c96-bfdd-972f948ac6f3”,“1-CentOS7-Library-e1a67f64-93f2-40d6-ac31-2de8d084e945”,“13868236-782b-470d-9c72-c59c60e0ac7a”,“a7dfc47f-76e6-442d-b1fe-12861de9b2b8”,“1-CentOS8-Library-e54c4dd5-3031-45cc-b1f2-0411d9637ccd”,“7a35868f-1275-4c33-a772-a77a6e630410”,“e030a61b-6428-4f80-beef-1f28344d8f01”,“008cf69b-0a3b-47ba-9839-29d09fb96c65”,“573acc08-6bd0-4bce-b5d4-7398dcf8e016”,“1-CentOS7-Library-26d89656-6665-41b7-a8fb-07261d05f1aa”,“1-CentOS7-Library-4fc0cd2f-ec8c-40a8-8842-1684374dbfea”,“1-CentOS7-Library-a47afa7b-17bc-413d-bc43-33434c5f030a”,“1-CentOS7-Library-1fb70347-b5ab-4c96-bfdd-972f948ac6f3”,“1-CentOS7-Library-a7dfc47f-76e6-442d-b1fe-12861de9b2b8”,“1-CentOS8-Library-e09ecaaa-563d-49e4-ba06-9eadf9dc6cec”,“1-CentOS8-Library-e1725b26-8477-4da5-ae51-0b34968c6073”,“1-CentOS8-Library-e6301558-5819-450f-867c-d534e65aae33”,“1-CentOS8-Library-8cb3f075-2238-4124-a62c-983598e536fd”,“ba888921-be45-42ab-9292-ddfb33c3764e”,“1-CentOS7-Library-7a35868f-1275-4c33-a772-a77a6e630410”,“1-CentOS7-Library-ba888921-be45-42ab-9292-ddfb33c3764e”,“1-CentOS8-Library-61896f8e-2213-4dc1-810e-2be768bd00b9”,“e6301558-5819-450f-867c-d534e65aae33”,“1-CentOS7-Library-5a4f4dc3-067d-4481-b777-5a292b65441c”,“5a4f4dc3-067d-4481-b777-5a292b65441c”,“8cb3f075-2238-4124-a62c-983598e536fd”,“f85d407f-6963-453a-b21a-4b761f6de56d”,“b6b5627f-ece9-42ec-9cc5-01609183b4b3”,“574cf3ae-64fb-492d-b41f-8faca09f49f8”,“95d3fd0f-4af2-4409-8008-ca888f278998”,“07742ddb-98e8-44e4-9eba-4c4e808db44e”,“3f667ac9-4705-4bce-a6fd-33619d293ec8”,“c1205a22-50f8-405d-ab30-1b53153b48b2”,“4df90022-e444-4529-ab9f-8619fea95053”,“61896f8e-2213-4dc1-810e-2be768bd00b9”,“cb6e6a2e-e087-491b-8756-5e9fa9303218”,“d857f442-ea06-4d38-83cf-b156ea3861d2”,“e1725b26-8477-4da5-ae51-0b34968c6073”,“cb72efaf-7e4d-4f9e-9e27-2b770da2ad6d”,“e466ef9b-737a-4a63-8a48-d94ed8fba0bb”,“e09ecaaa-563d-49e4-ba06-9eadf9dc6cec”,“ffb5b951-204d-4c32-a76e-1c02e4ed57d8”]}}}}: {“content_type”=>“application/json”, “accept”=>“application/json”}
09d28b48 | 500 Internal Server Error:
09d28b48 |
09d28b48 | 500 Internal Server Error
09d28b48 |
09d28b48 |

Internal Server Error


09d28b48 |

The server encountered an internal error or
09d28b48 | misconfiguration and was unable to complete
09d28b48 | your request.


09d28b48 |

Please contact the server administrator at
09d28b48 | [no address given] to inform them of the time this error occurred,
09d28b48 | and the actions you performed just before this error.


09d28b48 |

More information about this error may be available
09d28b48 | in the server error log.


09d28b48 |
09d28b48 |
2021-01-27T13:33:01 [W|app|09d28b48] 500 Internal Server Error
2021-01-27T13:33:01 [I|app|09d28b48] Backtrace for ‘500 Internal Server Error’ error (RestClient::InternalServerError): 500 Internal Server Error
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:223:in exception_with_response' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/abstract_response.rb:103:in return!’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:95:in block in get_response' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:807:in process_result’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:725:in block in transmit' 09d28b48 | /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:910:in start’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in transmit' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in post’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/repository.rb:66:in search' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/extensions/repository.rb:96:in search_by_repository_ids’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/models/katello/glue/pulp/repos.rb:24:in prepopulate!' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/helpers/katello/sync_management_helper.rb:38:in collect_repos’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/app/controllers/katello/sync_management_controller.rb:29:in index' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:195:in process_action' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rendering.rb:30:in process_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:42:in block in process_action' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:112:in block in run_callbacks’
09d28b48 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
09d28b48 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in run_callbacks' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in process_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in process_action' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in block in process_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in block in instrument' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in instrument’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in instrument' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in process_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in process_action' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in process_action’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in process' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in process’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in dispatch' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in dispatch’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in dispatch' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in serve’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in block in serve' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in each’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in serve' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in public_send’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in method_missing' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:19:in block in class:Constraints
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in serve' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in block in serve’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in each' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in serve’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/middleware/event_daemon.rb:10:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 09d28b48 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call' 09d28b48 | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in call' 09d28b48 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in run_callbacks' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in call_app’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call’
09d28b48 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1/lib/katello/prevent_json_parsing.rb:12:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in public_send’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in method_missing' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in call' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in handle_request’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in process_client' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in block in run’
09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in block in spawn_thread' 09d28b48 | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
2021-01-27T13:33:01 [I|app|09d28b48] Rendering common/500.html.erb within katello/layouts/katello
2021-01-27T13:33:01 [I|app|09d28b48] Rendered common/500.html.erb within katello/layouts/katello (Duration: 2.1ms | Allocations: 738)
2021-01-27T13:33:01 [I|app|09d28b48] Completed 500 Internal Server Error in 100ms (ActiveRecord: 4.0ms | Allocations: 19311)
2021-01-27T13:33:01 [F|app|09d28b48]
09d28b48 | ActionView::Template::Error (wrong number of arguments (given 1, expected 0)):
09d28b48 | 1: <%= alert :header => (“Oops, we’re sorry but something went wrong”), :class => ‘alert-danger’,
09d28b48 | 2: :actions => link_to(
(‘Back’), main_app.root_path, :class => ‘btn btn-default’),
09d28b48 | 3: :text => h(exception_message),
09d28b48 | 4: :close => false %>
09d28b48 |
09d28b48 | app/views/common/500.html.erb:1:in _d2f2a3c822caef3da0171aae76eab687' 09d28b48 | app/controllers/application_controller.rb:354:in generic_exception’
09d28b48 | katello (3.18.1) lib/katello/middleware/event_daemon.rb:10:in call' 09d28b48 | lib/foreman/middleware/telemetry.rb:10:in call’
09d28b48 | lib/foreman/middleware/catch_json_parse_errors.rb:9:in call' 09d28b48 | lib/foreman/middleware/logging_context_session.rb:22:in call’
09d28b48 | lib/foreman/middleware/logging_context_request.rb:11:in call' 09d28b48 | katello (3.18.1) lib/katello/prevent_json_parsing.rb:12:in call’

Not sure if this is related:

foreman]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 79ms
candlepin_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
candlepin_auth:
Status: ok
Server Response: Duration: 45ms
katello_events:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
pulp:
Status: FAIL
Server Response: Message: 500 Internal Server Error
pulp_auth:
Status: FAIL
Server Response: Message: Skipped pulp_auth check after failed pulp check
pulp3:
Status: ok
Server Response: Duration: 314ms
foreman_tasks:
Status: ok
Server Response: Duration: 5ms

Looks like a pulp problem for me:

Jan 27 14:15:58 foreman vmware_vra_software_agent: Wed Jan 27 14:15:58 ACDT 2021 WAIT_COUNTER is 5250, WAIT_LIMIT is 21600, sleep 30
Jan 27 14:16:08 foreman pulp: pulp.server.db.migrate.models:ERROR: pulp.server.db.migrations hasn’t been updated to the latest available migration.
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:CRITICAL: *************************************************************
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:CRITICAL: The Pulp server failed to start due to the following reasons:
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) Traceback (most recent call last):
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) File “/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py”, line 111, in wsgi_application
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) _initialize_web_services()
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) File “/usr/lib/python2.7/site-packages/pulp/server/webservices/application.py”, line 74, in _initialize_web_services
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) migration_models.check_package_versions()
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) File “/usr/lib/python2.7/site-packages/pulp/server/db/migrate/models.py”, line 331, in check_package_versions
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) raise Exception(error_message)
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:ERROR: (11423-13760) InitializationException: The database has not been migrated to the current version. Run pulp-manage-db and restart the application.
Jan 27 14:16:08 foreman pulp: pulp.server.webservices.application:CRITICAL: ************************************************************

Did you just update pulp-server and some other pulp rpms on your server? They have arrived yesterday or the day before in the katello pulp archive. I have had 500 errors after the update until I run foreman-installer again.

Spot on mate. Some weird ideas about auto patching servers here! I’ll be disabling that ASAP.

I’ve run pulp-manage-db and all is well now.