Getting "Internal Server Error" when running hammer job-invocation output --id XX --host YYYY

Support
1

When running hammer job-invocation output --id XX --host YYYYY I get:
Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server’s logs.

Expected outcome:
Job output is shown

Foreman and Proxy versions:
foreman-release-2.3.5-1
katello-common-3.18.3-1

Foreman and Proxy plugin versions:
tfm-rubygem-foreman_remote_execution-4.2.2-1.fm2_3
tfm-rubygem-foreman_remote_execution_core-1.4.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.2.1-1.fm2_3.el7.noarch

Distribution and version:
Centos 7.9

Other relevant data:
Asking for the data via the API works OK

2

Hi,
what you described is not nearly enough for anyone to really tell what’s going on. The error message guides you to tak a look at the server’s logs. What did you see there? Could you post relevant part of the logs here?

1 Like
3

Hi, thanks for the reply. I believe I’ve found the relevant part of the log indicating the failure. Apologies for not including it earlier.

2021-09-01T09:24:59 [I|app|8d95731f] Started GET “/api/hosts?search=name+%3D+%22c05drddrv116.dslab.ad.xxx.com%22&per_page=1000&page=1” for 100.112.248.94 at 2021-09-01 09:24:59 +0100
2021-09-01T09:24:59 [I|app|8d95731f] Processing by Api::V2::HostsController#index as JSON
2021-09-01T09:24:59 [I|app|8d95731f] Parameters: {“search”=>“name = “c05drddrv116.dslab.ad.xxx.com””, “per_page”=>“1000”, “page”=>“1”, “apiv”=>“v2”, “host”=>{}}
2021-09-01T09:24:59 [I|app|8d95731f] Authorized user admin(Admin User)
2021-09-01T09:24:59 [I|app|8d95731f] Rendering api/v2/hosts/index.json.rabl within api/v2/layouts/index_layout
2021-09-01T09:24:59 [I|app|8d95731f] Rendered api/v2/hosts/index.json.rabl within api/v2/layouts/index_layout (Duration: 43.9ms | Allocations: 14807)
2021-09-01T09:24:59 [I|app|8d95731f] Completed 200 OK in 215ms (Views: 47.4ms | ActiveRecord: 37.2ms | Allocations: 37346)
2021-09-01T09:24:59 [I|app|8ee5140d] Started GET “/api/job_invocations/37/hosts/36?location_id=2&organization_id=3&since” for 100.112.248.94 at 2021-09-01 09:24:59 +0100
2021-09-01T09:24:59 [I|app|8ee5140d] Processing by Api::V2::JobInvocationsController#output as JSON
2021-09-01T09:24:59 [I|app|8ee5140d] Parameters: {“location_id”=>“2”, “organization_id”=>“3”, “since”=>nil, “apiv”=>“v2”, “job_invocation_id”=>“37”, “host_id”=>“36”, “job_invocation”=>{}}
2021-09-01T09:24:59 [I|app|8ee5140d] Authorized user admin(Admin User)
2021-09-01T09:24:59 [W|app|8ee5140d] Action failed
2021-09-01T09:24:59 [I|app|8ee5140d] Backtrace for ‘Action failed’ error (NoMethodError): undefined method targeting' for #<Location:0x0000000017159480> 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activemodel-6.0.3.4/lib/active_model/attribute_methods.rb:432:in method_missing’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/auditor.rb:99:in method_missing' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.2.2/app/controllers/api/v2/job_invocations_controller.rb:178:in find_host’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:428:in block in make_lambda' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:181:in block (2 levels) in halting_and_conditional’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:34:in block (2 levels) in <module:Callbacks>' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:182:in block in halting_and_conditional’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in block in invoke_before' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in each’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in invoke_before' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:110:in block in run_callbacks’
8ee5140d | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
8ee5140d | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
8ee5140d | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in block in run_callbacks’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in run_callbacks' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in process_action’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in process_action' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in block in process_action’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in block in instrument' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in instrument’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in instrument' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in process_action’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in process_action' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in process_action’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in process' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in process’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in dispatch' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in dispatch’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in dispatch' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in serve’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:18:in block in <class:Constraints>' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in serve’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in block in serve' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in each’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in serve' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/lib/katello/middleware/event_daemon.rb:10:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
8ee5140d | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call’
8ee5140d | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in call’
8ee5140d | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in block in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in run_callbacks’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in call_app' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call' 8ee5140d | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.3.1/lib/katello/prevent_json_parsing.rb:12:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in public_send' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in method_missing’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in call’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in handle_request' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in process_client’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in block in run' 8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in block in spawn_thread’
8ee5140d | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context’
2021-09-01T09:24:59 [I|app|8ee5140d] Rendering api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout
2021-09-01T09:24:59 [I|app|8ee5140d] Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 8.8ms | Allocations: 5924)
2021-09-01T09:24:59 [I|app|8ee5140d] Completed 500 Internal Server Error in 147ms (Views: 16.9ms | ActiveRecord: 12.8ms | Allocations: 26588)

4

Yes, that’s it.

Looks like you have a default organization and/or location set hammer defaults list should confirm it.

This breaks things when you’re retrieving jobs, which are not scoped into organizations and locations. This issue is being tracked as Bug #33007: Hammer Command Fails to List Job Invocation Details if Organization is Used - Foreman Remote Execution - Foreman, however even if it gets in, it will only get backported to 2.5 and maybe 2.4.

Until it gets fixed unsetting the defaults is the only way to make it work as far as I know.

1 Like
5

That fixed it, thank you - much appreciated.