Hammer user-group external create throws Internal Server Error

Problem:
When creating an external user-group, hammer throws an error. Although, I can successfully create the user-group and external user-group using the Web UI.

[root@acilformn001 ~]# hammer user-group external create \
>   --name foreman-auditor \
>   --auth-source acil389ds001 \
>   --user-group foreman-auditor
Could not create external user group:
  Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs.

Expected outcome:
Expecting hammer to create the external group.

Foreman and Proxy versions:
Foreman 3.4 with Katello 4.6

Foreman and Proxy plugin versions:
rubygem-foreman_puppet-4.0.1-1.fm3_4.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.6-1.fm3_4.el8.noarch
foreman-debug-3.4.0-1.el8.noarch
rubygem-foreman-tasks-7.0.0-1.fm3_4.el8.noarch
foreman-installer-3.4.0-1.el8.noarch
foreman-selinux-3.4.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
foreman-3.4.0-1.el8.noarch
rubygem-foreman_maintain-1.1.3-1.el8.noarch
foreman-cli-3.4.0-1.el8.noarch
foreman-installer-katello-3.4.0-1.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
rubygem-foreman_openscap-5.2.2-2.fm3_3.el8.noarch
foreman-service-3.4.0-1.el8.noarch
rubygem-hammer_cli_foreman-3.4.0-1.el8.noarch
foreman-dynflow-sidekiq-3.4.0-1.el8.noarch
rubygem-foreman_remote_execution-8.0.0-2.fm3_4.el8.noarch
foreman-proxy-3.4.0-1.el8.noarch
foreman-release-3.4.0-1.el8.noarch
foreman-postgresql-3.4.0-1.el8.noarch

Distribution and version:
Red Hat Enterprise Linux release 8.6 (Ootpa)
4.18.0-372.26.1.el8_6.x86_64

Other relevant data:
Output from production.log:

2022-10-20T15:12:47 [I|app|a18dab71] Started GET "/apidoc/apipie_checksum" for 192.168.0.17 at 2022-10-20 15:12:47 +0000
2022-10-20T15:12:47 [I|app|a18dab71] Processing by Apipie::ApipiesController#apipie_checksum as JSON
2022-10-20T15:12:47 [I|app|a18dab71]   Parameters: {"apipy"=>{}}
2022-10-20T15:12:47 [I|app|a18dab71]   Rendered /usr/share/gems/gems/apipie-rails-0.5.20/app/views/apipie/apipies/apipie_checksum.json.erb (Duration: 5.2ms | Allocations: 5660)
2022-10-20T15:12:47 [I|app|a18dab71] Completed 200 OK in 6ms (Views: 6.0ms | ActiveRecord: 0.0ms | Allocations: 6096)
2022-10-20T15:12:47 [I|app|1c2487ce] Started GET "/apidoc/v2.en.json" for 192.168.0.17 at 2022-10-20 15:12:47 +0000
2022-10-20T15:12:47 [I|app|1c2487ce] Processing by Apipie::ApipiesController#index as JSON
2022-10-20T15:12:47 [I|app|1c2487ce]   Parameters: {"version"=>"v2.en.json", "apipy"=>{}}
2022-10-20T15:12:50 [I|app|1c2487ce] Completed 200 OK in 3216ms (Views: 655.5ms | ActiveRecord: 0.0ms | Allocations: 2717715)
2022-10-20T15:12:51 [I|app|0950e784] Started GET "/api/locations?search=title+%3D+%22site1%22&per_page=1000&page=1" for 192.168.0.17 at 2022-10-20 15:12:51 +0000
2022-10-20T15:12:51 [I|app|0950e784] Processing by Api::V2::LocationsController#index as JSON
2022-10-20T15:12:51 [I|app|0950e784]   Parameters: {"search"=>"title = \"site1\"", "per_page"=>"1000", "page"=>"1", "apiv"=>"v2", "location"=>{}}
2022-10-20T15:12:51 [I|app|0950e784] Authorized user admin(Admin User)
2022-10-20T15:12:52 [I|app|0950e784]   Rendered api/v2/taxonomies/index.json.rabl within api/v2/layouts/index_layout (Duration: 4.8ms | Allocations: 1011)
2022-10-20T15:12:52 [I|app|0950e784]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 11.3ms | Allocations: 6959)
2022-10-20T15:12:52 [I|app|0950e784] Completed 200 OK in 215ms (Views: 26.6ms | ActiveRecord: 14.8ms | Allocations: 16262)
2022-10-20T15:12:52 [I|app|ac6811d7] Started GET "/katello/api/organizations?search=title+%3D+%22Customer%22&per_page=1000&page=1" for 192.168.0.17 at 2022-10-20 15:12:52 +0000
2022-10-20T15:12:52 [I|app|ac6811d7] Processing by Katello::Api::V2::OrganizationsController#index as JSON
2022-10-20T15:12:52 [I|app|ac6811d7]   Parameters: {"search"=>"title = \"Customer\"", "per_page"=>"1000", "page"=>"1", "api_version"=>"v2", "organization"=>{}}
2022-10-20T15:12:52 [I|app|ac6811d7] Authorized user admin(Admin User)
2022-10-20T15:12:52 [I|app|ac6811d7]   Rendered /usr/share/gems/gems/katello-4.6.0.rc2/app/views/katello/api/v2/organizations/index.json.rabl within api/v2/layouts/index_layout (Duration: 3.1ms | Allocations: 641)
2022-10-20T15:12:52 [I|app|ac6811d7]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 6.7ms | Allocations: 3403)
2022-10-20T15:12:52 [I|app|ac6811d7] Completed 200 OK in 224ms (Views: 38.7ms | ActiveRecord: 12.6ms | Allocations: 13057)
2022-10-20T15:12:52 [I|app|521566ab] Started GET "/api/usergroups?search=name+%3D+%22foreman-auditor%22&per_page=1000&page=1" for 192.168.0.17 at 2022-10-20 15:12:52 +0000
2022-10-20T15:12:52 [I|app|521566ab] Processing by Api::V2::UsergroupsController#index as JSON
2022-10-20T15:12:52 [I|app|521566ab]   Parameters: {"search"=>"name = \"foreman-auditor\"", "per_page"=>"1000", "page"=>"1", "apiv"=>"v2", "usergroup"=>{}}
2022-10-20T15:12:52 [I|app|521566ab] Authorized user admin(Admin User)
2022-10-20T15:12:52 [I|app|521566ab]   Rendered api/v2/usergroups/index.json.rabl within api/v2/layouts/index_layout (Duration: 7.1ms | Allocations: 6316)
2022-10-20T15:12:52 [I|app|521566ab]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 14.8ms | Allocations: 12446)
2022-10-20T15:12:52 [I|app|521566ab] Completed 200 OK in 200ms (Views: 14.7ms | ActiveRecord: 13.1ms | Allocations: 20962)
2022-10-20T15:12:52 [I|app|e7483ad6] Started GET "/api/auth_sources?search=name+%3D+%22acil389ds001%22&per_page=1000&page=1" for 192.168.0.17 at 2022-10-20 15:12:52 +0000
2022-10-20T15:12:52 [I|app|e7483ad6] Processing by Api::V2::AuthSourcesController#index as JSON
2022-10-20T15:12:52 [I|app|e7483ad6]   Parameters: {"search"=>"name = \"acil389ds001\"", "per_page"=>"1000", "page"=>"1", "apiv"=>"v2", "auth_source"=>{}}
2022-10-20T15:12:52 [I|app|e7483ad6] Authorized user admin(Admin User)
2022-10-20T15:12:52 [I|app|e7483ad6]   Rendered api/v2/auth_sources/index.json.rabl within api/v2/layouts/index_layout (Duration: 13.4ms | Allocations: 8369)
2022-10-20T15:12:52 [I|app|e7483ad6]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 21.5ms | Allocations: 14541)
2022-10-20T15:12:52 [I|app|e7483ad6] Completed 200 OK in 210ms (Views: 19.9ms | ActiveRecord: 15.6ms | Allocations: 23088)
2022-10-20T15:12:52 [I|app|121d3fbe] Started POST "/api/usergroups/4/external_usergroups" for 192.168.0.17 at 2022-10-20 15:12:52 +0000
2022-10-20T15:12:52 [I|app|121d3fbe] Processing by Api::V2::ExternalUsergroupsController#create as JSON
2022-10-20T15:12:52 [I|app|121d3fbe]   Parameters: {"location_id"=>2, "organization_id"=>1, "external_usergroup"=>{"name"=>"external-foreman-auditor", "auth_source_id"=>5}, "apiv"=>"v2", "usergroup_id"=>"4"}
2022-10-20T15:12:52 [I|app|121d3fbe] Authorized user admin(Admin User)
2022-10-20T15:12:53 [W|app|121d3fbe] Action failed
2022-10-20T15:12:53 [I|app|121d3fbe] Backtrace for 'Action failed' error (NoMethodError): undefined method `external_usergroups' for #<Location:0x0000563354d64218>
 121d3fbe | /usr/share/gems/gems/activemodel-6.1.6.1/lib/active_model/attribute_methods.rb:469:in `method_missing'
 121d3fbe | /usr/share/foreman/app/controllers/api/v2/external_usergroups_controller.rb:39:in `create'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/base.rb:228:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 121d3fbe | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 121d3fbe | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 121d3fbe | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 121d3fbe | /usr/share/gems/gems/audited-4.10.0/lib/audited/sweeper.rb:14:in `around'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 121d3fbe | /usr/share/gems/gems/audited-4.10.0/lib/audited/sweeper.rb:14:in `around'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:137:in `run_callbacks'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications.rb:203:in `block in instrument'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/notifications.rb:203:in `instrument'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 121d3fbe | /usr/share/gems/gems/activerecord-6.1.6.1/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/abstract_controller/base.rb:165:in `process'
 121d3fbe | /usr/share/gems/gems/actionview-6.1.6.1/lib/action_view/rendering.rb:39:in `process'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal.rb:190:in `dispatch'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_controller/metal.rb:254:in `dispatch'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/mapper.rb:49:in `serve'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:32:in `each'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/journey/router.rb:32:in `serve'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/routing/route_set.rb:842:in `call'
 121d3fbe | /usr/share/gems/gems/katello-4.6.0.rc2/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 121d3fbe | /usr/share/gems/gems/katello-4.6.0.rc2/lib/katello/middleware/event_daemon.rb:10:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 121d3fbe | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/static_dispatcher.rb:66:in `call'
 121d3fbe | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/extractor/recorder.rb:137:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 121d3fbe | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 121d3fbe | /usr/share/gems/gems/apipie-rails-0.5.20/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/tempfile_reaper.rb:15:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/etag.rb:27:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/conditional_get.rb:40:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/head.rb:12:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 121d3fbe | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:266:in `context'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/session/abstract/id.rb:260:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/cookies.rb:689:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 121d3fbe | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/rack/logger.rb:37:in `call_app'
 121d3fbe | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/rack/logger.rb:28:in `call'
 121d3fbe | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 121d3fbe | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 121d3fbe | /usr/share/gems/gems/katello-4.6.0.rc2/lib/katello/prevent_json_parsing.rb:12:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/method_override.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/runtime.rb:22:in `call'
 121d3fbe | /usr/share/gems/gems/activesupport-6.1.6.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/static.rb:24:in `call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/sendfile.rb:110:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 121d3fbe | /usr/share/gems/gems/actionpack-6.1.6.1/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 121d3fbe | /usr/share/gems/gems/secure_headers-6.3.4/lib/secure_headers/middleware.rb:11:in `call'
 121d3fbe | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/engine.rb:539:in `call'
 121d3fbe | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `public_send'
 121d3fbe | /usr/share/gems/gems/railties-6.1.6.1/lib/rails/railtie.rb:207:in `method_missing'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:74:in `block in call'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `each'
 121d3fbe | /usr/share/gems/gems/rack-2.2.4/lib/rack/urlmap.rb:58:in `call'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/configuration.rb:252:in `call'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/request.rb:77:in `block in handle_request'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/request.rb:76:in `handle_request'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/server.rb:441:in `process_client'
 121d3fbe | /usr/share/gems/gems/puma-5.6.4/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 121d3fbe | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-10-20T15:12:53 [I|app|121d3fbe]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 5.6ms | Allocations: 5750)
2022-10-20T15:12:53 [I|app|121d3fbe]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 10.8ms | Allocations: 11479)
2022-10-20T15:12:53 [I|app|121d3fbe] Completed 500 Internal Server Error in 262ms (Views: 12.5ms | ActiveRecord: 14.0ms | Allocations: 25097)
[root@acilformn001 ~]# hammer auth-source ldap list
---|--------------|--------------|------|-------
ID | NAME         | SERVER       | PORT | LDAPS?
---|--------------|--------------|------|-------
5  | acil389ds001 | acil389ds001 | 636  | yes
---|--------------|--------------|------|-------

[root@acilformn001 ~]# hammer user-group list
---|-----------------|------
ID | NAME            | ADMIN
---|-----------------|------
5  | foreman-auditor | no
---|-----------------|------

That seems to be an isssue in what we detect as a @nested_obj, it should be user group but for some reason it’s location. Is the user under which you execute it an admin?

As a very ugly workaround, you can use the UI to create the external group mapping.

I am running the hammer commands as root which is an admin user. I have only used the root user to build the Foreman server. I can create the group through the UI with no problems, however, I would rather do it through the command line to script it.

Yeah, that’s fair ask and I’m afraid a valid bug you found.

As first, sorry for reopenning this quite old thread. I’m running v. 3.7.1 and now I’m having the same issue. Was this bug fixed or is there some workaround please? Using GUI is sadly not an option for me.

Here is a part from production.log:

2024-01-03T10:11:54 [I|app|f5f7c6b4] Authorized user admin(Admin User)
2024-01-03T10:11:54 [I|app|f5f7c6b4]   Rendered api/v2/auth_sources/index.json.rabl within api/v2/layouts/index_layout (Duration: 16.1ms | Allocations: 3585)
2024-01-03T10:11:54 [I|app|f5f7c6b4]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 24.7ms | Allocations: 10272)
2024-01-03T10:11:54 [I|app|f5f7c6b4] Completed 200 OK in 208ms (Views: 22.4ms | ActiveRecord: 14.2ms | Allocations: 19044)
2024-01-03T10:11:54 [I|app|1e95b023] Started POST "/api/usergroups/10/external_usergroups" for 10.188.194.199 at 2024-01-03 10:11:54 +0100
2024-01-03T10:11:54 [I|app|1e95b023] Processing by Api::V2::ExternalUsergroupsController#create as JSON
2024-01-03T10:11:54 [I|app|1e95b023]   Parameters: {"organization_id"=>13, "external_usergroup"=>{"name"=>"Local-SysAdmins", "auth_source_id"=>4}, "apiv"=>"v2", "usergroup_id"=>"10"}
2024-01-03T10:11:54 [I|app|1e95b023] Authorized user admin(Admin User)
2024-01-03T10:11:54 [W|app|1e95b023] Action failed
2024-01-03T10:11:54 [I|app|1e95b023] Backtrace for 'Action failed' error (NoMethodError): undefined method `external_usergroups' for #<Organization:0x00007f24316bd918>
 1e95b023 | /usr/share/gems/gems/activemodel-6.1.7.6/lib/active_model/attribute_methods.rb:469:in `method_missing'
 1e95b023 | /usr/share/foreman/app/controllers/api/v2/external_usergroups_controller.rb:39:in `create'

Thank You for any kind of help.

What exact hammer command are you using?

On my box, it seems to work if I don’t pass organization to it in any way. Alternatively doing the same thing through api seems to work for me as well.

I’ve tested probably every possible combination for hammer user-group external create, e.g.:

hammer user-group external create --name 'Local_Admins' --auth-source-id 4 --user-group-id $ugid
hammer user-group external create --name 'Local_Admins' --auth-source-id 4 --user-group 'AD_SysAdmins'
hammer user-group external create --name 'Local_Admins' --auth-source-id 4 --user-group-id $ugid --organization-id $orgid

Every run fails with error:

Could not create external user group:
  Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs.

It doesn’t matter if there is some ORG or not, the result is still the same. All services was already resterted and health check is ok:

foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------

Mismatches test looks ok:

I’ve tested probably every possible combination for hammer user-group external create, e.g.:

Could you run each of these with -v -d after hammer (like hammer -v -d user-group external create --name 'Local_Admins' --auth-source-id 4 --user-group-id $ugid) and attach here the output as well as the relevant part of production.log so that we can see what exactly gets sent over?

Sure, here is complete stdout & stderr:

[ INFO 2024-01-03T14:19:21 Init] Initialization of Hammer CLI (3.7.0) has started...
[DEBUG 2024-01-03T14:19:21 Init] Running at ruby 2.7.8-p225
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli_config.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_admin.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_admin_logging_core.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_admin_logging_katello.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_ansible.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_bootdisk.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_discovery.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_kubevirt.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_openscap.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_remote_execution.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_tasks.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_templates.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_webhooks.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /etc/hammer/cli.modules.d/katello.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /root/.hammer/defaults.yml has been loaded
[ INFO 2024-01-03T14:19:21 Init] Configuration from the file /root/.hammer/cli.modules.d/foreman.yml has been loaded
[DEBUG 2024-01-03T14:19:22 SSLoptions] SSL options: {
    :ssl_ca_file => "/etc/pki/katello/certs/katello-server-ca.crt",
     :verify_ssl => true
}
[DEBUG 2024-01-03T14:19:22 API] Global headers: {
        :content_type => "application/json",
              :accept => "application/json;version=2",
    "Accept-Language" => "en"
}
[DEBUG 2024-01-03T14:19:22 API] Follow redirects: never
[DEBUG 2024-01-03T14:19:22 Connection] Registered: foreman
[ INFO 2024-01-03T14:19:22 API] Server: https://foreman0001.cz.corp
[ INFO 2024-01-03T14:19:22 API] GET /apidoc/apipie_checksum
[DEBUG 2024-01-03T14:19:22 API] Params: {}
[DEBUG 2024-01-03T14:19:22 API] Headers: {
    :accept => "application/json",
    :params => {}
}
[DEBUG 2024-01-03T14:19:22 API] Response: {
    "checksum" => "fad4f002d36dd92503cbb498976397a6d894b422"
}
[DEBUG 2024-01-03T14:19:22 API] Response headers: {
                                 :date => "Wed, 03 Jan 2024 13:19:22 GMT",
                               :server => "Apache",
                         :content_type => "application/json; charset=utf-8",
                      :apipie_checksum => "fad4f002d36dd92503cbb498976397a6d894b422",
                                 :etag => "W/\"ed58b680e0ec339948c80d540482cc3a\"",
                        :cache_control => "max-age=0, private, must-revalidate",
                         :x_request_id => "88c13549-846f-4a0f-87aa-d022576cb545",
                            :x_runtime => "0.015110",
            :strict_transport_security => "max-age=631139040; includeSubdomains",
                      :x_frame_options => "sameorigin",
               :x_content_type_options => "nosniff",
                     :x_xss_protection => "1; mode=block",
                   :x_download_options => "noopen",
    :x_permitted_cross_domain_policies => "none",
              :content_security_policy => "default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data:; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'",
                       :content_length => "59",
                                  :via => "1.1 foreman0001.cz.corp"
}
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::OptionSources] Added predefined options for HammerCLIForeman::Command: 
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::OptionSources for *all* except *nothing* on HammerCLIForeman::Command.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Fields] Added predefined options for HammerCLIForeman::ListCommand: [:fields]
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Fields for *all* except *nothing* on HammerCLIForeman::ListCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Fields] Added predefined options for HammerCLIForeman::InfoCommand: [:fields]
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Fields for *all* except *nothing* on HammerCLIForeman::InfoCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::UpdateCommon] Added predefined options for HammerCLIForeman::UpdateCommand: 
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::UpdateCommon for *all* except *nothing* on HammerCLIForeman::UpdateCommand.
[ INFO 2024-01-03T14:19:22 Modules] Extension module hammer_cli_foreman (3.7.0) loaded.
[ INFO 2024-01-03T14:19:22 Modules] Extension module hammer_cli_foreman_admin (1.1.0) loaded.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Saved block for HammerCLIForeman::Host::CreateCommand help definition:
	#<Proc:0x0000557b106dda18 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/hosts/help/interfaces.rb:6>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2024-01-03T14:19:22 #<Class:0x0000557b10cf3bc8>] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2024-01-03T14:19:22 #<Class:0x0000557b10cf3bc8>] Saved block for HammerCLIForeman::Host::CreateCommand help definition:
	#<Proc:0x0000557b10cf38d0 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/hosts/help/compute_resources.rb:45>
[ INFO 2024-01-03T14:19:22 Extensions] Applied #<Class:0x0000557b10cf3bc8> for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Saved block for HammerCLIForeman::Host::UpdateCommand help definition:
	#<Proc:0x0000557b106dda18 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/hosts/help/interfaces.rb:6>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2024-01-03T14:19:22 #<Class:0x0000557b104f13d0>] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2024-01-03T14:19:22 #<Class:0x0000557b104f13d0>] Saved block for HammerCLIForeman::Host::UpdateCommand help definition:
	#<Proc:0x0000557b104f11c8 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/hosts/help/compute_resources.rb:45>
[ INFO 2024-01-03T14:19:22 Extensions] Applied #<Class:0x0000557b104f13d0> for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanTasks::CommandExtensions::RecurringLogic] Added predefined options for HammerCLIForemanTasks::RecurringLogic::InfoCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanTasks::CommandExtensions::RecurringLogic] Called block for HammerCLIForemanTasks::RecurringLogic::InfoCommand output definition:
	#<Proc:0x0000557b10c82360 /usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.19/lib/hammer_cli_foreman_tasks/command_extensions/recurring_logic.rb:14>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForemanTasks::CommandExtensions::RecurringLogic for *all* except *nothing* on HammerCLIForemanTasks::RecurringLogic::InfoCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo] Added predefined options for HammerCLIForeman::Interface::InfoCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo] Called block for HammerCLIForeman::Interface::InfoCommand output definition:
	#<Proc:0x0000557b113162a8 /usr/share/gems/gems/hammer_cli_foreman_remote_execution-0.2.3/lib/hammer_cli_foreman_remote_execution/interface_extensions.rb:7>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo for *all* except *nothing* on HammerCLIForeman::Interface::InfoCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Added predefined options for HammerCLIForeman::Subnet::CreateCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Added option for HammerCLIForeman::Subnet::CreateCommand: {:switches=>"--prefix", :type=>"PREFIX", :description=>"Network prefix in CIDR notation (e.g. 64) for this subnet", :opts=>{}, :block=>nil}
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x0000557b106fc9b8 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:15>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x0000557b106fc918 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:21>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x0000557b106fc878 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:27>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x0000557b106fc7d8 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:33>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Subnet for *all* except *nothing* on HammerCLIForeman::Subnet::CreateCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Added predefined options for HammerCLIForeman::Subnet::UpdateCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Added option for HammerCLIForeman::Subnet::UpdateCommand: {:switches=>"--prefix", :type=>"PREFIX", :description=>"Network prefix in CIDR notation (e.g. 64) for this subnet", :opts=>{}, :block=>nil}
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x0000557b106fc9b8 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:15>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x0000557b106fc918 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:21>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x0000557b106fc878 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:27>
[DEBUG 2024-01-03T14:19:22 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x0000557b106fc7d8 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:33>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForeman::CommandExtensions::Subnet for *all* except *nothing* on HammerCLIForeman::Subnet::UpdateCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanRemoteExecution::SubnetExtensions] Added predefined options for HammerCLIForeman::Subnet::InfoCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanRemoteExecution::SubnetExtensions] Called block for HammerCLIForeman::Subnet::InfoCommand output definition:
	#<Proc:0x0000557b1164eb50 /usr/share/gems/gems/hammer_cli_foreman_remote_execution-0.2.3/lib/hammer_cli_foreman_remote_execution/subnet_extensions.rb:5>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForemanRemoteExecution::SubnetExtensions for *all* except *nothing* on HammerCLIForeman::Subnet::InfoCommand.
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanAnsible::CommandExtensions::JobTemplate] Added predefined options for HammerCLIForemanRemoteExecution::JobTemplate::InfoCommand: 
[DEBUG 2024-01-03T14:19:22 HammerCLIForemanAnsible::CommandExtensions::JobTemplate] Called block for HammerCLIForemanRemoteExecution::JobTemplate::InfoCommand output definition:
	#<Proc:0x0000557b113d9f50 /usr/share/gems/gems/hammer_cli_foreman_ansible-0.5.0/lib/hammer_cli_foreman_ansible/command_extensions/job_template.rb:12>
[ INFO 2024-01-03T14:19:22 Extensions] Applied HammerCLIForemanAnsible::CommandExtensions::JobTemplate for *all* except *nothing* on HammerCLIForemanRemoteExecution::JobTemplate::InfoCommand.
[ INFO 2024-01-03T14:19:22 Modules] Extension module hammer_cli_foreman_ansible (0.5.0) loaded.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_foreman_bootdisk (0.3.0) loaded.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_foreman_discovery (1.1.0) loaded.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_foreman_openscap (0.1.13) loaded.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_foreman_templates (0.2.0) loaded.
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForemanWebhooks::CommandExtensions::Webhook for [:before_print] only on HammerCLIForemanWebhooks::Webhook::InfoCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIForemanWebhooks::CommandExtensions::Webhook] Added option for HammerCLIForemanWebhooks::Webhook::CreateCommand: {:switches=>"--ssl-ca-certs", :type=>"PATH_TO_CA_FILE", :description=>"File containing X509 Certification Authorities concatenated in PEM format", :opts=>{:format=>#<HammerCLI::Options::Normalizers::File:0x0000557b11566f80>}, :block=>nil}
[DEBUG 2024-01-03T14:19:23 HammerCLIForemanWebhooks::CommandExtensions::Webhook] Added option for HammerCLIForemanWebhooks::Webhook::CreateCommand: {:switches=>"--http-headers", :type=>"HTTP_HEADERS", :description=>"", :opts=>{:format=>#<HammerCLI::Options::Normalizers::KeyValueList:0x0000557b11566ee0>}, :block=>nil}
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForemanWebhooks::CommandExtensions::Webhook for [:option, :request_params] only on HammerCLIForemanWebhooks::Webhook::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIForemanWebhooks::CommandExtensions::Webhook] Added option for HammerCLIForemanWebhooks::Webhook::UpdateCommand: {:switches=>"--ssl-ca-certs", :type=>"PATH_TO_CA_FILE", :description=>"File containing X509 Certification Authorities concatenated in PEM format", :opts=>{:format=>#<HammerCLI::Options::Normalizers::File:0x0000557b11566f80>}, :block=>nil}
[DEBUG 2024-01-03T14:19:23 HammerCLIForemanWebhooks::CommandExtensions::Webhook] Added option for HammerCLIForemanWebhooks::Webhook::UpdateCommand: {:switches=>"--http-headers", :type=>"HTTP_HEADERS", :description=>"", :opts=>{:format=>#<HammerCLI::Options::Normalizers::KeyValueList:0x0000557b11566ee0>}, :block=>nil}
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForemanWebhooks::CommandExtensions::Webhook for [:option, :request_params] only on HammerCLIForemanWebhooks::Webhook::UpdateCommand.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_foreman_webhooks (0.0.4) loaded.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::SyncCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::SyncCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::SyncCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::SyncCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Repository::ListCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Repository::ListCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Repository::ListCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Repository::ListCommand.
[ INFO 2024-01-03T14:19:23 HammerCLI::MainCommand] subcommand organization (HammerCLIForeman::Organization) was removed.
[ INFO 2024-01-03T14:19:23 HammerCLI::MainCommand] subcommand organization (HammerCLIKatello::Organization) was created.
[ INFO 2024-01-03T14:19:23 HammerCLI::MainCommand] subcommand alternate-content-source (HammerCLIKatello::AcsCommand) was created.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::HostErrata::ListCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostErrata::ListCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostErrata::ListCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::HostErrata::ListCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::HostSubscription::CreateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostSubscription::CreateCommand:
	#<Proc:0x0000557b1055aba0 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostSubscription::CreateCommand:
	#<Proc:0x0000557b1055a948 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::HostSubscription::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Host::CreateCommand:
	#<Proc:0x0000557b10509ae8 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Host::CreateCommand:
	#<Proc:0x0000557b104f3978 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Host::UpdateCommand:
	#<Proc:0x0000557b10509ae8 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Host::UpdateCommand:
	#<Proc:0x0000557b104f3978 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Hostgroup::CreateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Hostgroup::CreateCommand:
	#<Proc:0x0000557b10509ae8 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Hostgroup::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Hostgroup::CreateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Hostgroup::CreateCommand:
	#<Proc:0x0000557b104f3978 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Hostgroup::CreateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Hostgroup::UpdateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Hostgroup::UpdateCommand:
	#<Proc:0x0000557b10509ae8 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Hostgroup::UpdateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Hostgroup::UpdateCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Hostgroup::UpdateCommand:
	#<Proc:0x0000557b104f3978 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Hostgroup::UpdateCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIForeman::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::PingCommand::ForemanCommand: 
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForeman::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::PingCommand::ForemanCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::PingCommand::ForemanCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::Ping] Called block for HammerCLIForeman::PingCommand::ForemanCommand output definition:
	#<Proc:0x0000557b10517828 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/ping.rb:13>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::PingCommand::ForemanCommand.
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForeman::CommandExtensions::Ping for [:data] only on HammerCLIForeman::StatusCommand::ForemanCommand.
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIForeman::CommandExtensions::Status for [:data] only on HammerCLIForeman::StatusCommand::ForemanCommand.
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::StatusCommand::ForemanCommand: 
[DEBUG 2024-01-03T14:19:23 HammerCLIKatello::CommandExtensions::Ping] Called block for HammerCLIForeman::StatusCommand::ForemanCommand output definition:
	#<Proc:0x0000557b10517828 /usr/share/gems/gems/hammer_cli_katello-1.9.1/lib/hammer_cli_katello/command_extensions/ping.rb:13>
[ INFO 2024-01-03T14:19:23 Extensions] Applied HammerCLIKatello::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::StatusCommand::ForemanCommand.
[ INFO 2024-01-03T14:19:23 Modules] Extension module hammer_cli_katello (1.9.1) loaded.
[DEBUG 2024-01-03T14:19:23 Init] Using locale 'en'
[DEBUG 2024-01-03T14:19:23 Init] 'mo' files for locale domain 'hammer-cli' loaded from '/usr/share/gems/gems/hammer_cli-3.7.0/locale'
[DEBUG 2024-01-03T14:19:23 Init] 'mo' files for locale domain 'hammer-cli-foreman' loaded from '/usr/share/gems/gems/hammer_cli_foreman-3.7.0/locale'
[DEBUG 2024-01-03T14:19:23 Init] 'mo' files for locale domain 'hammer-cli-katello' loaded from '/usr/share/gems/gems/hammer_cli_katello-1.9.1/locale'
[ INFO 2024-01-03T14:19:23 HammerCLI::MainCommand] Called with options: {"option_verbose"=>2, "option_debug"=>true}
[DEBUG 2024-01-03T14:19:23 HammerCLIForeman::CommandExtensions::OptionSources] Called block for HammerCLIForeman::Command option sources:
	#<Proc:0x0000557b106d0458 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/option_sources.rb:6>
[ INFO 2024-01-03T14:19:23 HammerCLIForeman::Usergroup] Called with options: {}
[DEBUG 2024-01-03T14:19:23 HammerCLIForeman::CommandExtensions::OptionSources] Called block for HammerCLIForeman::Command option sources:
	#<Proc:0x0000557b106d0458 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/option_sources.rb:6>
[ INFO 2024-01-03T14:19:23 HammerCLIForeman::ExternalUsergroup] Called with options: {}
[DEBUG 2024-01-03T14:19:23 HammerCLIForeman::CommandExtensions::OptionSources] Called block for HammerCLIForeman::Command option sources:
	#<Proc:0x0000557b106d0458 /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/command_extensions/option_sources.rb:6>
[ INFO 2024-01-03T14:19:23 API] Server: https://foreman0001.cz.corp
[ INFO 2024-01-03T14:19:23 API] GET /api/auth_sources
[DEBUG 2024-01-03T14:19:23 API] Params: {
      :search => "name = \"MYORG_LDAP\"",
    :per_page => 1000,
        :page => 1
}
[DEBUG 2024-01-03T14:19:23 API] Headers: {
    :params => {
          :search => "name = \"MYORG_LDAP\"",
        :per_page => 1000,
            :page => 1
    }
}
[DEBUG 2024-01-03T14:19:23 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[DEBUG 2024-01-03T14:19:23 API] Response: {
       "total" => 4,
    "subtotal" => 1,
        "page" => 1,
    "per_page" => 1000,
      "search" => "name = \"MYORG_LDAP\"",
        "sort" => {
           "by" => nil,
        "order" => nil
    },
     "results" => [
        [0] {
               "created_at" => "2023-03-29 12:02:09 UTC",
               "updated_at" => "2024-01-03 10:39:31 UTC",
                       "id" => 4,
                     "type" => "AuthSourceLdap",
                     "name" => "MYORG_LDAP",
                "locations" => [
                [0] {
                             "id" => 10,
                           "name" => "Default Location",
                          "title" => "Default Location",
                    "description" => ""
                }
            ],
            "organizations" => [
                [0] {
                             "id" => 1,
                           "name" => "Default Organization",
                          "title" => "Default Organization",
                    "description" => ""
                },
                [1] {
                             "id" => 13,
                           "name" => "MYORG",
                          "title" => "MYORG",
                    "description" => "Default target organization"
                }
            ]
        }
    ]
}
[DEBUG 2024-01-03T14:19:23 API] Response headers: {
                                 :date => "Wed, 03 Jan 2024 13:19:23 GMT",
                               :server => "Apache",
                      :foreman_version => "3.7.1",
                  :foreman_api_version => "2",
         :foreman_current_organization => "; ANY",
             :foreman_current_location => "; ANY",
                         :content_type => "application/json; charset=utf-8",
                      :apipie_checksum => "fad4f002d36dd92503cbb498976397a6d894b422",
                                 :etag => "W/\"c0a7ffe19c2f367c74bb630dd53925fa\"",
                        :cache_control => "max-age=0, private, must-revalidate",
                         :x_request_id => "b8df19a7-eac1-45f6-be37-651011d9872e",
                            :x_runtime => "0.211789",
            :strict_transport_security => "max-age=631139040; includeSubdomains",
                      :x_frame_options => "sameorigin",
               :x_content_type_options => "nosniff",
                     :x_xss_protection => "1; mode=block",
                   :x_download_options => "noopen",
    :x_permitted_cross_domain_policies => "none",
              :content_security_policy => "default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data:; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'",
                       :content_length => "590",
                           :set_cookie => [
        [0] "_session_id=9362a911c209e7d09cebd438e82182f4; path=/; secure; HttpOnly; SameSite=Lax"
    ],
                                  :via => "1.1 foreman0001.cz.corp"
}
[ INFO 2024-01-03T14:19:23 HammerCLIForeman::ExternalUsergroup::CreateCommand] Called with options: {"option_organization_id"=>13, "option_usergroup_id"=>"10", "option_name"=>"Local_Admins", "option_auth_source_name"=>"MYORG_LDAP", "option_auth_source_id"=>4}
[ INFO 2024-01-03T14:19:23 API] Server: https://foreman0001.cz.corp
[ INFO 2024-01-03T14:19:23 API] POST /api/usergroups/10/external_usergroups
[DEBUG 2024-01-03T14:19:23 API] Params: {
       "organization_id" => 13,
    "external_usergroup" => {
                  "name" => "Local_Admins",
        "auth_source_id" => 4
    }
}
[DEBUG 2024-01-03T14:19:23 API] Headers: {}
[DEBUG 2024-01-03T14:19:23 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[ERROR 2024-01-03T14:19:23 API] 500 Internal Server Error
[DEBUG 2024-01-03T14:19:23 API] {
    "error" => {
        "message" => "Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs."
    }
}
[DEBUG 2024-01-03T14:19:23 Exception] Using exception handler HammerCLIForeman::ExceptionHandler#handle_internal_error
[ERROR 2024-01-03T14:19:23 Exception] Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs.
Could not create external user group:
  Internal Server Error: the server was unable to finish the request. This may be caused by unavailability of some required service, incorrect API call or a server-side bug. There may be more information in the server's logs.
[ERROR 2024-01-03T14:19:23 Exception] 

RestClient::InternalServerError (500 Internal Server Error):
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:353:in `block in rest_client_call_block'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:834:in `process_result'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
    /usr/share/ruby/net/http.rb:933:in `start'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:69:in `post'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:185:in `call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/resource.rb:21:in `call'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/share/gems/gems/hammer_cli_foreman-3.7.0/lib/hammer_cli_foreman/commands.rb:180:in `send_request'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.7.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:140:in `run'
    /usr/share/gems/gems/hammer_cli-3.7.0/bin/hammer:147:in `<top (required)>'
    /bin/hammer:23:in `load'
    /bin/hammer:23:in `<main>'

Appropriate part of production.log:

2024-01-03T14:38:47 [I|app|637c5ab7] Started GET "/apidoc/apipie_checksum" for 10.188.194.199 at 2024-01-03 14:38:47 +0100
2024-01-03T14:38:47 [I|app|637c5ab7] Processing by Apipie::ApipiesController#apipie_checksum as JSON
2024-01-03T14:38:47 [I|app|637c5ab7]   Parameters: {"apipy"=>{}}
2024-01-03T14:38:47 [I|app|637c5ab7]   Rendered /usr/share/gems/gems/apipie-rails-1.1.0/app/views/apipie/apipies/apipie_checksum.json.erb (Duration: 6.4ms | Allocations: 6126)
2024-01-03T14:38:47 [I|app|637c5ab7] Completed 200 OK in 8ms (Views: 7.5ms | ActiveRecord: 0.0ms | Allocations: 6661)
2024-01-03T14:38:48 [I|app|7cbab1a4] Started GET "/api/auth_sources?search=name+%3D+%22MYORG_LDAP%22&per_page=1000&page=1" for 10.188.194.199 at 2024-01-03 14:38:48 +0100
2024-01-03T14:38:48 [I|app|7cbab1a4] Processing by Api::V2::AuthSourcesController#index as JSON
2024-01-03T14:38:48 [I|app|7cbab1a4]   Parameters: {"search"=>"name = \"MYORG_LDAP\"", "per_page"=>"1000", "page"=>"1", "apiv"=>"v2", "auth_source"=>{}}
2024-01-03T14:38:48 [I|app|7cbab1a4] Authorized user admin(Admin User)
2024-01-03T14:38:48 [I|app|7cbab1a4]   Rendered api/v2/auth_sources/index.json.rabl within api/v2/layouts/index_layout (Duration: 15.9ms | Allocations: 8992)
2024-01-03T14:38:48 [I|app|7cbab1a4]   Rendered layout api/v2/layouts/index_layout.json.erb (Duration: 22.2ms | Allocations: 15616)
2024-01-03T14:38:48 [I|app|7cbab1a4] Completed 200 OK in 201ms (Views: 20.7ms | ActiveRecord: 12.6ms | Allocations: 23789)
2024-01-03T14:38:48 [I|app|9a54e361] Started POST "/api/usergroups/10/external_usergroups" for 10.188.194.199 at 2024-01-03 14:38:48 +0100
2024-01-03T14:38:48 [I|app|9a54e361] Processing by Api::V2::ExternalUsergroupsController#create as JSON
2024-01-03T14:38:48 [I|app|9a54e361]   Parameters: {"organization_id"=>13, "external_usergroup"=>{"name"=>"Local_Admins", "auth_source_id"=>4}, "apiv"=>"v2", "usergroup_id"=>"10"}
2024-01-03T14:38:49 [I|app|9a54e361] Authorized user admin(Admin User)
2024-01-03T14:38:49 [W|app|9a54e361] Action failed
2024-01-03T14:38:49 [I|app|9a54e361] Backtrace for 'Action failed' error (NoMethodError): undefined method `external_usergroups' for #<Organization:0x00007faf31571858>
 9a54e361 | /usr/share/gems/gems/activemodel-6.1.7.6/lib/active_model/attribute_methods.rb:469:in `method_missing'
 9a54e361 | /usr/share/foreman/app/controllers/api/v2/external_usergroups_controller.rb:39:in `create'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:228:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rendering.rb:30:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 9a54e361 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 9a54e361 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 9a54e361 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 9a54e361 | /usr/share/gems/gems/audited-5.3.3/lib/audited/sweeper.rb:16:in `around'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 9a54e361 | /usr/share/gems/gems/audited-5.3.3/lib/audited/sweeper.rb:16:in `around'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:137:in `run_callbacks'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/callbacks.rb:41:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/rescue.rb:22:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `block in instrument'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/notifications.rb:203:in `instrument'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 9a54e361 | /usr/share/gems/gems/activerecord-6.1.7.6/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/abstract_controller/base.rb:165:in `process'
 9a54e361 | /usr/share/gems/gems/actionview-6.1.7.6/lib/action_view/rendering.rb:39:in `process'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:190:in `dispatch'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_controller/metal.rb:254:in `dispatch'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/mapper.rb:49:in `serve'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `each'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/journey/router.rb:32:in `serve'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/routing/route_set.rb:842:in `call'
 9a54e361 | /usr/share/gems/gems/katello-4.9.2/lib/katello/middleware/organization_created_enforcer.rb:18:in `call'
 9a54e361 | /usr/share/gems/gems/katello-4.9.2/lib/katello/middleware/event_daemon.rb:10:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 9a54e361 | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/static_dispatcher.rb:68:in `call'
 9a54e361 | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/extractor/recorder.rb:139:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/static.rb:24:in `call'
 9a54e361 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 9a54e361 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 9a54e361 | /usr/share/gems/gems/apipie-rails-1.1.0/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/tempfile_reaper.rb:15:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/etag.rb:27:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/conditional_get.rb:40:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/head.rb:12:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 9a54e361 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/session/abstract/id.rb:266:in `context'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/session/abstract/id.rb:260:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/callbacks.rb:98:in `run_callbacks'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 9a54e361 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:37:in `call_app'
 9a54e361 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/rack/logger.rb:28:in `call'
 9a54e361 | /usr/share/gems/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 9a54e361 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 9a54e361 | /usr/share/gems/gems/request_store-1.5.1/lib/request_store/middleware.rb:19:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 9a54e361 | /usr/share/gems/gems/katello-4.9.2/lib/katello/prevent_json_parsing.rb:12:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/method_override.rb:24:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/runtime.rb:22:in `call'
 9a54e361 | /usr/share/gems/gems/activesupport-6.1.7.6/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/executor.rb:14:in `call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/sendfile.rb:110:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 9a54e361 | /usr/share/gems/gems/actionpack-6.1.7.6/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 9a54e361 | /usr/share/gems/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 9a54e361 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/engine.rb:539:in `call'
 9a54e361 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `public_send'
 9a54e361 | /usr/share/gems/gems/railties-6.1.7.6/lib/rails/railtie.rb:207:in `method_missing'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:74:in `block in call'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:58:in `each'
 9a54e361 | /usr/share/gems/gems/rack-2.2.7/lib/rack/urlmap.rb:58:in `call'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/configuration.rb:270:in `call'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/request.rb:98:in `block in handle_request'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/request.rb:97:in `handle_request'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/server.rb:431:in `process_client'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/server.rb:233:in `block in run'
 9a54e361 | /usr/share/gems/gems/puma-6.2.2/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 9a54e361 | /usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2024-01-03T14:38:49 [I|app|9a54e361]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 5.1ms | Allocations: 6249)
2024-01-03T14:38:49 [I|app|9a54e361]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 8.7ms | Allocations: 12425)
2024-01-03T14:38:49 [I|app|9a54e361] Completed 500 Internal Server Error in 222ms (Views: 9.3ms | ActiveRecord: 10.1ms | Allocations: 26423)

Hrm, wonder where it comes from. Would an api-only workaround be ok for you? Would the following work?

cat <<EOF | curl -X POST -H 'Content-Type: application/json' -d@- -u $username:$pass https://$foreman_fqdn/api/v2/usergroups/$ugid/external_usergroups
{ "external_usergroup": { "name": "Local_Admins", "auth_source_id": 4 } }
EOF
2 Likes

@kubik256, I suspect you have some defaults configured for hammer. You can list them with hammer defaults list. If there is a default organization in the output, it will be used as a default parameter for the command. Removing this with hammer defaults delete or temporarily with hammer --no-use-defaults should fix the issue with the command.

API workaround works like a charm :slightly_smiling_face:
Awesome, thank You!

There are no Hammer defaults set - except organization_id | 1.

I suspect two things:

  1. I have two organizations and here I’m working with second one - non default.
  2. This test server already get two major version upgrades.
    ,
    I’ll do complete reinstall from scratch and shrink organizations to single default org, if it helps I’ll send a result. But it will take a while. For now thanks a lot for workaround and have a nice day!
1 Like

UPDATE: On clean installation (Foreman 3.9 + Katello 4.11) and single ORG issue is still present.
Workaround from @aruzicka still works.