Can't create new users in Foreman

Problem:
When I try to create a new user in Foreman, the UI “hangs” (i.e. never responds) on the action, and the new user will not be created. However, the UI still allows to navigate normally, and perform other tasks.
On the server side, a “rails” process starts running at 100% flat out.
The system does not recover on it’s own.
Reboot doesn’t fix the issue. (however it stops that hanging “rails” processes)
I can reproduce the problem every time.

If I try to create several users, for each attempt, the server will start a new “rails” process, and all of them will run at 100%, until there are no more CPU resources left.

I wasn’t able to find any relevant information in the logs. Where should I look for?

Expected outcome:
New user account should be created immediately.

Foreman and Proxy versions:
3.5.1

Foreman and Proxy plugin versions:
katello 4.7.2
foreman-tasks 7.1.1.
foreman_remote_execution 8.2.0

Distribution and version:
Rocky Linux 8.7

Other relevant data:

1 attempt to create a new user:
image

4 attempts to create a new user:

Dear community, I’m stuck at this point.
Is there any advice, how to troubleshoot the server?
This installation is two weeks old.

When using the “hammer” CLI, the result is the same.
There is no workaround.

# hammer user create --ask-password true --auth-source INTERNAL --firstname "test" --lastname "user" --login "testuser1" --mail "test@my.org"
Could not create the user:
  Error: Timed out reading data from server

After several minutes, the “hammer” command times out.
And also, the “rails” command runs flat out at 100% and it never stops.

However, after a while, there will be a 2nd “rails” process running at 100%.
That’s slightly different when using “hammer” compared to using the Web UI.

I also tried to use “psql” to directly create a new user record in the “users” table (using SQL language), with the idea that I will be able to modify it in the Foreman UI.
I was able to create a new user record directly in the Postgres database, but when I try to use the Foreman UI to modify the user, again, the UI hangs, the command times out, and the “rails” process runs forever on 100% CPU load.

No solution, no workaround found yet.

I noticed that “hammer” has a “--debug” option.
So, I reran the command to get more data.

I noticed there is a call stack at the end of the output:

RestClient::Exceptions::ReadTimeout (Timed out reading data from server):
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:751:in `rescue in transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:651:in `transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:69:in `post'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:185:in `call'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/resource.rb:21:in `call'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/commands.rb:180:in `send_request'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/bin/hammer:147:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

This is the full output (anonymized):

$ hammer --debug user create --login "testuser" --password "password123" --mail "testuser1@my.org" --auth-source-id 1 
[ INFO 2023-02-20T17:11:13 Init] Initialization of Hammer CLI (3.5.0) has started...
[DEBUG 2023-02-20T17:11:13 Init] Running at ruby 2.7.6-p219
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /etc/hammer/cli_config.yml has been loaded
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman.yml has been loaded
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_remote_execution.yml has been loaded
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /etc/hammer/cli.modules.d/foreman_tasks.yml has been loaded
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /etc/hammer/cli.modules.d/katello.yml has been loaded
[ INFO 2023-02-20T17:11:13 Init] Configuration from the file /home/admin.tfc/.hammer/cli.modules.d/foreman.yml has been loaded
[DEBUG 2023-02-20T17:11:13 SSLoptions] SSL options: {
    :ssl_ca_file => "/etc/pki/katello/certs/katello-server-ca.crt",
     :verify_ssl => true
}
[DEBUG 2023-02-20T17:11:13 API] Global headers: {
        :content_type => "application/json",
              :accept => "application/json;version=2",
    "Accept-Language" => "en"
}
[DEBUG 2023-02-20T17:11:13 API] Follow redirects: never
[DEBUG 2023-02-20T17:11:13 Connection] Registered: foreman
[ INFO 2023-02-20T17:11:13 API] Server: https://foreman.my.org
[ INFO 2023-02-20T17:11:13 API] GET /apidoc/apipie_checksum
[DEBUG 2023-02-20T17:11:13 API] Params: {}
[DEBUG 2023-02-20T17:11:13 API] Headers: {
    :accept => "application/json",
    :params => {}
}
[DEBUG 2023-02-20T17:11:13 API] Response: {
    "checksum" => "4e1243bd22c66e76c2ba9eddc1f91394e57f9f83"
}
[DEBUG 2023-02-20T17:11:13 API] Response headers: {
                                 :date => "Mon, 20 Feb 2023 16:11:13 GMT",
                               :server => "Apache",
                         :content_type => "application/json; charset=utf-8",
                      :apipie_checksum => "4e1243bd22c66e76c2ba9eddc1f91394e57f9f83",
                                 :etag => "W/\"dba7673010f19a94af4345453005933f\"",
                        :cache_control => "max-age=0, private, must-revalidate",
                         :x_request_id => "19e45a79-112c-487b-9f2e-2d7842d5f6ba",
                            :x_runtime => "0.007377",
            :strict_transport_security => "max-age=631139040; includeSubdomains",
                      :x_frame_options => "sameorigin",
               :x_content_type_options => "nosniff",
                     :x_xss_protection => "1; mode=block",
                   :x_download_options => "noopen",
    :x_permitted_cross_domain_policies => "none",
              :content_security_policy => "default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data:; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'",
                                  :via => "1.1 foreman.my.org",
                    :transfer_encoding => "chunked"
}
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::OptionSources] Added predefined options for HammerCLIForeman::Command: 
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::OptionSources for *all* except *nothing* on HammerCLIForeman::Command.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Fields] Added predefined options for HammerCLIForeman::ListCommand: [:fields]
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Fields for *all* except *nothing* on HammerCLIForeman::ListCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Fields] Added predefined options for HammerCLIForeman::InfoCommand: [:fields]
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Fields for *all* except *nothing* on HammerCLIForeman::InfoCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::UpdateCommon] Added predefined options for HammerCLIForeman::UpdateCommand: 
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::UpdateCommon for *all* except *nothing* on HammerCLIForeman::UpdateCommand.
[ INFO 2023-02-20T17:11:13 Modules] Extension module hammer_cli_foreman (3.5.0) loaded.
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanTasks::CommandExtensions::RecurringLogic] Added predefined options for HammerCLIForemanTasks::RecurringLogic::InfoCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanTasks::CommandExtensions::RecurringLogic] Called block for HammerCLIForemanTasks::RecurringLogic::InfoCommand output definition:
	#<Proc:0x00005631767c9fb8 /usr/share/gems/gems/hammer_cli_foreman_tasks-0.0.18/lib/hammer_cli_foreman_tasks/command_extensions/recurring_logic.rb:14>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForemanTasks::CommandExtensions::RecurringLogic for *all* except *nothing* on HammerCLIForemanTasks::RecurringLogic::InfoCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo] Added predefined options for HammerCLIForeman::Interface::InfoCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo] Called block for HammerCLIForeman::Interface::InfoCommand output definition:
	#<Proc:0x0000563176b80300 /usr/share/gems/gems/hammer_cli_foreman_remote_execution-0.2.2/lib/hammer_cli_foreman_remote_execution/interface_extensions.rb:7>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForemanRemoteExecution::InterfaceExtensionsInfo for *all* except *nothing* on HammerCLIForeman::Interface::InfoCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Added predefined options for HammerCLIForeman::Subnet::CreateCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Added option for HammerCLIForeman::Subnet::CreateCommand: {:switches=>"--prefix", :type=>"PREFIX", :description=>"Network prefix in CIDR notation (e.g. 64) for this subnet", :opts=>{}, :block=>nil}
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x00005631752bd1b0 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:15>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x00005631752bd110 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:21>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x00005631752bd070 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:27>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::CreateCommand:
	#<Proc:0x00005631752bcfa8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:33>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Subnet for *all* except *nothing* on HammerCLIForeman::Subnet::CreateCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Added predefined options for HammerCLIForeman::Subnet::UpdateCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Added option for HammerCLIForeman::Subnet::UpdateCommand: {:switches=>"--prefix", :type=>"PREFIX", :description=>"Network prefix in CIDR notation (e.g. 64) for this subnet", :opts=>{}, :block=>nil}
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x00005631752bd1b0 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:15>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x00005631752bd110 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:21>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x00005631752bd070 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:27>
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Subnet] Called option family block for HammerCLIForeman::Subnet::UpdateCommand:
	#<Proc:0x00005631752bcfa8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/subnet.rb:33>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Subnet for *all* except *nothing* on HammerCLIForeman::Subnet::UpdateCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanRemoteExecution::SubnetExtensions] Added predefined options for HammerCLIForeman::Subnet::InfoCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForemanRemoteExecution::SubnetExtensions] Called block for HammerCLIForeman::Subnet::InfoCommand output definition:
	#<Proc:0x00005631767cb160 /usr/share/gems/gems/hammer_cli_foreman_remote_execution-0.2.2/lib/hammer_cli_foreman_remote_execution/subnet_extensions.rb:5>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForemanRemoteExecution::SubnetExtensions for *all* except *nothing* on HammerCLIForeman::Subnet::InfoCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::AddLifecycleEnvironmentCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::RemoveLifecycleEnvironmentCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Capsule::Content::SyncCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::SyncCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Capsule::Content::SyncCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Capsule::Content::SyncCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::Repository::ListCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Repository::ListCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:13 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::Repository::ListCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::Repository::ListCommand.
[ INFO 2023-02-20T17:11:13 HammerCLI::MainCommand] subcommand organization (HammerCLIForeman::Organization) was removed.
[ INFO 2023-02-20T17:11:13 HammerCLI::MainCommand] subcommand organization (HammerCLIKatello::Organization) was created.
[ INFO 2023-02-20T17:11:13 HammerCLI::MainCommand] subcommand alternate-content-source (HammerCLIKatello::AcsCommand) was created.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Saved block for HammerCLIForeman::Host::CreateCommand help definition:
	#<Proc:0x0000563175311580 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/hosts/help/interfaces.rb:6>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2023-02-20T17:11:13 #<Class:0x0000563177005b50>] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2023-02-20T17:11:13 #<Class:0x0000563177005b50>] Saved block for HammerCLIForeman::Host::CreateCommand help definition:
	#<Proc:0x0000563177005a60 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/hosts/help/compute_resources.rb:45>
[ INFO 2023-02-20T17:11:13 Extensions] Applied #<Class:0x0000563177005b50> for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2023-02-20T17:11:13 HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces] Saved block for HammerCLIForeman::Host::UpdateCommand help definition:
	#<Proc:0x0000563175311580 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/hosts/help/interfaces.rb:6>
[ INFO 2023-02-20T17:11:13 Extensions] Applied HammerCLIForeman::CommandExtensions::Hosts::Help::Interfaces for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2023-02-20T17:11:13 #<Class:0x00005631772a6c88>] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2023-02-20T17:11:13 #<Class:0x00005631772a6c88>] Saved block for HammerCLIForeman::Host::UpdateCommand help definition:
	#<Proc:0x00005631772a6be8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/hosts/help/compute_resources.rb:45>
[ INFO 2023-02-20T17:11:13 Extensions] Applied #<Class:0x00005631772a6c88> for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::HostErrata::ListCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostErrata::ListCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostErrata::ListCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::HostErrata::ListCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Added predefined options for HammerCLIKatello::HostSubscription::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostSubscription::CreateCommand:
	#<Proc:0x00005631767922c0 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:8>
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::LifecycleEnvironment] Called option family block for HammerCLIKatello::HostSubscription::CreateCommand:
	#<Proc:0x0000563176792220 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/lifecycle_environment.rb:22>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::LifecycleEnvironment for *all* except *nothing* on HammerCLIKatello::HostSubscription::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Host::CreateCommand:
	#<Proc:0x000056317678d388 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Host::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Host::CreateCommand:
	#<Proc:0x00005631767872f8 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Host::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Host::UpdateCommand:
	#<Proc:0x000056317678d388 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Host::UpdateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Host::UpdateCommand:
	#<Proc:0x00005631767872f8 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Host::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Hostgroup::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Hostgroup::CreateCommand:
	#<Proc:0x000056317678d388 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Hostgroup::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Hostgroup::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Hostgroup::CreateCommand:
	#<Proc:0x00005631767872f8 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Hostgroup::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Added predefined options for HammerCLIForeman::Hostgroup::UpdateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::ContentSource] Called option family block for HammerCLIForeman::Hostgroup::UpdateCommand:
	#<Proc:0x000056317678d388 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/content_source.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::ContentSource for *all* except *nothing* on HammerCLIForeman::Hostgroup::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Added predefined options for HammerCLIForeman::Hostgroup::UpdateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::KickstartRepository] Called option family block for HammerCLIForeman::Hostgroup::UpdateCommand:
	#<Proc:0x00005631767872f8 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/kickstart_repository.rb:4>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::KickstartRepository for *all* except *nothing* on HammerCLIForeman::Hostgroup::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::PingCommand::ForemanCommand: 
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIForeman::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::PingCommand::ForemanCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::PingCommand::ForemanCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::Ping] Called block for HammerCLIForeman::PingCommand::ForemanCommand output definition:
	#<Proc:0x000056317678ed78 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/ping.rb:13>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::PingCommand::ForemanCommand.
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIForeman::CommandExtensions::Ping for [:data] only on HammerCLIForeman::StatusCommand::ForemanCommand.
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIForeman::CommandExtensions::Status for [:data] only on HammerCLIForeman::StatusCommand::ForemanCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::Ping] Added predefined options for HammerCLIForeman::StatusCommand::ForemanCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIKatello::CommandExtensions::Ping] Called block for HammerCLIForeman::StatusCommand::ForemanCommand output definition:
	#<Proc:0x000056317678ed78 /usr/share/gems/gems/hammer_cli_katello-1.7.2/lib/hammer_cli_katello/command_extensions/ping.rb:13>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIKatello::CommandExtensions::Ping for *all* except *nothing* on HammerCLIForeman::StatusCommand::ForemanCommand.
[ INFO 2023-02-20T17:11:14 Modules] Extension module hammer_cli_katello (1.7.2) loaded.
[DEBUG 2023-02-20T17:11:14 Init] Using locale 'en'
[DEBUG 2023-02-20T17:11:14 Init] 'mo' files for locale domain 'hammer-cli' loaded from '/usr/share/gems/gems/hammer_cli-3.5.0/locale'
[DEBUG 2023-02-20T17:11:14 Init] 'mo' files for locale domain 'hammer-cli-foreman' loaded from '/usr/share/gems/gems/hammer_cli_foreman-3.5.0/locale'
[DEBUG 2023-02-20T17:11:14 Init] 'mo' files for locale domain 'hammer-cli-katello' loaded from '/usr/share/gems/gems/hammer_cli_katello-1.7.2/locale'
[ INFO 2023-02-20T17:11:14 HammerCLI::MainCommand] Called with options: {"option_debug"=>true}
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Added predefined options for HammerCLIForeman::User::CreateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Added option for HammerCLIForeman::User::CreateCommand: {:switches=>"--ask-password", :type=>"ASK_PW", :description=>" ", :opts=>{:format=>#<HammerCLI::Options::Normalizers::Bool:0x00005631752bfbe0>}, :block=>nil}
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Called option family block for HammerCLIForeman::User::CreateCommand:
	#<Proc:0x00005631752bfa28 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/user.rb:11>
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Called option family block for HammerCLIForeman::User::CreateCommand:
	#<Proc:0x00005631752bf7a8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/user.rb:15>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIForeman::CommandExtensions::User for *all* except *nothing* on HammerCLIForeman::User::CreateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Added predefined options for HammerCLIForeman::User::UpdateCommand: 
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Added option for HammerCLIForeman::User::UpdateCommand: {:switches=>"--ask-password", :type=>"ASK_PW", :description=>" ", :opts=>{:format=>#<HammerCLI::Options::Normalizers::Bool:0x00005631752bfbe0>}, :block=>nil}
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Called option family block for HammerCLIForeman::User::UpdateCommand:
	#<Proc:0x00005631752bfa28 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/user.rb:11>
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Called option family block for HammerCLIForeman::User::UpdateCommand:
	#<Proc:0x00005631752bf7a8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/user.rb:15>
[ INFO 2023-02-20T17:11:14 Extensions] Applied HammerCLIForeman::CommandExtensions::User for *all* except *nothing* on HammerCLIForeman::User::UpdateCommand.
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::OptionSources] Called block for HammerCLIForeman::Command option sources:
	#<Proc:0x000056317533bad8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/option_sources.rb:6>
[ INFO 2023-02-20T17:11:14 HammerCLIForeman::User] Called with options: {}
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::OptionSources] Called block for HammerCLIForeman::Command option sources:
	#<Proc:0x000056317533bad8 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/option_sources.rb:6>
[DEBUG 2023-02-20T17:11:14 HammerCLIForeman::CommandExtensions::User] Called block for HammerCLIForeman::User::UpdateCommand option sources:
	#<Proc:0x00005631752bfaf0 /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/command_extensions/user.rb:6>
[ INFO 2023-02-20T17:11:14 HammerCLIForeman::User::CreateCommand] Called with options: {"option_login"=>"testuser", "option_mail"=>"testuser1@my.org", "option_password"=>"***", "option_auth_source_id"=>1}
[ INFO 2023-02-20T17:11:14 API] Server: https://foreman.my.org
[ INFO 2023-02-20T17:11:14 API] POST /api/users
[DEBUG 2023-02-20T17:11:14 API] Params: {
    "user" => {
                 "login" => "testuser",
                  "mail" => "testuser1@my.org",
              "password" => "***",
        "auth_source_id" => 1
    }
}
[DEBUG 2023-02-20T17:11:14 API] Headers: {}
[DEBUG 2023-02-20T17:11:14 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[ERROR 2023-02-20T17:13:14 API] Timed out reading data from server
[DEBUG 2023-02-20T17:13:14 API] ""
[DEBUG 2023-02-20T17:13:14 Exception] Using exception handler HammerCLIForeman::ExceptionHandler#handle_general_exception
[ERROR 2023-02-20T17:13:14 Exception] Error: Timed out reading data from server
Could not create the user:
  Error: Timed out reading data from server
[ERROR 2023-02-20T17:13:14 Exception] 

RestClient::Exceptions::ReadTimeout (Timed out reading data from server):
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:751:in `rescue in transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:651:in `transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:69:in `post'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/api.rb:185:in `call'
    /usr/share/gems/gems/apipie-bindings-0.5.0/lib/apipie_bindings/resource.rb:21:in `call'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/share/gems/gems/hammer_cli_foreman-3.5.0/lib/hammer_cli_foreman/commands.rb:180:in `send_request'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
    /usr/share/gems/gems/hammer_cli-3.5.0/bin/hammer:147:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

it seems like the client does not reveal any relevant debug data.
further debugging will have to be done on the server side.

I have enabled all debugging I could find in /etc/foreman/settings.yaml (except :sql:), particularly telemetry.logger.

This is what I am getting without :sql: logging:

2023-02-20T20:08:02 [D|dyn|] Executor heartbeat
2023-02-20T20:08:17 [D|dyn|] Executor heartbeat
2023-02-20T20:08:22 [I|app|9709ac29] Started GET "/apidoc/apipie_checksum" for 10.212.2.31 at 2023-02-20 20:08:22 +0100
2023-02-20T20:08:22 [D|tel|9709ac29] Incrementing counter fm_rails_activerecord_instances by 1.00 {:class=>"User"}
2023-02-20T20:08:22 [D|tel|9709ac29] Incrementing counter fm_rails_activerecord_instances by 1.00 {:class=>"AuthSource"}
2023-02-20T20:08:22 [D|per|9709ac29] Current user set to foreman_admin (admin)
2023-02-20T20:08:22 [I|app|9709ac29] Processing by Apipie::ApipiesController#apipie_checksum as JSON
2023-02-20T20:08:22 [I|app|9709ac29]   Parameters: {"apipy"=>{}}
2023-02-20T20:08:22 [D|app|9709ac29]   Rendering /usr/share/gems/gems/apipie-rails-0.8.2/app/views/apipie/apipies/apipie_checksum.json.erb
2023-02-20T20:08:24 [I|app|9709ac29]   Rendered /usr/share/gems/gems/apipie-rails-0.8.2/app/views/apipie/apipies/apipie_checksum.json.erb (Duration: 1323.4ms | Allocations: 1772899)
2023-02-20T20:08:24 [I|app|9709ac29] Completed 200 OK in 1333ms (Views: 1328.3ms | ActiveRecord: 0.6ms | Allocations: 1776079)
2023-02-20T20:08:24 [D|tel|9709ac29] Incrementing counter fm_rails_http_requests by 1.00 {:controller=>"other", :action=>"other", :status=>200}
2023-02-20T20:08:24 [D|tel|9709ac29] Observing histogram fm_rails_http_request_total_duration value 1332.70 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Observing histogram fm_rails_http_request_db_duration value 0.60 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Observing histogram fm_rails_http_request_view_duration value 1328.33 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Incrementing counter fm_rails_ruby_gc_count by 5.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Incrementing counter fm_rails_ruby_gc_minor_count by 5.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Incrementing counter fm_rails_ruby_gc_allocated_objects by 1789097.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [D|tel|9709ac29] Incrementing counter fm_rails_ruby_gc_freed_objects by 1789440.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:08:24 [I|app|f529536b] Started POST "/api/users" for 10.212.2.31 at 2023-02-20 20:08:24 +0100
2023-02-20T20:08:24 [I|app|f529536b] Processing by Api::V2::UsersController#create as JSON
2023-02-20T20:08:24 [I|app|f529536b]   Parameters: {"user"=>{"login"=>"testuser", "mail"=>"testuser1@my.org", "password"=>"[FILTERED]", "auth_source_id"=>1}, "apiv"=>"v2"}
2023-02-20T20:08:24 [D|tel|f529536b] Incrementing counter fm_rails_activerecord_instances by 1.00 {:class=>"User"}
2023-02-20T20:08:25 [D|tel|f529536b] Observing histogram fm_rails_login_pwhash_duration value 366.03 {:algorithm=>:pbkdf2sha1}
2023-02-20T20:08:32 [D|dyn|] Executor heartbeat
2023-02-20T20:08:47 [D|dyn|] Executor heartbeat

And the following even includes :sql: logging (everything set to “debug”):

2023-02-20T20:13:05 [D|sql|01396098]   Katello::Event Load (0.2ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:05.924455"], ["in_progress", false], ["LIMIT", 1]]
2023-02-20T20:13:08 [D|sql|01396098]   Katello::Event Load (0.2ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:08.926681"], ["in_progress", false], ["LIMIT", 1]]
2023-02-20T20:13:11 [D|sql|01396098]   Katello::Event Load (0.3ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:11.929095"], ["in_progress", false], ["LIMIT", 1]]
2023-02-20T20:13:13 [D|dyn|] Executor heartbeat
2023-02-20T20:13:14 [D|sql|01396098]   Katello::Event Load (0.2ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:14.932780"], ["in_progress", false], ["LIMIT", 1]]
2023-02-20T20:13:15 [I|app|31b21abd] Started GET "/apidoc/apipie_checksum" for 10.212.2.31 at 2023-02-20 20:13:15 +0100
2023-02-20T20:13:15 [I|app|31b21abd] Processing by Apipie::ApipiesController#apipie_checksum as JSON
2023-02-20T20:13:15 [I|app|31b21abd]   Parameters: {"apipy"=>{}}
2023-02-20T20:13:15 [D|app|31b21abd]   Rendering /usr/share/gems/gems/apipie-rails-0.8.2/app/views/apipie/apipies/apipie_checksum.json.erb
2023-02-20T20:13:16 [I|app|31b21abd]   Rendered /usr/share/gems/gems/apipie-rails-0.8.2/app/views/apipie/apipies/apipie_checksum.json.erb (Duration: 1524.1ms | Allocations: 1772575)
2023-02-20T20:13:16 [I|app|31b21abd] Completed 200 OK in 1530ms (Views: 1527.7ms | ActiveRecord: 0.0ms | Allocations: 1775713)
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_http_requests by 1.00 {:controller=>"other", :action=>"other", :status=>200}
2023-02-20T20:13:16 [D|tel|31b21abd] Observing histogram fm_rails_http_request_total_duration value 1530.35 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Observing histogram fm_rails_http_request_db_duration value 0.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Observing histogram fm_rails_http_request_view_duration value 1527.74 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_ruby_gc_count by 6.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_ruby_gc_major_count by 1.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_ruby_gc_minor_count by 5.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_ruby_gc_allocated_objects by 1777601.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:16 [D|tel|31b21abd] Incrementing counter fm_rails_ruby_gc_freed_objects by 1777086.00 {:controller=>"other", :action=>"other"}
2023-02-20T20:13:17 [I|app|c1c12435] Started POST "/api/users" for 10.212.2.31 at 2023-02-20 20:13:17 +0100
2023-02-20T20:13:17 [I|app|c1c12435] Processing by Api::V2::UsersController#create as JSON
2023-02-20T20:13:17 [I|app|c1c12435]   Parameters: {"user"=>{"login"=>"testuser", "mail"=>"testuser1@my.org", "password"=>"[FILTERED]", "auth_source_id"=>1}, "apiv"=>"v2"}
2023-02-20T20:13:17 [D|sql|c1c12435]   Setting Load (0.3ms)  SELECT "settings".* FROM "settings" WHERE "settings"."category" = $1 AND "settings"."value" IS NOT NULL AND (updated_at >= '2023-02-20 19:12:06.554863')  [["category", "Setting"]]
2023-02-20T20:13:17 [D|sql|c1c12435]   User Load (0.2ms)  SELECT "users".* FROM "users" WHERE "users"."lower_login" = $1 LIMIT $2  [["lower_login", "admin_01"], ["LIMIT", 1]]
2023-02-20T20:13:17 [D|tel|c1c12435] Incrementing counter fm_rails_activerecord_instances by 1.00 {:class=>"User"}
2023-02-20T20:13:17 [D|tel|c1c12435] Observing histogram fm_rails_login_pwhash_duration value 378.72 {:algorithm=>:pbkdf2sha1}
2023-02-20T20:13:17 [D|sql|c1c12435]   PersonalAccessToken Load (0.6ms)  SELECT "personal_access_tokens".* FROM "personal_access_tokens" WHERE "personal_access_tokens"."revoked" = $1 AND (expires_at >= '2023-02-20 19:13:17.081692' OR expires_at IS NULL) AND "personal_access_tokens"."user_id" = $2 AND "personal_access_tokens"."token" = $3 LIMIT $4  [["revoked", false], ["user_id", 6], ["token", "$pbkdf2sha1$600000$9054fbe0b622c638224d50d20824d2ff6782e308$67SUVb8SU8BCneMq+0wP3Gu1x2tsy/ab2649b7e58f7e32b0c75b"], ["LIMIT", 1]]
2023-02-20T20:13:17 [D|sql|01396098]   Katello::Event Load (0.6ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:17.935555"], ["in_progress", false], ["LIMIT", 1]]
2023-02-20T20:13:20 [D|sql|01396098]   Katello::Event Load (0.2ms)  SELECT "katello_events".* FROM "katello_events" WHERE ("katello_events"."process_after" IS NULL OR "katello_events"."process_after" BETWEEN $1 AND $2) AND "katello_events"."in_progress" = $3 ORDER BY "katello_events"."created_at" ASC LIMIT $4  [["process_after", "4713-01-01 BC"], ["process_after", "2023-02-20 19:13:20.940814"], ["in_progress", false], ["LIMIT", 1]]

It’s frustrating that full debug logging doesn’t provide more details.
Is there a way to further increase the verbosity of logging?

I found the cause of the problem.

Under “Administer” - “Settings” - “Authentication”, the value for “BCrypt password cost” was set to “30”, which is a too high value on my system.

Explanation:
What happened is that the user creation process was trying to hash the newly requested password with a BCrypt password cost of 30 (maximum value).
Using such a high value may have had to run several hours or several days on my system, as the value is increasing the cost exponentially, and not in a linear fashion.
The time to calculate the hash appeared to be significantly higher than the session timeout, so that the user creation never had a chance to finish.

Solution:
I figured out, that on my system, a value of “15” works well.
Of course, one may also go with the default value of “4”, but it comes with a lower level of protection against brute-force attacks.

Recommendations to Development:

  • The default value of “4” for “BCrypt password cost” seems too low in 2023. A higher value still seems conservative.
  • The “Description” of “BCrypt password cost” under “Administer” - “Settings” - “Authentication” could be clearer. Higher values take exponentially longer, and too high values may run into timeouts.
  • Even the most verbose server logging did not expose, that this problem was related to the “BCrypt password cost”. There should be a) a warning when the user creation process takes longer than the session timeout b) a warning when the function call for the BCrypt hashing function did not return in time, and c) debug logging should log the function call to the BCrypt hashing function, because it is a potentially long-lasting function.
  • It may be a good idea to to abort the BCrypt function call, when it takes longer then a timeout. An info message could be added to the server log explaining what happened. And of course a call stack on the Exception.
  • The BCrypt hashing function appears to be implemented in a single-threaded manner. Changing it to multi-threading should run faster for the same level of security, and it would better match against attackers, who will always use multi-threading.
1 Like