HTTP(S) Proxy is ignored

Support
1

Problem:
Proxy configured in Settings > HTTP(S) proxy is ignored when attempting to enable Red Hat repositories.

Foreman’s production log (/var/log/foreman/production.log) only produces valuable information if the hammer command is used. Message returned in Web UI states, “No repositories available.”

Expected outcome:
The proxy configured in Settings should be used for all outbound connections not destined for the Foreman host.

The same logging output (in /var/log/foreman/production.log; pasted below) should be produced regardless of whether attempting to enable the Red Hat repo via Web UI or hammer. The hammer command returns message of, “Request Timeout.” The Web UI should also produce the correct error of, “Request Timeout.”

Foreman and Proxy versions:
Foreman 1.21.0, Foreman-Proxy 1.21.0

Foreman and Proxy plugin versions:
Katello 3.11.0

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

When I attempt to enable a Red Hat repo, I can see the connections attempting to be made directly over HTTPS. These connections should be going through the defined HTTP(S) Proxy. We are on a secured network, so the Foreman server does not have direct access to the Internet.

COMMAND   PID    USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
ruby     7642 foreman   21u  IPv4 25982630      0t0  TCP my-foreman-server.foobar.com:42524->a23-61-124-251.deploy.static.akamaitechnologies.com:https (SYN_SENT)

I have attempted to set http_proxy and https_proxy in /etc/sysconfig/httpd and /etc/sysconfig/foreman, but they are also ignored. Pulp does not have any issue with using the proxy for mirroring other repos (aka EPEL, CentOS, etc.). This issue seems to be isolated to Red Hat Repositories. The configured proxy is also correctly used when refreshing the manifest via Manage Manifest > Update.

The following logging output is ONLY produced when using the hammer command to enable the repo. I would expect the Web UI to also result in producing the same content, but it does not (which left me banging my head for a number of days):

2019-04-08T11:05:55 [I|app|bc22775d] Started GET "/katello/api/organizations?search=name+%3D+%22RHEL%22" for 127.0.0.1 at 2019-04-08 11:05:55 -0400
2019-04-08T11:05:55 [I|app|bc22775d] Processing by Katello::Api::V2::OrganizationsController#index as JSON
2019-04-08T11:05:55 [I|app|bc22775d]   Parameters: {"search"=>"name = \"RHEL\"", "api_version"=>"v2", "organization"=>{}}
2019-04-08T11:05:55 [I|app|bc22775d] Authorized user admin(Admin User)
2019-04-08T11:05:55 [I|app|bc22775d] Current user set to admin (admin)
2019-04-08T11:05:55 [I|app|bc22775d]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/views/katello/api/v2/organizations/index.json.rabl within api/v2/layouts/index_layout
2019-04-08T11:05:55 [I|app|bc22775d]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/views/katello/api/v2/organizations/index.json.rabl within api/v2/layouts/index_layout (2.1ms)
2019-04-08T11:05:55 [I|app|bc22775d] Completed 200 OK in 35ms (Views: 3.2ms | ActiveRecord: 15.1ms)
2019-04-08T11:05:55 [I|app|2b29f78e] Started PUT "/katello/api/repository_sets/2456/enable" for 127.0.0.1 at 2019-04-08 11:05:55 -0400
2019-04-08T11:05:55 [I|app|2b29f78e] Processing by Katello::Api::V2::RepositorySetsController#enable as JSON
2019-04-08T11:05:55 [I|app|2b29f78e]   Parameters: {"basearch"=>"x86_64", "releasever"=>"7Server", "organization_id"=>3, "api_version"=>"v2", "id"=>"2456", "repository_set"=>{"basearch"=>"x86_64", "releasever"=>"7Server", "organization_id"=>3}}
2019-04-08T11:05:55 [I|app|2b29f78e] Authorized user admin(Admin User)
2019-04-08T11:05:55 [I|app|2b29f78e] Current user set to admin (admin)
2019-04-08T11:06:55 [E|bac|2b29f78e] Request Timeout (RestClient::RequestTimeout)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/resources/cdn.rb:114:in `rescue in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/resources/cdn.rb:94:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:81:in `valid_path?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `block in any_valid_metadata_file?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `any_valid_metadata_file?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:43:in `validate_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/models/katello/candlepin/repository_mapper.rb:48:in `validate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/actions/katello/repository_set/enable_repository.rb:13:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:492:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:491:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:395:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:490:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:489:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:277:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:285:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:284:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:200:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:198:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:198:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:174:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:19:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:25:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:45:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:44:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:23:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:54:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks/triggers.rb:22:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/controllers/katello/api/v2/repository_sets_controller.rb:81:in `enable'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2019-04-08T11:06:55 [E|app|2b29f78e] exception when talking to a remote client: Request Timeout RestClient::RequestTimeout: Request Timeout
Body: 
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/resources/cdn.rb:114:in `rescue in get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/resources/cdn.rb:94:in `get'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:81:in `valid_path?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `block in any_valid_metadata_file?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `any?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:51:in `any_valid_metadata_file?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/katello/util/cdn_var_substitutor.rb:43:in `validate_substitutions'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/models/katello/candlepin/repository_mapper.rb:48:in `validate!'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/lib/actions/katello/repository_set/enable_repository.rb:13:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:492:in `block (3 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:31:in `with_current_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_timezone.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:31:in `with_current_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_user.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:10:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:30:in `with_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/app/lib/actions/middleware/keep_current_taxonomies.rb:9:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:35:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:491:in `block (2 levels) in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:395:in `concurrence'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:490:in `block in execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:450:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:489:in `execute_plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/action.rb:277:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/plan_step.rb:54:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan/steps/plan_step.rb:53:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:286:in `block (2 levels) in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:384:in `switch_flow'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:374:in `with_planning_scope'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:285:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:43:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:26:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware.rb:18:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:16:in `block in rollback_on_error'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `block in transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/connection_adapters/abstract/database_statements.rb:254:in `transaction'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/transactions.rb:212:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/transaction_adapters/active_record.rb:5:in `transaction'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:15:in `rollback_on_error'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/common/transaction.rb:5:in `plan_phase'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/stack.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/middleware/world.rb:30:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/execution_plan.rb:284:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:200:in `block in plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:198:in `tap'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:198:in `plan'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.2.1/lib/dynflow/world.rb:174:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:19:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:25:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:45:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:44:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:23:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks.rb:54:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.14.5/lib/foreman_tasks/triggers.rb:22:in `sync_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/controllers/katello/api/v2/repository_sets_controller.rb:81:in `enable'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:163:in `disable_json_root'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.7.1/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/app/controllers/katello/concerns/api/api_controller.rb:39:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:19:in `block in <class:Constraints>'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/mapper.rb:48:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.11.0/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.0.0/lib/secure_headers/middleware.rb:13:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.5/lib/rack/urlmap.rb:53:in `call'
/usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
/usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
/usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2

This is the ONLY way I’ve been able to sync yum repos (CentOS & EPEL) from behind a proxy.

Add your proxy’s address to the below files:

/etc/foreman/settings.yaml

:http_proxy: http://yourProxyIP:3128

/etc/foreman/plugins/katello.yaml

:cdn_proxy:
    host: http://yourProxyIP
    port: 3128
    user: "" # this is not optional
    password: ""

/etc/pulp/server/plugins.conf.d/yum_importer.json

 {
   "proxy_host": "http://yourProxyIP",
   "proxy_port: 3128,
   "proxy_username": null,
   "proxy_password": null,
   "max_speed": null,
   "gpg_sign_metadata": false
 }

Restart foreman

3

Thanks for the update and your help!

I already have the changes you mentioned for yum_importer.json. I have tried the changes you mentioned for settings.yaml and katello.yaml before (found via Google), but to no avail.

I just tried them again and it did not fix the issue.

4

Did you restart foreman? I initially just did an systemctl restart httpd but actually had to reboot my VM. After it came back I was able to sync repos. This is the 2nd of 4 Foreman/Katello instances in which it’s worked reliably that way…

-HTH
Greg

5

Yep! I restarted both httpd and foreman.

6

Damn, man I love Foreman/Katello but I have to admit for me it’s still more of an art than a science to getting it working for my envs. I’m still closer to the bottom rather than the top of the learning curve for sure. In your katello.yaml file you have " " for the user and password fields? That will cause a problem if they’re not there. I’m using Foreman 1.21.0 and Katello 3.11 and it’s working as described above for me behind my proxy…

7

Yes, sir. I confirmed the use of double-quotes where mentioned. Just a friendly reminder, this is only not working when attempting to enable Red Hat repos. Other repos, like CentOS and EPEL, have been added and mirrored without incident.

8

Yeah, they may have some secret sauce embedded somewhere to encourage the use of Satellite… :wink:

9

Possibly, though, it was working just fine in a previous version. But that system died recently, so I’m building a new one and can’t find if I’m missing a setting or not.

10

The problem is that for synching repositories, Katello uses pulp, which also needs to know about the HTTP-proxy. As already mentioned, this has to be done in /etc/pulp/server/plugins.conf.d/yum_importer.json (for yum-repos).
The good news is that the foreman-installer can set this configuration for you:

foreman-installer --help
[...]
    --katello-proxy-password      Proxy password for authentication (current: UNDEF)
    --katello-proxy-port          Port the proxy is running on (current: UNDEF)
    --katello-proxy-url           URL of the proxy server (current: UNDEF)
    --katello-proxy-username      Proxy username for authentication (current: UNDEF)
[...]

You will have to restart pulp after setting the proxy however.
The best way to restart all foreman services is with foreman-maintain

foreman-installer --katello-proxy-url http://yourProxyIP --katello-proxy-port yourProxyPort
foreman-maintain service restart

Unfortunately, currently there are multiple ways to set HTTP-Proxy in Foreman/Katello and not all work as expected. For instance setting the HTTP-Proxy in the foreman-GUI should set the Proxy-usage for e.g. Compute-Resources. However, this does not work for the VMWare Compute-Resource.
It also does not work for the Smart-Proxy.
The safest way I currently know is to set the http_proxy environment-variables in /etc/sysconfig/httpd (Foreman) and /etc/sysconfig/foreman-proxy (Smart-Proxy) and use the foreman-installer to configure pulp correctly (see above).

The “Infrastructure” -> “HTTP Proxies” page is dedicated to be able to assign different HTTP-Proxies to different Compute-Resources, but AFAIK currently only the EC2 Compute Resource supports it.

1 Like
11

So, I found the solution to the problem. I had to add

http_proxy="http://my-proxy.com:3128"
https_proxy="http://my-proxy.com:3128"

To the file /etc/sysconfig/dynflowd, then systemctl restart foreman-tasks. For some reason, that allowed Foreman-Katello to contact Red Hat, verify repo CDN location, etc.

I was then able to enable the RHEL 7 repo and begin mirroring.

I hope this content helps someone else if they run into a similar issue.

Thanks to all for your input.

12

Hello, this worked for me… Katello was ignoring proxy for connection to RHEL repos on Foreman 1.24RC1 and Katello 3.14. foreman-installer with proxy settings worked fine.