Repo creation failure

Hello, I’ve done a new installation of Katello/Foreman. Upon trying to create a repo that task never completes due to an authentication error. I’ve seen some of the other posts here talking about authentication issues w/Pulp w/the same error I’m seeing however, I can’t find an actual solution to resolve this. Any help or pointers to where I should be looking are greatly appreciated.

Problem:
Trying to create repository. Task hangs at 6% and never completes

Expected outcome:
Empty repository successfully created

Foreman and Proxy versions:
Foreman v2.2.0-rc4
Katello v3.17-rc2

Foreman and Proxy plugin versions:

Distribution and version:
CentOS Linux release 7.8.2003 (Core)

Other relevant data:
Error message: the server returns an error
HTTP status code: 403
Response headers: {“date”=>“Tue, 27 Oct 2020 23:26:19 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“58”, “via”=>“1.1 4man.MYDOMAIN.net”, “connection”=>“close”}
Response body: {“detail”:“Authentication credentials were not provided.”}

Thanks,
-Greg

@thinkitdata Thank you for bringing this to our attention.

Can you check the state of these files on the katello/foreman host?
/etc/pki/katello/certs/pulp-client.crt
/etc/pki/katello/private/pulp-client.key

1 Like

Absolutely, here’s the info:

[root@4man ~]# ls -la /etc/pki/katello/certs
total 64
drwxr-xr-x. 2 root foreman 245 Oct 27 15:23 .
drwxr-xr-x. 6 root foreman 286 Oct 27 15:23 …
-rw-r–r--. 1 root root 8204 Oct 27 15:23 4man.MYDOMAIN.net-qpid-broker.crt
-r–r-----. 1 root foreman 8137 Oct 27 15:23 java-client.crt
-rw-r–r--. 1 root root 8179 Oct 27 15:23 katello-apache.crt
-rw-r–r--. 1 root foreman 8123 Oct 27 15:22 katello-default-ca.crt
-rw-r–r--. 1 root root 2439 Oct 27 15:22 katello-default-ca-stripped.crt
-rw-r–r--. 1 root foreman 8123 Oct 27 15:22 katello-server-ca.crt
-rw-r–r--. 1 root root 8131 Oct 27 15:23 katello-tomcat.crt
-rw-r–r--. 1 root root 8094 Oct 27 15:23 pulp-client.crt

[root@4man ~]# ls -la /etc/pki/katello/private
total 40
drwxr-x—. 2 root foreman 266 Oct 27 15:30 .
drwxr-xr-x. 6 root foreman 286 Oct 27 15:23 …
-rw-r–r--. 1 root root 11429 Oct 27 15:30 4man.MYDOMAIN.net-foreman-proxy-client-bundle.pem
-r–r-----. 1 root qpidd 3243 Oct 27 15:23 4man.MYDOMAIN.net-qpid-broker.key
-r–r-----. 1 root foreman 3247 Oct 27 15:23 java-client.key
-r–r-----. 1 root foreman 3247 Oct 27 15:23 katello-apache.key
-r–r-----. 1 root foreman 3247 Oct 27 15:22 katello-default-ca.key
-r--------. 1 root root 24 Oct 27 15:22 katello-default-ca.pwd
-r--------. 1 root root 3247 Oct 27 15:23 katello-tomcat.key
-r–r-----. 1 root foreman 3247 Oct 27 15:23 pulp-client.key

That looks to be in order.

Can you send the foreman production log from around the time of that failure? I would like to confirm the cause of the authentication failure.

Can you tell me where that specific log is and I’ll be happy to grab it.

I’ve found the log, will send the relevant lines as soon as I find them

Here are the requested foreman production log entries:

2020-10-27T17:40:19 [I|bac|] Task {label: Actions::Katello::Repository::CreateRoot, id: b78378a4-c16c-4848-913a-d1ad41a5191b, execution_plan_id: 9f195417-5152-41b5-8883-7c060c50873a} state changed: planned
2020-10-27T17:40:19 [I|bac|] Task {label: Actions::Katello::Repository::CreateRoot, id: b78378a4-c16c-4848-913a-d1ad41a5191b, execution_plan_id: 9f195417-5152-41b5-8883-7c060c50873a} state changed: running
2020-10-27T17:40:19 [E|bac|] Error message: the server returns an error
HTTP status code: 403
Response headers: {“date”=>“Tue, 27 Oct 2020 21:40:19 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“58”, “via”=>“1.1 4man.MYDOMAIN.net”, “connection”=>“close”}
Response body: {“detail”:“Authentication credentials were not provided.”} (PulpRpmClient::ApiError)
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.6.2/lib/pulp_rpm_client/api_client.rb:81:in call_api' /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.6.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:79:in create_with_http_info’
/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_rpm_client-3.6.2/lib/pulp_rpm_client/api/repositories_rpm_api.rb:28:in create' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/services/katello/pulp3/repository.rb:169:in create’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/pulp3/repository/create.rb:11:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in block (3 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/lib/actions/middleware/remote_action.rb:16:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in with_progress_calculation’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_request_id.rb:49:in restore_current_request_id’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_user.rb:44:in restore_curent_user’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in pass’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in block (2 levels) in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in block in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in catch’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in execute_run’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform’
[ sidekiq ]
[ concurrent-ruby ]
2020-10-27T17:40:19 [I|bac|] Task {label: Actions::Katello::Repository::CreateRoot, id: b78378a4-c16c-4848-913a-d1ad41a5191b, execution_plan_id: 9f195417-5152-41b5-8883-7c060c50873a} state changed: paused result: error
2020-10-27T17:40:19 [E|app|] ForemanTasks::TaskError: Task b78378a4-c16c-4848-913a-d1ad41a5191b: PulpRpmClient::ApiError: Error message: the server returns an error
HTTP status code: 403
Response headers: {“date”=>“Tue, 27 Oct 2020 21:40:19 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“58”, “via”=>“1.1 4man.MYDOMAIN.net”, “connection”=>“close”}
Response body: {“detail”:“Authentication credentials were not provided.”}

/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/lib/foreman_tasks.rb:59:in block in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/lib/foreman_tasks.rb:58:in tap’
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/lib/foreman_tasks.rb:58:in sync_task' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.0/lib/foreman_tasks/triggers.rb:22:in sync_task’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/controllers/katello/api/v2/repositories_controller.rb:244:in create' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/basic_implicit_render.rb:6:in send_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:195:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rendering.rb:30:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:42:in block in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:112:in block in run_callbacks’
/usr/share/foreman/app/controllers/api/v2/base_controller.rb:181:in disable_json_root' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in set_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in clear_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in set_topbar_sweeper_controller' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in around' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:121:in block in run_callbacks’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:139:in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/callbacks.rb:41:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/rescue.rb:22:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:33:in block in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in block in instrument' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications/instrumenter.rb:24:in instrument’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/notifications.rb:180:in instrument' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/instrumentation.rb:32:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal/params_wrapper.rb:245:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.1/lib/active_record/railties/controller_runtime.rb:27:in process_action’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/app/controllers/katello/concerns/api/api_controller.rb:40:in process_action' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/abstract_controller/base.rb:136:in process’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.1/lib/action_view/rendering.rb:39:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:190:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_controller/metal.rb:254:in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:50:in dispatch’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:33:in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:18:in block in class:Constraints
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in block in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in public_send' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in method_missing’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:19:in block in <class:Constraints>' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/mapper.rb:48:in serve’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:49:in block in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in each’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/journey/router.rb:32:in serve' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/routing/route_set.rb:834:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/lib/katello/middleware/event_daemon.rb:10:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.2.9/lib/apipie_dsl/static_dispatcher.rb:67:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in call' /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:40:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/http/content_security_policy.rb:18:in call' /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in context' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/cookies.rb:648:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:27:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/callbacks.rb:101:in run_callbacks' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/callbacks.rb:26:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/actionable_exceptions.rb:17:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/debug_exceptions.rb:32:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:37:in call_app’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/rack/logger.rb:28:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in call’
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/remote_ip.rb:81:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/request_id.rb:27:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.17.0.rc2.2/lib/katello/prevent_json_parsing.rb:12:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/executor.rb:14:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/static.rb:126:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.1/lib/action_dispatch/middleware/host_authorization.rb:76:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/engine.rb:527:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in public_send’
/opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.1/lib/rails/railtie.rb:190:in method_missing' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in block in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in call’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/configuration.rb:228:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:682:in handle_request’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:472:in process_client' /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/server.rb:328:in block in run’
/opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.3/lib/puma/thread_pool.rb:134:in block in spawn_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’
2020-10-27T17:40:19 [I|app|] Completed 500 Internal Server Error in 989ms (Views: 0.3ms | ActiveRecord: 41.0ms | Allocations: 135571)

@thinkitdata looks like this is a known issue at the moment.

https://projects.theforeman.org/issues/30770

There is a workaround in the meantime. Run this command:
sudo -u pulp PULP_SETTINGS=’/etc/pulp/settings.py’ DJANGO_SETTINGS_MODULE=‘pulpcore.app.settings’ python3-django-admin reset-admin-password --password password

Let us know if this gets you going again.

Hey man, thanks so very much. That fixed the not being able to create repos issue. I couldn’t tell from reading the ticket for the issue you referenced (as it’s listed as closed) whether that issue is fixed or still in play… Here’s the output of the command:

[root@4man pulp]# sudo -u pulp PULP_SETTINGS=’/etc/pulp/settings.py’ DJANGO_SETTINGS_MODULE=‘pulpcore.app.settings’ python3-django-admin reset-admin-password --password password

System check identified some issues:

WARNINGS:
?: (guardian.W001) Guardian authentication backend is not hooked. You can add this in settings as eg: AUTHENTICATION_BACKENDS = ('django.contrib.auth.backends.ModelBackend', 'guardian.backends.ObjectPermissionBackend').
Successfully set password for “admin” user.

Now not able to sync the repo but as I’m behind a proxy I have to make sure this isn’t a problem on my end.

thanks,
-Greg

Btw, got the sync working so this is looking good. Thanks again for the excellent help!

2 Likes

@thinkitdata Very glad we could help!