I'm running foreman 1.14.2, and Katello 3.3.0.
On the foreman server, I'm seeing these messages:
[Wed Mar 15 01:57:02.739257 2017] [ssl:error] [pid 18720] [client
10.9.0.1:42382] AH02039: Certificate Verification: Error (20): unable to
get local issuer certificate
…
eventually followed by a burst of something like this:
2017-03-15 01:57:02 [foreman-tasks/action] [E] RPM1004: Error retrieving
metadata: Not found (Katello::Errors::PulpError)
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:121:in
block in external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in each'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:119:in
external_task=' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:98:in poll_external_task_with_rescue'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/polling.rb:21:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/cancellable.rb:9:in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/pulp/abstract_async_task.rb:45:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:506:in block (3 levels) in execute_run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in block in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:40:in
block in as_remote_user' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:27:in
as_cp_user' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/remote_action.rb:16:in
run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in call'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:30:in
with_progress_calculation' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action/progress.rb:16:in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in block in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:22:in
with_locale' > /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.3.0.1/app/lib/actions/middleware/keep_locale.rb:11:in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:26:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:17:in
pass' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware.rb:30:in run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/stack.rb:22:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/middleware/world.rb:30:in execute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:505:in
block (2 levels) in execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in catch'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:504:in
block in execute_run' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in call'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in
block in with_error_handling' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in catch'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:419:in
with_error_handling' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:499:in execute_run'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/action.rb:260:in
execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in block (2 levels) in execute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in
call' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract.rb:155:in with_meta_calculation'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in
block in execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in open_action'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in
execute' > /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/director.rb:55:in execute'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/executors/parallel/worker.rb:11:in
on_message' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/context.rb:46:in on_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/executes_context.rb:7:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.17/lib/dynflow/actor.rb:26:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/awaits.rb:15:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:38:in
process_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/buffer.rb:20:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/termination.rb:55:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/removes_child.rb:10:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/abstract.rb:25:in pass'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/behaviour/sets_results.rb:14:in
on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:161:in process_envelope'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:95:in
block in on_envelope' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:118:in block (2 levels) in schedule_execution'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
block in synchronize' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/synchronization/mri_lockable_object.rb:38:in
synchronize' > /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.0/lib/concurrent/actor/core.rb:115:in block in schedule_execution'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in
call' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:18:in call'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:96:in
work' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/serialized_execution.rb:77:in block in call_job'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in
call' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:333:in run_task'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:322:in
block (3 levels) in create_worker' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in loop'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:305:in
block (2 levels) in create_worker' > /opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in catch'
>
/opt/rh/sclo-ror42/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in
block in create_worker' > /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in call'
>
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'
On the proxy side, I see this:
Mar 15 01:56:58 smart-proxy-02 pulp: nectar.downloaders.threaded:ERROR:
Skipping requests to <foreman server> due to repeated connection failures:
[SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:579)
…
eventually followed by this:
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Exception while
retrieving metadata for repository <blah blah>
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) Traceback (most
recent call last):
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File
"/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py",
line 113, in _parse_metadata
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880)
metadata_json_docs = downloader.retrieve_metadata(self.progress_report)
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) File
"/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py",
line 57, in retrieve_metadata
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880) raise
exceptions.FileRetrievalException(report.error_msg)
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31246-98880)
FileRetrievalException: FileRetrievalException: A connection error occurred
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Exception while
retrieving metadata for repository <nuance_mobility-Production-Smart-Proxy>
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) Traceback (most
recent call last):
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File
"/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/forge.py",
line 113, in _parse_metadata
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096)
metadata_json_docs = downloader.retrieve_metadata(self.progress_report)
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) File
"/usr/lib/python2.7/site-packages/pulp_puppet/plugins/importers/downloaders/web.py",
line 57, in retrieve_metadata
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096) raise
exceptions.FileRetrievalException(report.error_msg)
Mar 15 01:57:04 smart-proxy-02 pulp:
pulp_puppet.plugins.importers.forge:ERROR: (31234-04096)
FileRetrievalException: FileRetrievalException: A connection error occurred
On my foreman server, pulp is configured with this:
[security]
cacert: /etc/pki/pulp/ca.crt
cakey: /etc/pki/pulp/ca.key
And the proxy is configured with this:
[security]
cacert: /etc/pki/katello/certs/katello-default-ca.crt
cakey: /etc/pki/pulp/ca.key
Every single proxy is experiencing the same error. I installed a new
proxy to test it, and it fails to sync with the same error as well.
It's probably worth noting that I did change the name of the foreman server
about a week ago (this was succeeding prior to that), and I used a new
script:
https://github.com/Katello/katello-packaging/pull/323/commits
Everything seemed to be fine after that, but I only recently got around to
checking out the proxies.
From the proxy, I could run:
openssl s_client -connect foreman-01.prod.mcs.som.mob.nuance.com:443
-CAfile /etc/pki/katello/certs/katello-default-ca.crt
…and it completes successfully.
Thanks for any help pointing me in the right direction!