Sync Error: PulpCertguardClient::ApiError: Error message: the server returns an error

Problem:

Lately we have started seeing these errors while syncing to our Smart Proxies:

PulpCertguardClient::ApiError: Error message: the server returns an error

Expected outcome:

That we can sync content from our main server out to our various environments.

Foreman and Proxy versions:

Foreman and Proxy plugin versions:

Foreman: 3.1.3-1.el7 and Katello: 4.3.1-1.el7

Distribution and version:

CentOS 7.9

Other relevant data:

Hoping someone out there in Cyberspace has some insight into this / how we can fix it. Thank you!

Here is the backtrace:

Actions::Pulp3::ContentGuard::Refresh

Input:

{“smart_proxy_id”=>4, “remote_user”=>“admin”, “remote_cp_user”=>“admin”, “current_request_id”=>“a76cacc9-08fd-4d15-abb1-679e8d80bb91”, “current_timezone”=>“America/Los_Angeles”, “current_organization_id”=>3, “current_location_id”=>nil, “current_user_id”=>1}

Output:

{}

Exception:

PulpCertguardClient::ApiError: Error message: the server returns an error

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.5.0/lib/pulp_certguard_client/api_client.rb:90:in rescue in call_api' /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.5.0/lib/pulp_certguard_client/api_client.rb:68:in call_api’ /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.5.0/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:225:in list_with_http_info' /opt/theforeman/tfm/root/usr/share/gems/gems/pulp_certguard_client-1.5.0/lib/pulp_certguard_client/api/contentguards_rhsm_api.rb:161:in list’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/services/katello/pulp3/api/content_guard.rb:65:in list' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/services/katello/pulp3/api/content_guard.rb:32:in refresh’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/pulp3/content_guard/refresh.rb:10:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:582:in block (3 levels) in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:40:in block in as_remote_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/models/katello/concerns/user_extensions.rb:21:in cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:27:in as_cp_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:39:in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.3.1/app/lib/actions/middleware/remote_action.rb:16:in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/rails_executor_wrap.rb:14:in block in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/rails_executor_wrap.rb:13:in run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:31:in with_progress_calculation’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action/progress.rb:17:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/load_setting_values.rb:20:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:52:in restore_current_request_id’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_request_id.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:44:in restore_curent_timezone’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_timezone.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in restore_current_taxonomies’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:32:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:27:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware.rb:19:in pass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:54:in restore_curent_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-5.2.3/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/stack.rb:23:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/middleware/world.rb:31:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:581:in block (2 levels) in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:580:in block in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in catch’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:483:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:575:in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/action.rb:296:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in block (2 levels) in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract.rb:167:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in block in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/director.rb:69:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in block (2 levels) in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors.rb:18:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in block in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in perform’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.6.4/lib/dynflow/executors/sidekiq/serialization.rb:27:in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in execute_job’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in block in invoke’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in block in process’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in local’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in block in module:Sidekiq’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in stats’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in call’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in global’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in with_context’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in dispatch’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in process_one’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in watchdog’ /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in block in create_with_logging_context’

Do you have the pulp logs from around this error in journalctl logs?

1 Like

Hi @sajha

Thanks for pinging me back.

Do you mean from the main server or the proxy in question I guess is my first question.

Secondly, does katello log pulp errors somewhere specific that I can go and look at them? When I grep for pulp in /var/log/messages on the main server there are entries there but they basically look like GETs for Content Views and they are returning 200’s.

We have the timestamp of when the proxy sync failed so we just need to know where to go look.

  • Eledor

p.s.

I did try this on the main server:

journalctl --since “21 hour ago” -l --no-pager | less | grep 1004

Where the string “1004” is part of the FQDN of the problem proxy and it didnt return anything

If these syncs just stopped working and the failure is on certguard, I wonder if something happened with certs. Are those valid?

Yes, they are. Are there any other locations to check on the main katello server for pulp errors other than /var/log/messages?

Another bit of info- I have checked the sync status on the problem proxy this morning and it is working again. So not sure why it would throw that error and then start working again. We haven’t changed any configs- just restarted the services on the main katello host.

I’d imagine something to do with certs since the certguard_plugin was complaining but not sure. We call a refresh on the certguard in pulp before every sync that would fix any cert related issues when there are valid certs.

journalctl -fu pulp* is how you can look up pulp logs. All the pulp logs are logged into /var/log/messages. Foreman log does capture pulp errors but most of the time it’s whatever message pulp error handling send us which in many cases isn’t as detailed as the actual traceback if any.

Hi Sajha,

Thanks for the command of: journalctl -fu pulp*

I’ve made a note of that.

At the moment we aren’t seeing this error anymore from certguard and the proxies are synching fine so I think we’re good for now.

Cheers,

Eledor

1 Like