Cannot retrieve updates or packages

I rolled out the new foreman server yesterday, and am now plagued with this issue all over the place. Have you guys seen this before? After some checking it looks like our old Foreman server did it too.

When I click on the link in the error

To add more to thisā€¦ Iā€™m using v3.8. I see that this was tagged as pulpā€¦is this a pulp only issue?

If i take away the S in https, it works. Iā€™ve since installed a real SSL certā€¦ but the issue remains.

As you can see, the cert is working

Alsoā€¦ since adding the SSL certā€¦ foreman will not let me delete hosts

image

Action:

Actions::BulkAction

Input:

{ā€œaction_classā€=>ā€œActions::Katello::Host::Destroyā€, ā€œtarget_idsā€=>[9, 10], ā€œtarget_classā€=>ā€œHost::Managedā€, ā€œargsā€=>, ā€œcurrent_user_idā€=>6}

Output:

{ā€œplanned_countā€=>0, ā€œcancelled_countā€=>0, ā€œtotal_countā€=>2, ā€œfailed_countā€=>2, ā€œpending_countā€=>0, ā€œsuccess_countā€=>0}

Exception:

RuntimeError: A sub task failed

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:230:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:122:inwait_for_sub_plansā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:42:in spawn_plans' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:70:inspawn_plansā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:36:in initiate' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:35:ininitiateā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:inblock in assignsā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:inassignsā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:inblock in match?ā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:inmatch?ā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:11:inrunā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:23:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/bulk_action.rb:21:inrunā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:538:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:inpassā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:14:inblock in runā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:13:inrunā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:inpassā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:30:inwith_progress_calculationā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incallā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpassā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:11:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:22:inwith_localeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:11:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incallā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpassā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:43:inrestore_curent_userā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incallā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpassā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:31:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incallā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/world.rb:30:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:537:inblock (2 levels) in execute_runā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:inblock in execute_runā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:incatchā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:531:inexecute_runā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:278:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:inblock (2 levels) in executeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract.rb:162:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:inblock in executeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:inexecuteā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/director.rb:43:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:12:inblock in on_messageā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors.rb:12:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:11:inon_messageā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/actor.rb:26:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:inblock in on_envelopeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronizeā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronizeā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in block in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:incallā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in work' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:inblock in call_jobā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:inblock (3 levels) in create_workerā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:inblock (2 levels) in create_workerā€™ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:inblock in create_workerā€™ /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_c

additionallyā€¦ all new servers now fail :frowning:

chef-client on 10.64.43.16 APP [3:45 PM]
:skull: Chef run failed on env us_dev, node us08dv2app23
rhsm_register[US08DV2APP23] (pl_katello_client::default line 23) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[Register to RHSM] (/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/resource/rhsm_register.rb line 87) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ā€˜70ā€™
---- Begin output of subscription-manager register --activationkey=Centos --org=Protolabs ----
STDOUT:
STDERR: Unable to verify serverā€™s identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
---- End output of subscription-manager register --activationkey=Centos --org=Protolabs ----
Ran subscription-manager register --activationkey=Centos --org=Protolabs returned 70

I am not sure if i can help you. But i did SSL change last week for foreman with katello scenario without manual changes in configurations.

I used katello utility script
usage: /sbin/katello-certs-check -c CERT_FILE -k KEY_FILE -b CA_BUNDLE_FILE

Then I ran command
/sbin/katello-certs-check -c /etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt -k /etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key -b /etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem

it checked certs and CA, and generated command for foreman-installer .
All was done automatically.

Then i just had to send new CA authority to all clients we had registered to foreman and it works great.

It was painless. But i see you use chef and i am not sure if it will work for you too.

I will try thatā€¦ thank you. All my servers are reporting problems with
foreman now. I tried manual steps yesterday, but it all failed. Did you
change your host name to one that matches the cert?

Letā€™s say the machine is: server1.internaldomain.com

And the ssl cert is:
Forman.mycompanyname.com

I want everything to use the ssl cert name, and not the internal name. Make
sense?

I am not sure it will work if you have certificate for different hostname.
You may go with self-signed certificate

I asked our CA to generate(or sign) for exact hostname. pr0mgm01.blabla.com (it is in our corporate domain and internal DNS and goal for this was to have green lock in our corporate network)

Because our CA is Windows team, they gave us pfx certificate.
So i extracted certificate pr0mgm01_encrypted.crt. key pr0mgm01_encrypted_pem.key and bundle from pfx .
I run
/sbin/katello-certs-check on pr0mgm01.blabla.com server.

And that was all.
I tried 2 weeks to understand all those certificates in Foreman, but then i luckily found that utility script.
I saw that HowTo from 2015 too, but i would not follow it now.

Give that script chance. It generate something like
ā€™
Checking server certificateā€™s encoding: [OK]
Checking expiration of certificate: [OK]
Checking expiration of CA bundle: [OK]
Checking if server cert has CA:TRUE flag[OK]
Validating the certificate subject= /CN=pr0mgm01.blabla.com
Checking to see if the private key matches the certificate: [OK]
Checking ca bundle against the cert file: [OK]
Checking Subject Alt Name on certificate[OK]
Checking Key Usage extension on certificate for Key Encipherment[OK]

Validation succeeded.

To install the Katello main server with the custom certificates, run:

foreman-installer --scenario katello\
                  --certs-server-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt"\
                  --certs-server-key "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key"\
                  --certs-server-ca-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem"\

To update the certificates on a currently running Katello installation, run:

foreman-installer --scenario katello\
                  --certs-server-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt"\
                  --certs-server-key "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key"\
                  --certs-server-ca-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem"\
                  --certs-update-server --certs-update-server-ca

ā€™

And foreman-installer will care of it.

Thanks for the information againā€¦ it didnā€™t workā€¦

I ran the cert check by itselfā€¦ and it validatedā€¦ but when I run the command they (and you) suggestā€¦ it errors out. Seems to me this is an issue with their software.

And hereā€™s the command I copied:

can someone from @katello team please take a look?

Hey guysā€¦ please do not invest any more time on this. Weā€™ve made the decision to move away from this product line. For those that helpedā€¦ I sincerely appreciate your time. If anyone from the @katello team is interested in a debriefing, please reach out to me privately. @tbrisker @W73, thanks again.

We got some weird behaviour with the subscription-manager these days tooā€¦maybe itā€™s related.

Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Forbidden

Thatā€™s what I got if Iā€™m trying to sync some reposā€¦standard repos to be exact (RHEL 7.5)

Check your imported Manifest file from RedHat. It may be expired.

1 Like

Syncs are working againā€¦but that was just one of many problems these days with subscriptions. Even the subscription plugin for yum took ages and so on. Just stated that Iā€™ve the feeling of external changes involved.