Cannot retrieve updates or packages

[smeads]

I rolled out the new foreman server yesterday, and am now plagued with this issue all over the place. Have you guys seen this before? After some checking it looks like our old Foreman server did it too.

When I click on the link in the error

[smeads]

To add more to this… I’m using v3.8. I see that this was tagged as pulp…is this a pulp only issue?

If i take away the S in https, it works. I’ve since installed a real SSL cert… but the issue remains.

As you can see, the cert is working

[smeads]

Also… since adding the SSL cert… foreman will not let me delete hosts

[smeads]

[smeads]

Action:

Actions::BulkAction

Input:

{“action_class”=>“Actions::Katello::Host::Destroy”, “target_ids”=>[9, 10], “target_class”=>“Host::Managed”, “args”=>, “current_user_id”=>6}

Output:

{“planned_count”=>0, “cancelled_count”=>0, “total_count”=>2, “failed_count”=>2, “pending_count”=>0, “success_count”=>0}

Exception:

RuntimeError: A sub task failed

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:230:in check_for_errors!' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:122:inwait_for_sub_plans’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:42:in spawn_plans' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:70:inspawn_plans’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:36:in initiate' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:35:ininitiate’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:16:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:inblock in assigns’ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:inassigns’ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:inblock in match?’ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:inmatch?’ /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_sub_plans.rb:11:inrun’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/with_bulk_sub_plans.rb:23:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/bulk_action.rb:21:inrun’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:538:in block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:inpass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:14:inblock in run’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/rails_executor_wrap.rb:13:inrun’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:inpass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:30:inwith_progress_calculation’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action/progress.rb:16:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incall’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpass’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:11:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:22:inwith_locale’ /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.8.1/app/lib/actions/middleware/keep_locale.rb:11:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incall’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpass’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:43:inrestore_curent_user’ /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4/app/lib/actions/middleware/keep_current_user.rb:15:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incall’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:26:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:18:inpass’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware.rb:31:in run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/stack.rb:22:incall’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/middleware/world.rb:30:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:537:inblock (2 levels) in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:in catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:536:inblock in execute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:incatch’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:451:in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:531:inexecute_run’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/action.rb:278:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:inblock (2 levels) in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract.rb:162:in with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:inblock in execute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:inexecute’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/director.rb:43:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:12:inblock in on_message’ /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors.rb:12:in run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/executors/parallel/worker.rb:11:inon_message’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.1.0/lib/dynflow/actor.rb:26:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:inprocess_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:inon_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:inblock in on_envelope’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:inblock in synchronize’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:insynchronize’ /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in block in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:incall’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in work' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:inblock in call_job’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in run_task' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:inblock (3 levels) in create_worker’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in loop' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:inblock (2 levels) in create_worker’ /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in catch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:inblock in create_worker’ /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_c

[smeads]

additionally… all new servers now fail

chef-client on 10.64.43.16 APP [3:45 PM]
Chef run failed on env us_dev, node us08dv2app23
rhsm_register[US08DV2APP23] (pl_katello_client::default line 23) had an error: Mixlib::ShellOut::ShellCommandFailed: execute[Register to RHSM] (/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.7.17/lib/chef/resource/rhsm_register.rb line 87) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘70’
---- Begin output of subscription-manager register --activationkey=Centos --org=Protolabs ----
STDOUT:
STDERR: Unable to verify server’s identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)
---- End output of subscription-manager register --activationkey=Centos --org=Protolabs ----
Ran subscription-manager register --activationkey=Centos --org=Protolabs returned 70

[W73]

I am not sure if i can help you. But i did SSL change last week for foreman with katello scenario without manual changes in configurations.

I used katello utility script
usage: /sbin/katello-certs-check -c CERT_FILE -k KEY_FILE -b CA_BUNDLE_FILE

Then I ran command
/sbin/katello-certs-check -c /etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt -k /etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key -b /etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem

it checked certs and CA, and generated command for foreman-installer .
All was done automatically.

Then i just had to send new CA authority to all clients we had registered to foreman and it works great.

It was painless. But i see you use chef and i am not sure if it will work for you too.

[smeads]

I will try that… thank you. All my servers are reporting problems with
foreman now. I tried manual steps yesterday, but it all failed. Did you
change your host name to one that matches the cert?

Let’s say the machine is: server1.internaldomain.com

And the ssl cert is:
Forman.mycompanyname.com

I want everything to use the ssl cert name, and not the internal name. Make
sense?

[W73]

I am not sure it will work if you have certificate for different hostname.
You may go with self-signed certificate

I asked our CA to generate(or sign) for exact hostname. pr0mgm01.blabla.com (it is in our corporate domain and internal DNS and goal for this was to have green lock in our corporate network)

Because our CA is Windows team, they gave us pfx certificate.
So i extracted certificate pr0mgm01_encrypted.crt. key pr0mgm01_encrypted_pem.key and bundle from pfx .
I run
/sbin/katello-certs-check on pr0mgm01.blabla.com server.

And that was all.
I tried 2 weeks to understand all those certificates in Foreman, but then i luckily found that utility script.
I saw that HowTo from 2015 too, but i would not follow it now.

Give that script chance. It generate something like
’
Checking server certificate’s encoding: [OK]
Checking expiration of certificate: [OK]
Checking expiration of CA bundle: [OK]
Checking if server cert has CA:TRUE flag[OK]
Validating the certificate subject= /CN=pr0mgm01.blabla.com
Checking to see if the private key matches the certificate: [OK]
Checking ca bundle against the cert file: [OK]
Checking Subject Alt Name on certificate[OK]
Checking Key Usage extension on certificate for Key Encipherment[OK]

Validation succeeded.

To install the Katello main server with the custom certificates, run:

foreman-installer --scenario katello\
                  --certs-server-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt"\
                  --certs-server-key "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key"\
                  --certs-server-ca-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem"\

To update the certificates on a currently running Katello installation, run:

foreman-installer --scenario katello\
                  --certs-server-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted.crt"\
                  --certs-server-key "/etc/pki/pr0mgm01-ssl/pr0mgm01_encrypted_pem.key"\
                  --certs-server-ca-cert "/etc/pki/pr0mgm01-ssl/pr0mgm01_cabundle.pem"\
                  --certs-update-server --certs-update-server-ca

’

And foreman-installer will care of it.

[smeads]

Thanks for the information again… it didn’t work…

[smeads]

I ran the cert check by itself… and it validated… but when I run the command they (and you) suggest… it errors out. Seems to me this is an issue with their software.

And here’s the command I copied:

[tbrisker]

can someone from @katello team please take a look?

[smeads]

Hey guys… please do not invest any more time on this. We’ve made the decision to move away from this product line. For those that helped… I sincerely appreciate your time. If anyone from the @katello team is interested in a debriefing, please reach out to me privately. @tbrisker @W73, thanks again.

[martin_angermeier]

We got some weird behaviour with the subscription-manager these days too…maybe it’s related.

Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Forbidden

That’s what I got if I’m trying to sync some repos…standard repos to be exact (RHEL 7.5)

[W73]

Check your imported Manifest file from RedHat. It may be expired.

[martin_angermeier]

Syncs are working again…but that was just one of many problems these days with subscriptions. Even the subscription plugin for yum took ages and so on. Just stated that I’ve the feeling of external changes involved.