Update to 2.4.0 with Katello 4.0 fails

Support
, , ,
#1

Problem:
Update to 2.4.0 with Katello 4.0 fails with this message:

Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/4: 3-8-default-v20_0-puppet-ff4590c3-68ae-4507-8c76-8c0e305e30bf (37)
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.

If I run the command on its own I get this output:

foreman-rake katello:correct_puppet_environments
All operations will be skipped.  Re-run with COMMIT=true to perform corrections.
Processing Puppet Environment 1/4: 3-8-default-v20_0-puppet-ff4590c3-68ae-4507-8c76-8c0e305e30bf (37)
rake aborted!
Cannot find repository type puppet, is it enabled?
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:226:in `pulp3_repository_type_support?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:197:in `pulp3_support?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:140:in `repo_exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:81:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_puppet_environments
(See full trace by running task with --trace)

Expected outcome:
Installer upgrades to 2.4.0 with Katello 4.0.

Foreman and Proxy versions:
Foreman 2.4.0
Katello 4.0

Proxies are on 2.2.2 and Katello 3.17, except one which I installed too early with 2.4.0 and Katello 4.0

Distribution and version:
CentOS 7 with patches from Februar/March

Other relevant data:
Basic installation after following the documentation on theforeman.org.

#2

Which version are you updating from?

#3

The system had been upgraded to Foreman 2.3.3 with Katello 3.18.2,
including migration to pulp3.

#4

@foreman-materna Seems as if the puppet content type isn’t enabled. Can you post the contents of your foreman config/settings.plugin.d/katello.yaml?

#5

@jjeffers
I would like to, but such a path doesn’t exist on my system.
Instead “find -name katello.yml” found these:

/etc/foreman/plugins/katello.yaml
/etc/foreman-installer/scenarios.d/katello.yaml
/usr/share/foreman-installer/config/katello.yaml
/usr/share/foreman-installer/parser_cache/katello.yaml

Content of /etc/foreman/plugins/katello.yaml :

File managed with puppet

Module: puppet-katello

:katello:
:rest_client_timeout: 3600

:content_types:
:file: true
:yum: true
:deb: false
:puppet: false
:docker: true
:ostree: false

:candlepin:
:url: https://localhost:23443/candlepin
:oauth_key: “MyKey”
:oauth_secret: “MySecret”
:ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt

:candlepin_events:
:ssl_cert_file: /etc/pki/katello/certs/java-client.crt
:ssl_key_file: /etc/pki/katello/private/java-client.key
:ssl_ca_file: /etc/pki/katello/certs/katello-default-ca.crt

:agent:
:broker_url: amqps://localhost:5671
:event_queue_name: katello.agent

:katello_applicability: true

Content of /etc/foreman-installer/scenarios.d/katello.yaml:

:answer_file: “/etc/foreman-installer/scenarios.d/katello-answers.yaml”
:color_of_background: :dark
:colors: true
:custom:
:lock_package_versions: false
:description: Install Foreman with Katello
:dont_save_answers: false
:enabled: true
:facts:
tuning: default
mongo_cache_size: 4.67
:hiera_config: “/usr/share/foreman-installer/config/foreman-hiera.yaml”
:hook_dirs:

  • “/usr/share/foreman-installer/katello/hooks”
    :ignore_undocumented: false
    :installer_dir: “/usr/share/foreman-installer”
    :log_dir: “/var/log/foreman-installer”
    :log_level: DEBUG
    :log_name: katello.log
    :low_priority_modules:
    :mapping: {}
    :module_dirs: “/usr/share/foreman-installer/modules”
    :name: Katello
    :no_prefix: false
    :order:
  • certs
  • foreman
  • katello
  • foreman_proxy
  • foreman_proxy::plugin::pulp
  • foreman_proxy_content
  • puppet
    :parser_cache_path: “/usr/share/foreman-installer/parser_cache/katello.yaml”
    :skip_puppet_version_check: false
    :store_dir: ‘’
    :verbose: true
    :verbose_log_level: notice

I never touched those files, only the command foreman-installer, if it is responsible for both.

#6

Ok - that’s fine - I was forgetting that the production deployment paths are different!

We might have located a bug with a migration where there could be puppet repos left over when they should have been removed.

You can try opening the foreman console and run:
::Katello::ContentViewPuppetEnvironment.destroy_all

Then retry the upgrade.

1 Like
Certificates are invalid after update to 2.4.0 and Katello 4.0
#7

For tracking purposes a new issue was created - Bug #32590: 20210119162528_delete_puppet_and_ostree_repos.rb fails to remove puppet repos in CVs, leading to upgrade errors - Katello - Foreman

#8

The command you provided fixed it sort of.
Now I am getting to step 3 from 3, but getting another error:

Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.

Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.

I used this command “foreman-rake katello:clean_backend_objects --trace” and got this output:

RestClient::SSLCertificateNotVerified: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:758:in rescue in transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in retrieve’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in populate!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’

Caused by:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in connect_nonblock' /opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in ssl_socket_connect’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in connect' /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in do_start’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in retrieve’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in populate!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’
Tasks: TOP => katello:clean_backend_objects

We did not deploy a custom cert, so it is a bit surprising that now this error appears.