Problem:
Update to 2.4.0 with Katello 4.0 fails with this message:
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Processing Puppet Environment 1/4: 3-8-default-v20_0-puppet-ff4590c3-68ae-4507-8c76-8c0e305e30bf (37)
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
If I run the command on its own I get this output:
foreman-rake katello:correct_puppet_environments
All operations will be skipped. Re-run with COMMIT=true to perform corrections.
Processing Puppet Environment 1/4: 3-8-default-v20_0-puppet-ff4590c3-68ae-4507-8c76-8c0e305e30bf (37)
rake aborted!
Cannot find repository type puppet, is it enabled?
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:226:in `pulp3_repository_type_support?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:197:in `pulp3_support?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:140:in `repo_exists?'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:81:in `block (3 levels) in <top (required)>'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block (2 levels) in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in `block in find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in `block in find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in `block in in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `loop'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in `in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in `find_in_batches'
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in `find_each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in `with_index'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in `block (2 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:correct_puppet_environments
(See full trace by running task with --trace)
Expected outcome:
Installer upgrades to 2.4.0 with Katello 4.0.
Foreman and Proxy versions:
Foreman 2.4.0
Katello 4.0
Proxies are on 2.2.2 and Katello 3.17, except one which I installed too early with 2.4.0 and Katello 4.0
Distribution and version:
CentOS 7 with patches from Februar/March
Other relevant data:
Basic installation after following the documentation on theforeman.org .
Which version are you updating from?
The system had been upgraded to Foreman 2.3.3 with Katello 3.18.2,
including migration to pulp3.
@foreman-materna Seems as if the puppet content type isnât enabled. Can you post the contents of your foreman config/settings.plugin.d/katello.yaml?
@jjeffers
I would like to, but such a path doesnât exist on my system.
Instead âfind -name katello.ymlâ found these:
/etc/foreman/plugins/katello.yaml
/etc/foreman-installer/scenarios.d/katello.yaml
/usr/share/foreman-installer/config/katello.yaml
/usr/share/foreman-installer/parser_cache/katello.yaml
Content of /etc/foreman/plugins/katello.yaml :
File managed with puppet
Module: puppet-katello
:katello:
:rest_client_timeout: 3600
:content_types:
:file: true
true
:deb: false
:puppet: false
:docker: true
:ostree: false
:candlepin:
:url: https://localhost:23443/candlepin
:oauth_key: âMyKeyâ
:oauth_secret: âMySecretâ
:ca_cert_file: /etc/pki/katello/certs/katello-default-ca.crt
:candlepin_events:
:ssl_cert_file: /etc/pki/katello/certs/java-client.crt
:ssl_key_file: /etc/pki/katello/private/java-client.key
:ssl_ca_file: /etc/pki/katello/certs/katello-default-ca.crt
:agent:
:broker_url: amqps://localhost:5671
:event_queue_name: katello.agent
:katello_applicability: true
Content of /etc/foreman-installer/scenarios.d/katello.yaml:
:answer_file: â/etc/foreman-installer/scenarios.d/katello-answers.yamlâ
:color_of_background: :dark
:colors: true
:custom:
:lock_package_versions: false
:description: Install Foreman with Katello
:dont_save_answers: false
:enabled: true
:facts:
tuning: default
mongo_cache_size: 4.67
:hiera_config: â/usr/share/foreman-installer/config/foreman-hiera.yamlâ
:hook_dirs:
â/usr/share/foreman-installer/katello/hooksâ
:ignore_undocumented: false
:installer_dir: â/usr/share/foreman-installerâ
:log_dir: â/var/log/foreman-installerâ
:log_level: DEBUG
:log_name: katello.log
:low_priority_modules:
:mapping: {}
:module_dirs: â/usr/share/foreman-installer/modulesâ
:name: Katello
:no_prefix: false
:order:
certs
foreman
katello
foreman_proxy
foreman_proxy::plugin::pulp
foreman_proxy_content
puppet
:parser_cache_path: â/usr/share/foreman-installer/parser_cache/katello.yamlâ
:skip_puppet_version_check: false
:store_dir: ââ
:verbose: true
:verbose_log_level: notice
I never touched those files, only the command foreman-installer, if it is responsible for both.
Ok - thatâs fine - I was forgetting that the production deployment paths are different!
We might have located a bug with a migration where there could be puppet repos left over when they should have been removed.
You can try opening the foreman console and run:
::Katello::ContentViewPuppetEnvironment.destroy_all
Then retry the upgrade.
1 Like
foreman-materna:
foreman-rake katello:correct_puppet_environments
All operations will be skipped. Re-run with COMMIT=true to perform corrections.
Processing Puppet Environment 1/4: 3-8-default-v20_0-puppet-ff4590c3-68ae-4507-8c76-8c0e305e30bf (37)
rake aborted!
Cannot find repository type puppet, is it enabled?
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:226:in pulp3_repository_type_support?' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/app/models/katello/concerns/smart_proxy_extensions.rb:197:in
pulp3_support?â
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:140:in repo_exists?' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:81:in
block (3 levels) in <top (required)>â
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in block (2 levels) in find_each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in
eachâ
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:70:in block in find_each' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:136:in
block in find_in_batchesâ
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:238:in block in in_batches' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in
loopâ
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:222:in in_batches' /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:135:in
find_in_batchesâ
/opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation/batches.rb:69:in find_each' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in
with_indexâ
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/repository.rake:79:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in
<top (required)>â
Tasks: TOP => katello:correct_puppet_environments
(See full trace by running task with --trace)
For tracking purposes a new issue was created - Bug #32590: 20210119162528_delete_puppet_and_ostree_repos.rb fails to remove puppet repos in CVs, leading to upgrade errors - Katello - Foreman
The command you provided fixed it sort of.
Now I am getting to step 3 from 3, but getting another error:
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.
I used this command âforeman-rake katello:clean_backend_objects --traceâ and got this output:
RestClient::SSLCertificateNotVerified: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:758:in rescue in transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in
transmitâ
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in
executeâ
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in
get_responseâ
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in
retrieveâ
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in
populate!â
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in
block in executeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in
executeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in
mon_synchronizeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in
invokeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in
block (2 levels) in top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in
block in top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in
top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in
standard_exception_handlingâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in
<top (required)>â
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in
â
Caused by:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in connect_nonblock' /opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in
ssl_socket_connectâ
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in connect' /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in
do_startâ
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in
transmitâ
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in
executeâ
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in
get_responseâ
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in
retrieveâ
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in
populate!â
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in
block in executeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in
executeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in
mon_synchronizeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in
invokeâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in
block (2 levels) in top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in
block in top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in
top_levelâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in
standard_exception_handlingâ
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in
<top (required)>â
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in
â
Tasks: TOP => katello:clean_backend_objects
We did not deploy a custom cert, so it is a bit surprising that now this error appears.