Problem:
Hello everyone I was upgrading my Foreman+Katello instance from version 3.6/4.8 to latest, it was ok until the upgrade from 3.8 to 3.9 which make me struggle. First I had this error :
2024-08-21 16:25:25 [NOTICE] [root] Loading installer configuration. This will take some time.
2024-08-21 16:25:27 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2024-08-21 16:25:27 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2024-08-21 16:25:31 [NOTICE] [configure] Starting system configuration.
2024-08-21 16:25:38 [NOTICE] [configure] 250 configuration steps out of 1571 steps complete.
2024-08-21 16:25:39 [NOTICE] [configure] 500 configuration steps out of 1573 steps complete.
2024-08-21 16:25:41 [NOTICE] [configure] 750 configuration steps out of 1576 steps complete.
2024-08-21 16:25:41 [NOTICE] [configure] 1000 configuration steps out of 1582 steps complete.
2024-08-21 16:25:42 [NOTICE] [configure] 1250 configuration steps out of 1582 steps complete.
2024-08-21 16:25:56 [ERROR ] [configure] '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-21 16:25:56 [ERROR ] [configure] /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
2024-08-21 16:26:08 [NOTICE] [configure] 1500 configuration steps out of 1582 steps complete.
2024-08-21 16:26:18 [NOTICE] [configure] System configuration has finished.
Error 1: Puppet Exec resource 'foreman-rake-db:migrate' failed. Logs:
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]
Adding autorequire relationship with User[foreman]
Starting to evaluate the resource (1331 of 1582)
Evaluated in 13.56 seconds
Exec[foreman-rake-db:migrate](provider=posix)
Executing check '/usr/sbin/foreman-rake db:abort_if_pending_migrations'
Executing '/usr/sbin/foreman-rake db:migrate'
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/unless
rake aborted!
**Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)**
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
/usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
/usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
/usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
/usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:abort_if_pending_migrations => db:load_config => environment
(See full trace by running task with --trace)
/Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns
rake aborted!
**Errno::EAFNOSUPPORT: Address family not supported by protocol - socket(2)**
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:21:in `initialize'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `new'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:151:in `connect_addrinfo'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:193:in `block in connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `each_with_index'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:191:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/connection/ruby.rb:305:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:385:in `establish_connection'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:117:in `block in connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:344:in `with_reconnect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:116:in `connect'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:417:in `ensure_connected'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:269:in `block in process'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:356:in `logging'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:268:in `process'
/usr/share/gems/gems/redis-4.5.1/lib/redis/client.rb:162:in `call'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:564:in `block in del'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `block in synchronize'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:72:in `synchronize'
/usr/share/gems/gems/redis-4.5.1/lib/redis.rb:563:in `del'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block (2 levels) in delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:25:in `with'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:418:in `block in delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/redis_cache_store.rb:417:in `delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache/strategy/local_cache.rb:170:in `delete_entry'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:487:in `block in delete'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `block in instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/notifications.rb:205:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:726:in `instrument'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/cache.rb:486:in `delete'
/usr/share/foreman/app/services/topbar_sweeper.rb:18:in `block in expire_cache_all_users'
/usr/share/foreman/app/services/topbar_sweeper.rb:17:in `each'
/usr/share/foreman/app/services/topbar_sweeper.rb:17:in `expire_cache_all_users'
/usr/share/foreman/config/initializers/foreman.rb:35:in `block in <top (required)>'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `instance_exec'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:427:in `block in make_lambda'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:198:in `block (2 levels) in halting'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:604:in `block (2 levels) in default_terminator'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `catch'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:603:in `block in default_terminator'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:199:in `block in halting'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `block in invoke_before'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `each'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:512:in `invoke_before'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/callbacks.rb:105:in `run_callbacks'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/reloader.rb:88:in `prepare!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application/finisher.rb:124:in `block in <module:Finisher>'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `instance_exec'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:32:in `run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'
/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:61:in `block in run_initializers'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/initializable.rb:60:in `run_initializers'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:391:in `initialize!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `public_send'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/railtie.rb:207:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/usr/share/gems/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'
/usr/share/gems/gems/zeitwerk-2.6.12/lib/zeitwerk/kernel.rb:38:in `require'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `block in require'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:299:in `load_dependency'
/usr/share/gems/gems/activesupport-6.1.7.7/lib/active_support/dependencies.rb:332:in `require'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:367:in `require_environment!'
/usr/share/gems/gems/railties-6.1.7.7/lib/rails/application.rb:533:in `block in run_tasks_blocks'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => db:migrate => db:load_config => environment
(See full trace by running task with --trace)
change from 'notrun' to ['0'] failed: '/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]
1 error was detected during installation.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.
The full log is at /var/log/foreman-installer/katello.log
Which I immediately linked to disabled IPV6 on my server NIC, it seems to be more of a warning than an error, postgresql start correctly even with this error.
If I try to run manually a db:migrate it seems OK :
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke plugin:refresh_migrations (first_time)
** Invoke environment
** Execute plugin:refresh_migrations
** Execute db:migrate
** Invoke db:_dump (first_time)
** Execute db:_dump
** Invoke dynflow:migrate (first_time)
** Invoke environment
** Execute dynflow:migrate
If I try to stop/start foreman services after this, I get this error :
Running Start Services
================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Start applicable services:
Starting the following service(s):
redis, postgresql, pulpcore-api, pulpcore-content, pulpcore-worker@1.service, pulpcore-worker@2.service, pulpcore-worker@3.service, pulpcore-worker@4.service, tomcat, dynflow-sidekiq@orchestrator, foreman, httpd, dynflow-sidekiq@worker-1, dynflow-sidekiq@worker-hosts-queue-1, foreman-proxy
/ starting httpd
Job for dynflow-sidekiq@orchestrator.service failed because the control process exited with error code.
See "systemctl status dynflow-sidekiq@orchestrator.service" and "journalctl -xe" for details.
Job for foreman.service failed because the control process exited with error code.
See "systemctl status foreman.service" and "journalctl -xe" for details.
- starting dynflow-sidekiq@worker-hosts-queue-1
Job for dynflow-sidekiq@worker-1.service failed because the control process exited with error code.
See "systemctl status dynflow-sidekiq@worker-1.service" and "journalctl -xe" for details.
Job for dynflow-sidekiq@worker-hosts-queue-1.service failed because the control process exited with error code.
See "systemctl status dynflow-sidekiq@worker-hosts-queue-1.service" and "journalctl -xe" for details.
/ All services started [OK]
--------------------------------------------------------------------------------
I saw this error in journalctl :
ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found: [:view_lookup_values, :create_lookup_values, :edit_lookup_values, :destroy_lookup_values] (Foreman::PermissionMissingException)
And I’m stuck here, I saw another post that was talking about downgrading python jinja2 package which I tried but I don’t get better results with it.
Expected outcome:
Working upgrade to 3.9/4.11
Foreman and Proxy versions:
Foreman 3.8 Katello 4.10
Foreman and Proxy plugin versions:
Distribution and version:
Rocky Linux 8.10
Thank you for your help
but I still have the missing permission error, I found an RH KB for other missing permissions in Satellite 6.8 upgrade and they fix it by adding them directly in the database :