Upgrade Foreman 3.6 to 3.7 but Postgresql 10

Neeloj · September 5, 2023, 2:27pm

foreman=# show server_version;
            server_version             
---------------------------------------
 12.16 (Ubuntu 12.16-0ubuntu0.20.04.1)
(1 row)

Its version 12, but what I cannot understand its, after I change it back to version 10 and I run

# runuser -u postgres -- reindexdb -a

it goes automatically to version 12! I dont know why!

areyus · September 6, 2023, 6:05am

To be honest, I have no idea how that’s happening

Neeloj · September 6, 2023, 6:11am

same here ehmmm I’ll restart the server and then check all again

Neeloj · September 6, 2023, 7:13am

@areyus after yesterday changes I have big problems with foreman:
I get this error in journalctl do you have any idea please ?
Failed to start Foreman jobs daemon - worker-1 on sidekiq.

Neeloj · September 6, 2023, 9:11am

I think its related to the changes which I/(we) make yesterday, do you have any idea how I can solve this @areyus thank you again:


f9293ee9 | /usr/share/foreman/vendor/ruby/2.7.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-09-06T11:09:27 [E|app|9e8c232b] Fact aio_agent_version could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [I|app|f9293ee9]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 0.4ms | Allocations: 97)
2023-09-06T11:09:27 [I|app|f9293ee9]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.0ms | Allocations: 562)
2023-09-06T11:09:27 [I|app|f9293ee9] Completed 500 Internal Server Error in 105ms (Views: 3.7ms | ActiveRecord: 60.6ms | Allocations: 29682)
2023-09-06T11:09:27 [E|app|9e8c232b] Fact facterversion could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact identity::user could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact identity::privileged could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact id could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact processor0 could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact processor1 could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact interfaces could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact fips_enabled could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact ipaddress_Ethernet could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact is_virtual could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact kernelmajversion could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact kernelrelease could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact kernelversion could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact macaddress_Ethernet could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact timezone could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact system_uptime::days could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact system_uptime::hours could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
 9e8c232b | 
2023-09-06T11:09:27 [E|app|9e8c232b] Fact system_uptime::seconds could not be imported because of PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block

areyus · September 6, 2023, 9:23am

I can only guess at this point, but it looks like your DB is in some broken state, probably because you are somehow running on the postgresql 12 database that you tried to import the dump with the problematic indexes to.
What is the output of

systemctl status postgresql@10-main.service
systemctl status postgresql@12-main.service

?

areyus · September 6, 2023, 9:33am

As expected, postgres 12 is the active DB for some reason.
To get you back up and running properly again, I suggest:

foreman-maintain service stop
systemctl stop postgresql@12-main.service
systemctl disable postgresql@12-main.service 
systemctl start postgresql@10-main.service 
systemctl enable postgresql@10-main.service 
foreman-maintain service start

I can’t remember if foreman-maintain exists for deb based systems. If not, I think you can replace that with systemctl stop/start foreman

areyus · September 6, 2023, 10:08am

According to the systemctl man page, enabled-runtime means it is enabled via some sort of dependency.
Glad to hear it somehow works not, although I’m not sure why this chain of actions lead to the desired outcome…
You might want to check a little more on where that dependency for the postgres 10 service comes from and sort that out. Once you are sure you don’t need to postgres 10 anymore and everything works on postgres 12, I suggest getting rid of the postgres 10 installation.

Neeloj · September 6, 2023, 10:45am

thank you so much for your help I really appreciate it @areyus

it was my mistake I stopped the version 12 after restart the server and I started the 10 manually ( panic ) so I stop version 10 and start version 12 again and then I dont see any errors.

You are absolutely right, so I can easily remove it right ? it doesnt effect version 12 right ?

Neeloj · September 6, 2023, 10:51am

I only those two error:puppetserver.log

2023-09-06T12:36:36.761+02:00 ERROR [qtp2121333850-56] [p.s.c.certificate-authority-core] ptest.local already has a requested certificate; ignoring certificate request
2023-09-06T12:36:37.033+02:00 ERROR [qtp2121333850-56] [p.s.c.certificate-authority-core] sfact.local already has a requested certificate; ignoring certificate request
2023-09-06T12:37:50.875+02:00 ERROR [qtp2121333850-56] [p.s.c.certificate-authority-core] test .intranet already has a requested certificate; ignoring certificate request

and only this one in production.log:

2023-09-06T12:26:11 [I|aud|bb7c15ee] Host::Base (719) create event on build_errors

areyus · September 6, 2023, 11:30am

As mentioned, I don’t know ubuntu very well, but I would assume that should be the case. But I guess that should be easy to google

Those sound more like problems with your hosts. Looks like some hosts are trying to request new certificates despite them already existing. You might want to check that, but this is definetly not related.

I’m pretty sure that’s not even an error. This looks to be a regular step in host creation. I don’t know what it does, but we have that one in our logs, too

Neeloj · September 6, 2023, 11:38am

before I ask I google/then search here when I dont find anything I ask for help you are right I’ll do that

I solve it already some, revoke the key and create new one for clients

you are right Im so happy!! thank you so much Today was to much headache

Neeloj · September 6, 2023, 1:47pm

I have same error again now

production.log:


2023-09-06T15:42:42 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:42:42 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:42:43 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:42:43 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:42:44 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:42:44 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:42:50 [I|app|] Rails cache backend: File
2023-09-06T15:42:51 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:42:52 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:42:52 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:42:56 [I|app|] Rails cache backend: File
2023-09-06T15:42:57 [I|app|] Rails cache backend: File
2023-09-06T15:42:57 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:42:57 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:42:58 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:42:58 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:42:58 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:42:59 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:43:05 [I|app|] Rails cache backend: File
2023-09-06T15:43:06 [W|app|] You are trying to replace import_subnets from . Adding allowed actions from plugin permissions to the existing one.
2023-09-06T15:43:07 [W|app|] Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
2023-09-06T15:43:07 [W|app|] Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
2023-09-06T15:43:11 [I|app|] Rails cache backend: File
2023-09-06T15:43:12 [I|app|] Rails cache backend: File

puppetserver.log


2023-09-06T15:44:30.338+02:00 INFO  [qtp2121333850-27055] [puppetserver] Puppet Compiled catalog for test.local in environment production in 0.15 seconds
2023-09-06T15:44:30.349+02:00 WARN  [qtp2121333850-27055] [o.e.j.s.HttpChannel] handleException /puppet/v3/catalog/test.local java.io.IOException: Broken pipe
2023-09-06T15:45:27.314+02:00 WARN  [qtp2121333850-46406] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: execution expired

2023-09-06T15:45:27.470+02:00 INFO  [qtp2121333850-46406] [puppetserver] Puppet Compiled catalog for test.internal in environment production in 0.15 seconds
2023-09-06T15:45:28.134+02:00 WARN  [qtp2121333850-32377] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: execution expired

I dont change anything! maybe you have an idea… @areyus

when I want restart foreman:


ep 06 15:48:22 test systemd[1]: foreman.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- An ExecStart= process belonging to unit foreman.service has exited.
-- 
-- The process' exit code is 'exited' and its exit status is 1.
Sep 06 15:48:22 test systemd[1]: foreman.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- The unit foreman.service has entered the 'failed' state with result 'exit-code'.
Sep 06 15:48:22 test systemd[1]: Failed to start Foreman.
-- Subject: A start job for unit foreman.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit foreman.service has finished with a failure.
-- 
-- The job identifier is 334665 and the job result is failed.
Sep 06 15:48:22 test audit[1]: SERVICE_START pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=foreman comm="systemd" exe="/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
Sep 06 15:48:22 test systemd[1]: Starting Foreman...
-- Subject: A start job for unit foreman.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
-- 
-- A start job for unit foreman.service has begun execution.
-- 
-- The job identifier is 334932.
Sep 06 15:48:24 test qemu-ga[879]: info: guest-ping called
Sep 06 15:48:27 test foreman[261700]: => Booting Puma
Sep 06 15:48:27 test foreman[261700]: => Rails 6.1.7.4 application starting in production
Sep 06 15:48:27 test foreman[261700]: => Run `bin/rails server --help` for more startup options

Neeloj · September 7, 2023, 5:51am

I decide yesterday to upgrade from 3.6 to 3.7 because I had the problem again, through the upgrade process I get below error, do you have any idea how can I solve it:

~# foreman-rake db:seed --trace

** Invoke db:seed (first_time)

** Invoke db:load_config (first_time)

** Invoke environment (first_time)

** Execute environment

rake aborted!

ActiveRecord::UnknownPrimaryKey: Unknown primary key for table permissions in model Permission.

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:307:in `primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:719:in `association_primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:724:in `join_primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:872:in `join_primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:872:in `join_primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/reflection.rb:1050:in `join_primary_key'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:79:in `next_chain_scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:121:in `block in add_constraints'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:120:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:120:in `each_cons'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:120:in `add_constraints'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:29:in `scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association_scope.rb:7:in `scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association.rb:244:in `association_scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/association.rb:103:in `scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/collection_association.rb:298:in `scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/collection_proxy.rb:930:in `scope'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/associations/collection_proxy.rb:1109:in `where'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:100:in `block (2 levels) in include_permissions_for_role'

/usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.3/lib/audited/auditor.rb:448:in `without_auditing'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:99:in `block in include_permissions_for_role'

/usr/share/foreman/app/models/role.rb:239:in `ignore_locking'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:98:in `include_permissions_for_role'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:112:in `add_all_permissions_to_role'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:11:in `block in add_all_permissions_to_default_roles'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `block in transaction'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/database_statements.rb:320:in `transaction'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/transactions.rb:209:in `transaction'

/usr/share/foreman/app/registries/foreman/plugin/rbac_support.rb:10:in `add_all_permissions_to_default_roles'

/usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:73:in `block (2 levels) in setup_roles!'

/usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.3/lib/audited/auditor.rb:448:in `without_auditing'

/usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:59:in `block in setup_roles!'

/usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.3/lib/audited/auditor.rb:448:in `without_auditing'

/usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:58:in `setup_roles!'

/usr/share/foreman/app/registries/foreman/plugin/rbac_registry.rb:46:in `setup!'

/usr/share/foreman/app/registries/foreman/plugin.rb:309:in `finalize_setup!'

/usr/share/foreman/config/initializers/foreman.rb:28:in `block (2 levels) in <top (required)>'

/usr/share/foreman/config/initializers/foreman.rb:27:in `each'

/usr/share/foreman/config/initializers/foreman.rb:27:in `block in <top (required)>'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:68:in `block in execute_hook'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:61:in `with_execution_control'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:66:in `execute_hook'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:52:in `block in run_load_hooks'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:51:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/lazy_load_hooks.rb:51:in `run_load_hooks'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/application/finisher.rb:140:in `block in <module:Finisher>'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/initializable.rb:32:in `instance_exec'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/initializable.rb:32:in `run'

/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `block in run'

/usr/share/foreman/config/initializers/0_print_time_spent.rb:17:in `benchmark'

/usr/share/foreman/config/initializers/0_print_time_spent.rb:45:in `run'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/initializable.rb:61:in `block in run_initializers'

/usr/lib/ruby/2.7.0/tsort.rb:228:in `block in tsort_each'

/usr/lib/ruby/2.7.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'

/usr/lib/ruby/2.7.0/tsort.rb:431:in `each_strongly_connected_component_from'

/usr/lib/ruby/2.7.0/tsort.rb:349:in `block in each_strongly_connected_component'

/usr/lib/ruby/2.7.0/tsort.rb:347:in `each'

/usr/lib/ruby/2.7.0/tsort.rb:347:in `call'

/usr/lib/ruby/2.7.0/tsort.rb:347:in `each_strongly_connected_component'

/usr/lib/ruby/2.7.0/tsort.rb:226:in `tsort_each'

/usr/lib/ruby/2.7.0/tsort.rb:205:in `tsort_each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/initializable.rb:60:in `run_initializers'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/application.rb:391:in `initialize!'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/railtie.rb:207:in `public_send'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/railtie.rb:207:in `method_missing'

/usr/share/foreman/config/environment.rb:5:in `<top (required)>'

/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'

/usr/share/foreman/vendor/ruby/2.7.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/dependencies.rb:332:in `block in require'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/dependencies.rb:299:in `load_dependency'

/usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.4/lib/active_support/dependencies.rb:332:in `require'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/application.rb:367:in `require_environment!'

/usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.4/lib/rails/application.rb:533:in `block in run_tasks_blocks'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:243:in `block in invoke_prerequisites'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:241:in `invoke_prerequisites'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:218:in `block in invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'

/usr/share/foreman/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'

/usr/bin/rake:23:in `load'

/usr/bin/rake:23:in `<main>'

Tasks: TOP => db:seed => db:load_config => environment

Neeloj · September 7, 2023, 6:02am

I think this is the error duplicate primary keys in permissions table:


id  |              name              |            resource_type            |         created_at         |         updated_at         
-----+--------------------------------+-------------------------------------+----------------------------+----------------------------
   1 | view_config_groups             | ForemanPuppet::ConfigGroup          | 2023-09-05 11:17:34.898756 | 2023-09-05 11:17:34.898756
   2 | create_config_groups           | ForemanPuppet::ConfigGroup          | 2023-09-05 11:17:34.910377 | 2023-09-05 11:17:34.910377
   3 | edit_config_groups             | ForemanPuppet::ConfigGroup          | 2023-09-05 11:17:34.920494 | 2023-09-05 11:17:34.920494
   4 | destroy_config_groups          | ForemanPuppet::ConfigGroup          | 2023-09-05 11:17:34.926208 | 2023-09-05 11:17:34.926208
   5 | view_external_parameters       | ForemanPuppet::PuppetclassLookupKey | 2023-09-05 11:17:34.931477 | 2023-09-05 11:17:34.931477
   6 | create_external_parameters     | ForemanPuppet::PuppetclassLookupKey | 2023-09-05 11:17:34.936473 | 2023-09-05 11:17:34.936473
   7 | edit_external_parameters       | ForemanPuppet::PuppetclassLookupKey | 2023-09-05 11:17:34.941615 | 2023-09-05 11:17:34.941615
   8 | destroy_external_parameters    | ForemanPuppet::PuppetclassLookupKey | 2023-09-05 11:17:34.946533 | 2023-09-05 11:17:34.946533
   9 | view_environments              | ForemanPuppet::Environment          | 2023-09-05 11:17:34.951383 | 2023-09-05 11:17:34.951383
  10 | create_environments            | ForemanPuppet::Environment          | 2023-09-05 11:17:34.956186 | 2023-09-05 11:17:34.956186
  11 | edit_environments              | ForemanPuppet::Environment          | 2023-09-05 11:17:34.961131 | 2023-09-05 11:17:34.961131
  12 | destroy_environments           | ForemanPuppet::Environment          | 2023-09-05 11:17:34.965971 | 2023-09-05 11:17:34.965971
  13 | import_environments            | ForemanPuppet::Environment          | 2023-09-05 11:17:34.971146 | 2023-09-05 11:17:34.971146
  14 | view_puppetclasses             | ForemanPuppet::Puppetclass          | 2023-09-05 11:17:34.976055 | 2023-09-05 11:17:34.976055
  15 | create_puppetclasses           | ForemanPuppet::Puppetclass          | 2023-09-05 11:17:34.980722 | 2023-09-05 11:17:34.980722
  16 | edit_puppetclasses             | ForemanPuppet::Puppetclass          | 2023-09-05 11:17:34.985657 | 2023-09-05 11:17:34.985657
  17 | destroy_puppetclasses          | ForemanPuppet::Puppetclass          | 2023-09-05 11:17:34.990728 | 2023-09-05 11:17:34.990728
  18 | import_puppetclasses           | ForemanPuppet::Puppetclass          | 2023-09-05 11:17:34.995585 | 2023-09-05 11:17:34.995585
  19 | edit_classes                   | ForemanPuppet::HostClass            | 2023-09-05 11:17:35.000833 | 2023-09-05 11:17:35.000833
   1 | view_architectures             | Architecture                        | 2021-09-16 07:05:18.63034  | 2021-09-16 07:05:18.63034
   2 | create_architectures           | Architecture                        | 2021-09-16 07:05:18.637814 | 2021-09-16 07:05:18.637814
   3 | edit_architectures             | Architecture                        | 2021-09-16 07:05:18.642747 | 2021-09-16 07:05:18.642747
   4 | destroy_architectures          | Architecture                        | 2021-09-16 07:05:18.647321 | 2021-09-16 07:05:18.647321
   5 | view_audit_logs                | Audit                               | 2021-09-16 07:05:18.652067 | 2021-09-16 07:05:18.652067
   6 | view_authenticators            | AuthSource                          | 2021-09-16 07:05:18.656795 | 2021-09-16 07:05:18.656795
   7 | create_authenticators          | AuthSource                          | 2021-09-16 07:05:18.66411  | 2021-09-16 07:05:18.66411
   8 | edit_authenticators            | AuthSource                          | 2021-09-16 07:05:18.669191 | 2021-09-16 07:05:18.669191
   9 | destroy_authenticators         | AuthSource                          | 2021-09-16 07:05:18.673965 | 2021-09-16 07:05:18.673965
  10 | create_bookmarks               | Bookmark                            | 2021-09-16 07:05:18.679085 | 2021-09-16 07:05:18.679085
  11 | edit_bookmarks                 | Bookmark                            | 2021-09-16 07:05:18.683903 | 2021-09-16 07:05:18.683903
  12 | destroy_bookmarks              | Bookmark                            | 2021-09-16 07:05:18.688765 | 2021-09-16 07:05:18.688765
  13 | view_compute_profiles          | ComputeProfile                      | 2021-09-16 07:05:18.693697 | 2021-09-16 07:05:18.693697
  14 | create_compute_profiles        | ComputeProfile                      | 2021-09-16 07:05:18.698624 | 2021-09-16 07:05:18.698624
  15 | edit_compute_profiles          | ComputeProfile                      | 2021-09-16 07:05:18.703837 | 2021-09-16 07:05:18.703837
  16 | destroy_compute_profiles       | ComputeProfile                      | 2021-09-16 07:05:18.70885  | 2021-09-16 07:05:18.70885
  17 | view_compute_resources         | ComputeResource                     | 2021-09-16 07:05:18.713595 | 2021-09-16 07:05:18.713595
  18 | create_compute_resources       | ComputeResource                     | 2021-09-16 07:05:18.718402 | 2021-09-16 07:05:18.718402
  19 | edit_compute_resources         | ComputeResource                     | 2021-09-16 07:05:18.723157 | 2021-09-16 07:05:18.723157
  20 | destroy_compute_resources      | ComputeResource                     | 2021-09-16 07:05:18.727994 | 2021-09-16 07:05:18.727994
  21 | power_vm_compute_resources     | ComputeResource                     | 2021-09-16 07:05:18.732622 | 2021-09-16 07:05:18.732622
  22 | destroy_vm_compute_resources   | ComputeResource                     | 2021-09-16 07:05:18.737678 | 2021-09-16 07:05:18.737678
  23 | view_compute_resources_vms     | ComputeResource                     | 2021-09-16 07:05:18.742396 | 2021-09-16 07:05:18.742396
  24 | create_compute_resources_vms   | ComputeResource                     | 2021-09-16 07:05:18.747358 | 2021-09-16 07:05:18.747358
  25 | edit_compute_resources_vms     | ComputeResource                     | 2021-09-16 07:05:18.753023 | 2021-09-16 07:05:18.753023
  26 | destroy_compute_resources_vms  | ComputeResource                     | 2021-09-16 07:05:18.758193 | 2021-09-16 07:05:18.758193
  27 | power_compute_resources_vms    | ComputeResource                     | 2021-09-16 07:05:18.763632 | 2021-09-16 07:05:18.763632

areyus · September 7, 2023, 6:29am

General rule of thumb: When your application suddenly stops working, especially when it’s probably related to DB problems, upgrading is usually a bad idea. Do you have a backup from before you tried upgrading yesterday?
If so, I would highly recommend restoring that backup and we try to figure out why it stopped working in the first place. If not, this might get ugly.

Neeloj · September 7, 2023, 6:33am

yes I have both backups 10 and 12 before upgrading.

okay should I easily restore
pg_restore -d foreman foreman_12_2023.09.06_14-56.dump
when its finish, should I start foreman like:

systemctl start apache2 foreman.service foreman.socket

by the way above error is missing “permissins_pkey” table just want to notice it.

areyus · September 7, 2023, 6:36am

Do you have a full machine backup? That would probably be better, otherwise your still stuck in basically the same state.

Neeloj · September 7, 2023, 6:40am

its not so easy to get that back! I know you are right but I cant go to backup because more than 10 new hosts are connected since yesterday! as I guess its working again

should I restore that or create the permissions_pkey index… does this table contain also data ?

should I delete the duplicate entries ( primary-key 1,2,3 etc. ) ?

Neeloj · September 7, 2023, 7:05am

@areyus we can give it a try if you can tell me how can I add the permissions_pkey index to the foreman database ?

I compare all with our test system, only the index is missing…