Foreman We're sorry, but something went wrong

Hi all,

after upgrade Ubuntu from 18.04 to 20.04 we get this error:
foreman We’re sorry, but something went wrong.If you are the application owner check the logs for more information.

In the ~# less /var/log/foreman/production.log I could not find errors or so.

The status of foreman-proxy

~# systemctl status foreman-proxy
● foreman-proxy.service - Foreman Proxy
     Loaded: loaded (/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-05-14 10:21:16 CEST; 52min ago
   Main PID: 956 (ruby)
      Tasks: 3 (limit: 19088)
     Memory: 52.8M
     CGroup: /system.slice/foreman-proxy.service
             └─956 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Mai 14 10:21:12 ServerDM smart-proxy[956]: You should probably keep only one of them.
Mai 14 10:21:12 ServerDM smart-proxy[956]: Remove any duplicate entries and specify the gem only once.
Mai 14 10:21:12 ServerDM smart-proxy[956]: While it's not a problem now, it could cause errors if you change the version of one of them later.
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/regular.rb:22: warning: The called method `initialize' is defined here
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/regexp_based.rb:17: warning: The called method `initialize' is defined here
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:43: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:49: warning: The called method `pattern' is defined here
Mai 14 10:21:16 ServerDM systemd[1]: Started Foreman Proxy.

The status of the foreman:

# systemctl status foreman
● foreman.service - Foreman
     Loaded: loaded (/lib/systemd/system/foreman.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/foreman.service.d
             └─installer.conf
     Active: active (running) since Sat 2022-05-14 11:01:55 CEST; 8min ago
TriggeredBy: ● foreman.socket
       Docs: https://theforeman.org
   Main PID: 30915 (foreman-ruby)
      Tasks: 240 (limit: 19088)
     Memory: 3.2G
     CGroup: /system.slice/foreman.service
             ├─30915 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
             ├─30954 puma: cluster worker 0: 30915 [foreman]
             ├─30956 puma: cluster worker 1: 30915 [foreman]
             ├─30959 puma: cluster worker 2: 30915 [foreman]
             ├─30963 puma: cluster worker 3: 30915 [foreman]
             ├─30968 puma: cluster worker 4: 30915 [foreman]
             ├─30974 puma: cluster worker 5: 30915 [foreman]
             ├─30980 puma: cluster worker 6: 30915 [foreman]
             ├─30985 puma: cluster worker 7: 30915 [foreman]
             ├─30989 puma: cluster worker 8: 30915 [foreman]
             ├─30994 puma: cluster worker 9: 30915 [foreman]
             ├─31000 puma: cluster worker 10: 30915 [foreman]
             ├─31004 puma: cluster worker 11: 30915 [foreman]
             └─31008 puma: cluster worker 12: 30915 [foreman]

Mai 14 11:08:52 ServerDM foreman[31008]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:08:52 ServerDM foreman[31008]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here
Mai 14 11:10:03 ServerDM foreman[30994]: /usr/share/foreman/app/models/concerns/foreman/sti.rb:26: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:03 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:43: warning: The called method `save' is defined here
Mai 14 11:10:05 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:05 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here
Mai 14 11:10:28 ServerDM foreman[30974]: /usr/share/foreman/app/models/concerns/foreman/sti.rb:26: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:28 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:43: warning: The called method `save' is defined here
Mai 14 11:10:30 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:30 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here
apt policy foreman
foreman:
  Installiert:           3.0.2-2
  Installationskandidat: 3.0.2-2
  Versionstabelle:
 *** 3.0.2-2 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
        100 /var/lib/dpkg/status
     3.0.2-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.1-2 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.1-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0~rc2-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0~rc1-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages

:~# ls -alh /usr/share/foreman
total 84K
drwxr-xr-x  13 foreman foreman 4,0K Mai 14 10:59 .
drwxr-xr-x 167 root    root    4,0K Mai 14 10:17 ..
drwxr-xr-x  15 foreman foreman 4,0K Sep 16  2021 app
drwxr-xr-x   2 foreman foreman 4,0K Mai 14 10:59 bin
drwxr-xr-x   2 foreman foreman 4,0K Sep 16  2021 .bundle
drwxr-xr-x   2 foreman foreman 4,0K Mai 14 10:59 bundler.d
drwxr-xr-x   7 foreman foreman 4,0K Mai 14 10:59 config
-rw-r--r--   1 foreman foreman  383 Feb 25 17:09 config.ru
lrwxrwxrwx   1 foreman foreman   19 Sep  8  2021 db -> /var/lib/foreman/db
drwxr-xr-x   3 foreman foreman 4,0K Mai 14 10:59 extras
-rwxr-xr-x   1 foreman foreman 1,3K Mär  8 20:57 foreman-plugin.mk
-rw-r--r--   1 foreman foreman 2,1K Mär  8 20:57 Gemfile
-rw-r--r--   1 foreman foreman 8,6K Mai 14 10:59 Gemfile.lock
drwxr-xr-x   7 foreman foreman 4,0K Mai 14 10:59 lib
drwxr-xr-x  20 foreman foreman 4,0K Mai 14 10:59 locale
lrwxrwxrwx   1 foreman foreman   16 Sep  8  2021 log -> /var/log/foreman
lrwxrwxrwx   1 foreman foreman   23 Sep  8  2021 public -> /var/lib/foreman/public
-rw-r--r--   1 foreman foreman  198 Feb 25 17:09 Rakefile
drwxr-xr-x   5 foreman foreman 4,0K Mai 14 10:59 script
drwx------   2 foreman foreman 4,0K Sep  8  2021 .ssh
lrwxrwxrwx   1 foreman foreman   18 Sep  8  2021 tmp -> /var/cache/foreman
drwxr-xr-x   5 foreman foreman 4,0K Sep 16  2021 vendor
-rw-r--r--   1 foreman foreman    6 Feb 25 17:09 VERSION

thank you all

The errors which I found in the production.log:

2022-05-14T11:50:02 [I|app|39e73a97] Backtrace for 'Can't find entry point 'foreman_puppet:fills' in webpack manifest' error (ActionView::Template::Error): Can't find entry point 'foreman_puppet:fills' in webpack manifest
 39e73a97 | /usr/share/foreman/vendor/ruby/2.7.0/gems/webpack-rails-0.9.11/lib/webpack/rails/manifest.rb:39:in `asset_paths'
 39e73a97 | /usr/share/foreman/vendor/ruby/2.7.0/gems/webpack-rails-0.9.11/lib/webpack/rails/helper.rb:18:in `webpack_asset_paths'

 
 b2d6c9e0 | app/controllers/application_controller.rb:354:in `generic_exception'
 b2d6c9e0 | lib/foreman/middleware/telemetry.rb:10:in `call'
 b2d6c9e0 | lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 b2d6c9e0 | lib/foreman/middleware/logging_context_session.rb:22:in `call'

Is that error really cause below error:
grafik

Did you run foreman-installer after the upgrade?

@gvde no, should I run it without any arguments ?

Yes. You should run the installer after any update affecting any of the components of foreman just to be sure.

# foreman-maintain service stop
# foreman-installer

thank you for your answer @gvde
Ok ill stop it with systemctl because I dont see foreman-maintain:

# foreman-
foreman-debug          foreman-installer      foreman-prepare-realm  foreman-puma-status    foreman-rake           foreman-ruby           foreman-tail

I did that and this is the result:

x# foreman-installer
2022-05-14 14:29:16 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-05-14 14:31:34 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-05-14 14:31:34 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-05-14 14:33:47 [NOTICE] [configure] Starting system configuration.
2022-05-14 14:36:29 [NOTICE] [configure] 250 configuration steps out of 1331 steps complete.
2022-05-14 14:36:30 [NOTICE] [configure] 500 configuration steps out of 1333 steps complete.
2022-05-14 14:36:30 [NOTICE] [configure] 750 configuration steps out of 1441 steps complete.
2022-05-14 14:36:38 [NOTICE] [configure] 1000 configuration steps out of 1442 steps complete.
2022-05-14 14:42:43 [NOTICE] [configure] 1250 configuration steps out of 1442 steps complete.
2022-05-14 14:43:55 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
  Success!
  * Foreman is running at https://serverdm.servers.com
  * Foreman Proxy is running at https://serverdm.servers.com:8443

  The full log is at /var/log/foreman-installer/foreman.log

I still cannot login to the foreman

Seems foreman-maintain is still not supported on ubuntu. In that case you should reboot the server to make sure that all services have been stopped and started in the correct order.

Ok ill do that right now…

So I did that and the foreman service is running.
But the site cannot be reloaded… Im waiting and nothing happens…
Do you have maybe other idea :frowning:

I have now more problems my Puppet also doesnt work;

C:\Windows\system32>puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find xxxxx via exec: Execution of '/etc/puppetlabs/puppet/node.rb xxxxx' returned 1:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node xxxxx: Failed to find xxxxx via exec: Execution of '/etc/puppetlabs/puppet/node.rb xxxxxx' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

journalctk -xe

Mai 14 15:13:01 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit dynflow-sidekiq@orchestrator.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mai 14 15:13:01 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit dynflow-sidekiq@orchestrator.service has entered the 'failed' state with result 'exit-code'.
Mai 14 15:13:01 serverdm systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
-- Subject: A start job for unit dynflow-sidekiq@orchestrator.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit dynflow-sidekiq@orchestrator.service has finished with a failure.
--
-- The job identifier is 32078 and the job result is failed.
Mai 14 15:13:01 serverdm CRON[46394]: pam_unix(cron:session): session closed for user root
Mai 14 15:13:02 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:02 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
Mai 14 15:13:02 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Scheduled restart job, restart counter is at 126.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit dynflow-sidekiq@orchestrator.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Mai 14 15:13:02 serverdm systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
-- Subject: A stop job for unit dynflow-sidekiq@orchestrator.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit dynflow-sidekiq@orchestrator.service has finished.
--
-- The job identifier is 32327 and the job result is done.
Mai 14 15:13:02 serverdm systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
-- Subject: A start job for unit dynflow-sidekiq@orchestrator.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit dynflow-sidekiq@orchestrator.service has begun execution.
--
-- The job identifier is 32327.
Mai 14 15:13:03 serverdm foreman[46301]: => Booting Puma
Mai 14 15:13:03 serverdm foreman[46301]: => Rails 6.0.3.7 application starting in production
Mai 14 15:13:03 serverdm foreman[46301]: => Run `rails server --help` for more startup options
Mai 14 15:13:03 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1826: warning: The called method `has_and_belongs_to_many' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Mai 14 15:13:04 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:46: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/pg-1.3.3/lib/pg.rb:68: warning: The called method `connect' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1826: warning: The called method `has_and_belongs_to_many' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:46: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/pg-1.3.3/lib/pg.rb:68: warning: The called method `connect' is defined here
Mai 14 15:13:05 serverdm dynflow-sidekiq@orchestrator[46426]: 2022-05-14T13:13:05.127Z 46426 TID-x7y INFO: GitLab reliable fetch activated!
Mai 14 15:13:05 serverdm dynflow-sidekiq@orchestrator[46426]: 2022-05-14T13:13:05.128Z 46426 TID-xci INFO: Booting Sidekiq 5.2.10 with redis options {:id=>"Sidekiq-server-PID-46426", :url=>"redis://localhost:6379/0"}
Mai 14 15:13:06 serverdm qemu-ga[766]: info: guest-ping called
lines 1192-1274/1274 (END)

By the way I get this:

#su - postgres -c "psql foreman"
psql: error: FATAL:  database "foreman" does not exist


postgres-# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 |
 template0 | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres

That will be really very bad :frowning:

Is the database deleted ?

so I reboot the system again all is back, only the error is stille the same:

We’re sorry, but something went wrong.

If you are the application owner check the logs for more information

I solved the issue after upgrade from 3.0 to 3.1

Here is the link: Foreman :: Manual

Thanks for sharing this. We also ran into the same problem after upgrading from Ubuntu 18.04 → 20.04. Upgrading from 3.0 to 3.1 is what solved the problem. I didn’t want to move past 3.0 because I thought this issue was blocking us, but your post changed my mind.