Foreman We're sorry, but something went wrong

Support
#1

Hi all,

after upgrade Ubuntu from 18.04 to 20.04 we get this error:
foreman We’re sorry, but something went wrong.If you are the application owner check the logs for more information.

In the ~# less /var/log/foreman/production.log I could not find errors or so.

The status of foreman-proxy

~# systemctl status foreman-proxy
● foreman-proxy.service - Foreman Proxy
     Loaded: loaded (/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2022-05-14 10:21:16 CEST; 52min ago
   Main PID: 956 (ruby)
      Tasks: 3 (limit: 19088)
     Memory: 52.8M
     CGroup: /system.slice/foreman-proxy.service
             └─956 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Mai 14 10:21:12 ServerDM smart-proxy[956]: You should probably keep only one of them.
Mai 14 10:21:12 ServerDM smart-proxy[956]: Remove any duplicate entries and specify the gem only once.
Mai 14 10:21:12 ServerDM smart-proxy[956]: While it's not a problem now, it could cause errors if you change the version of one of them later.
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/regular.rb:22: warning: The called method `initialize' is defined here
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/pattern.rb:59: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/regexp_based.rb:17: warning: The called method `initialize' is defined here
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:43: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 10:21:15 ServerDM smart-proxy[956]: /usr/lib/ruby/vendor_ruby/mustermann/ast/compiler.rb:49: warning: The called method `pattern' is defined here
Mai 14 10:21:16 ServerDM systemd[1]: Started Foreman Proxy.

The status of the foreman:

# systemctl status foreman
● foreman.service - Foreman
     Loaded: loaded (/lib/systemd/system/foreman.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/foreman.service.d
             └─installer.conf
     Active: active (running) since Sat 2022-05-14 11:01:55 CEST; 8min ago
TriggeredBy: ● foreman.socket
       Docs: https://theforeman.org
   Main PID: 30915 (foreman-ruby)
      Tasks: 240 (limit: 19088)
     Memory: 3.2G
     CGroup: /system.slice/foreman.service
             ├─30915 puma 5.6.2 (unix:///run/foreman.sock) [foreman]
             ├─30954 puma: cluster worker 0: 30915 [foreman]
             ├─30956 puma: cluster worker 1: 30915 [foreman]
             ├─30959 puma: cluster worker 2: 30915 [foreman]
             ├─30963 puma: cluster worker 3: 30915 [foreman]
             ├─30968 puma: cluster worker 4: 30915 [foreman]
             ├─30974 puma: cluster worker 5: 30915 [foreman]
             ├─30980 puma: cluster worker 6: 30915 [foreman]
             ├─30985 puma: cluster worker 7: 30915 [foreman]
             ├─30989 puma: cluster worker 8: 30915 [foreman]
             ├─30994 puma: cluster worker 9: 30915 [foreman]
             ├─31000 puma: cluster worker 10: 30915 [foreman]
             ├─31004 puma: cluster worker 11: 30915 [foreman]
             └─31008 puma: cluster worker 12: 30915 [foreman]

Mai 14 11:08:52 ServerDM foreman[31008]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:08:52 ServerDM foreman[31008]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here
Mai 14 11:10:03 ServerDM foreman[30994]: /usr/share/foreman/app/models/concerns/foreman/sti.rb:26: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:03 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:43: warning: The called method `save' is defined here
Mai 14 11:10:05 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:05 ServerDM foreman[30994]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here
Mai 14 11:10:28 ServerDM foreman[30974]: /usr/share/foreman/app/models/concerns/foreman/sti.rb:26: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:28 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/suppressor.rb:43: warning: The called method `save' is defined here
Mai 14 11:10:30 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:21: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 11:10:30 ServerDM foreman[30974]: /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.0.3.7/lib/action_view/template.rb:130: warning: The called method `initialize_without_deface' is defined here

apt policy foreman
foreman:
  Installiert:           3.0.2-2
  Installationskandidat: 3.0.2-2
  Versionstabelle:
 *** 3.0.2-2 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
        100 /var/lib/dpkg/status
     3.0.2-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.1-2 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.1-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0~rc2-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages
     3.0.0~rc1-1 500
        500 http://deb.theforeman.org focal/3.0 amd64 Packages


:~# ls -alh /usr/share/foreman
total 84K
drwxr-xr-x  13 foreman foreman 4,0K Mai 14 10:59 .
drwxr-xr-x 167 root    root    4,0K Mai 14 10:17 ..
drwxr-xr-x  15 foreman foreman 4,0K Sep 16  2021 app
drwxr-xr-x   2 foreman foreman 4,0K Mai 14 10:59 bin
drwxr-xr-x   2 foreman foreman 4,0K Sep 16  2021 .bundle
drwxr-xr-x   2 foreman foreman 4,0K Mai 14 10:59 bundler.d
drwxr-xr-x   7 foreman foreman 4,0K Mai 14 10:59 config
-rw-r--r--   1 foreman foreman  383 Feb 25 17:09 config.ru
lrwxrwxrwx   1 foreman foreman   19 Sep  8  2021 db -> /var/lib/foreman/db
drwxr-xr-x   3 foreman foreman 4,0K Mai 14 10:59 extras
-rwxr-xr-x   1 foreman foreman 1,3K Mär  8 20:57 foreman-plugin.mk
-rw-r--r--   1 foreman foreman 2,1K Mär  8 20:57 Gemfile
-rw-r--r--   1 foreman foreman 8,6K Mai 14 10:59 Gemfile.lock
drwxr-xr-x   7 foreman foreman 4,0K Mai 14 10:59 lib
drwxr-xr-x  20 foreman foreman 4,0K Mai 14 10:59 locale
lrwxrwxrwx   1 foreman foreman   16 Sep  8  2021 log -> /var/log/foreman
lrwxrwxrwx   1 foreman foreman   23 Sep  8  2021 public -> /var/lib/foreman/public
-rw-r--r--   1 foreman foreman  198 Feb 25 17:09 Rakefile
drwxr-xr-x   5 foreman foreman 4,0K Mai 14 10:59 script
drwx------   2 foreman foreman 4,0K Sep  8  2021 .ssh
lrwxrwxrwx   1 foreman foreman   18 Sep  8  2021 tmp -> /var/cache/foreman
drwxr-xr-x   5 foreman foreman 4,0K Sep 16  2021 vendor
-rw-r--r--   1 foreman foreman    6 Feb 25 17:09 VERSION

thank you all

#2

The errors which I found in the production.log:

2022-05-14T11:50:02 [I|app|39e73a97] Backtrace for 'Can't find entry point 'foreman_puppet:fills' in webpack manifest' error (ActionView::Template::Error): Can't find entry point 'foreman_puppet:fills' in webpack manifest
 39e73a97 | /usr/share/foreman/vendor/ruby/2.7.0/gems/webpack-rails-0.9.11/lib/webpack/rails/manifest.rb:39:in `asset_paths'
 39e73a97 | /usr/share/foreman/vendor/ruby/2.7.0/gems/webpack-rails-0.9.11/lib/webpack/rails/helper.rb:18:in `webpack_asset_paths'

 
 b2d6c9e0 | app/controllers/application_controller.rb:354:in `generic_exception'
 b2d6c9e0 | lib/foreman/middleware/telemetry.rb:10:in `call'
 b2d6c9e0 | lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
 b2d6c9e0 | lib/foreman/middleware/logging_context_session.rb:22:in `call'

Is that error really cause below error:
grafik

#3

Did you run foreman-installer after the upgrade?

1 Like
#4

@gvde no, should I run it without any arguments ?

#5

Yes. You should run the installer after any update affecting any of the components of foreman just to be sure.

# foreman-maintain service stop
# foreman-installer
2 Likes
#6

thank you for your answer @gvde
Ok ill stop it with systemctl because I dont see foreman-maintain:

# foreman-
foreman-debug          foreman-installer      foreman-prepare-realm  foreman-puma-status    foreman-rake           foreman-ruby           foreman-tail
#7

I did that and this is the result:

x# foreman-installer
2022-05-14 14:29:16 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-05-14 14:31:34 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-05-14 14:31:34 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2022-05-14 14:33:47 [NOTICE] [configure] Starting system configuration.
2022-05-14 14:36:29 [NOTICE] [configure] 250 configuration steps out of 1331 steps complete.
2022-05-14 14:36:30 [NOTICE] [configure] 500 configuration steps out of 1333 steps complete.
2022-05-14 14:36:30 [NOTICE] [configure] 750 configuration steps out of 1441 steps complete.
2022-05-14 14:36:38 [NOTICE] [configure] 1000 configuration steps out of 1442 steps complete.
2022-05-14 14:42:43 [NOTICE] [configure] 1250 configuration steps out of 1442 steps complete.
2022-05-14 14:43:55 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
  Success!
  * Foreman is running at https://serverdm.servers.com
  * Foreman Proxy is running at https://serverdm.servers.com:8443

  The full log is at /var/log/foreman-installer/foreman.log

I still cannot login to the foreman

#8

Seems foreman-maintain is still not supported on ubuntu. In that case you should reboot the server to make sure that all services have been stopped and started in the correct order.

1 Like
#9

Ok ill do that right now…

#10

So I did that and the foreman service is running.
But the site cannot be reloaded… Im waiting and nothing happens…
Do you have maybe other idea :frowning:

#11

I have now more problems my Puppet also doesnt work;

C:\Windows\system32>puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find xxxxx via exec: Execution of '/etc/puppetlabs/puppet/node.rb xxxxx' returned 1:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node xxxxx: Failed to find xxxxx via exec: Execution of '/etc/puppetlabs/puppet/node.rb xxxxxx' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

journalctk -xe

Mai 14 15:13:01 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- An ExecStart= process belonging to unit dynflow-sidekiq@orchestrator.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.
Mai 14 15:13:01 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit dynflow-sidekiq@orchestrator.service has entered the 'failed' state with result 'exit-code'.
Mai 14 15:13:01 serverdm systemd[1]: Failed to start Foreman jobs daemon - orchestrator on sidekiq.
-- Subject: A start job for unit dynflow-sidekiq@orchestrator.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit dynflow-sidekiq@orchestrator.service has finished with a failure.
--
-- The job identifier is 32078 and the job result is failed.
Mai 14 15:13:01 serverdm CRON[46394]: pam_unix(cron:session): session closed for user root
Mai 14 15:13:02 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:02 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
Mai 14 15:13:02 serverdm systemd[1]: dynflow-sidekiq@orchestrator.service: Scheduled restart job, restart counter is at 126.
-- Subject: Automatic restarting of a unit has been scheduled
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Automatic restarting of the unit dynflow-sidekiq@orchestrator.service has been scheduled, as the result for
-- the configured Restart= setting for the unit.
Mai 14 15:13:02 serverdm systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
-- Subject: A stop job for unit dynflow-sidekiq@orchestrator.service has finished
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A stop job for unit dynflow-sidekiq@orchestrator.service has finished.
--
-- The job identifier is 32327 and the job result is done.
Mai 14 15:13:02 serverdm systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
-- Subject: A start job for unit dynflow-sidekiq@orchestrator.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit dynflow-sidekiq@orchestrator.service has begun execution.
--
-- The job identifier is 32327.
Mai 14 15:13:03 serverdm foreman[46301]: => Booting Puma
Mai 14 15:13:03 serverdm foreman[46301]: => Rails 6.0.3.7 application starting in production
Mai 14 15:13:03 serverdm foreman[46301]: => Run `rails server --help` for more startup options
Mai 14 15:13:03 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.18/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1826: warning: The called method `has_and_belongs_to_many' is defined here
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:03 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Mai 14 15:13:04 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:46: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm foreman[46301]: /usr/share/foreman/vendor/ruby/2.7.0/gems/pg-1.3.3/lib/pg.rb:68: warning: The called method `connect' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1370: warning: The called method `has_many' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1826: warning: The called method `has_and_belongs_to_many' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/associations.rb:1657: warning: The called method `belongs_to' is defined here
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.0.3.7/lib/active_record/connection_adapters/postgresql_adapter.rb:46: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
Mai 14 15:13:04 serverdm dynflow-sidekiq@worker-1[46303]: /usr/share/foreman/vendor/ruby/2.7.0/gems/pg-1.3.3/lib/pg.rb:68: warning: The called method `connect' is defined here
Mai 14 15:13:05 serverdm dynflow-sidekiq@orchestrator[46426]: 2022-05-14T13:13:05.127Z 46426 TID-x7y INFO: GitLab reliable fetch activated!
Mai 14 15:13:05 serverdm dynflow-sidekiq@orchestrator[46426]: 2022-05-14T13:13:05.128Z 46426 TID-xci INFO: Booting Sidekiq 5.2.10 with redis options {:id=>"Sidekiq-server-PID-46426", :url=>"redis://localhost:6379/0"}
Mai 14 15:13:06 serverdm qemu-ga[766]: info: guest-ping called
lines 1192-1274/1274 (END)
#12

By the way I get this:

#su - postgres -c "psql foreman"
psql: error: FATAL:  database "foreman" does not exist


postgres-# \l
                                  List of databases
   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
 postgres  | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 |
 template0 | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres
 template1 | postgres | UTF8     | de_DE.UTF-8 | de_DE.UTF-8 | =c/postgres          +
           |          |          |             |             | postgres=CTc/postgres

That will be really very bad :frowning:

Is the database deleted ?

#13

so I reboot the system again all is back, only the error is stille the same:

We’re sorry, but something went wrong.

If you are the application owner check the logs for more information

#14

I solved the issue after upgrade from 3.0 to 3.1

1 Like