'/usr/sbin/foreman-rake db:migrate' returned 1 instead of one of [0]

Support
43

I have snapsots of all 4 VMs (foreman, postgres, memcash and smart-proxy) VMware VMs.
That was after initial install of 3.0.3 foremen, and I can go back to that and start again.

I run this command on db VM.

echo “UPDATE settings SET category=‘Setting’ WHERE category=‘Setting::General’;” | sudo -u postgres psql foreman
could not change directory to “/root”: Permission denied
UPDATE 17

44

Okay, cool. Does foreman-rake db:seed now complete?

45

Yes, it did.

 /usr/sbin/foreman-rake db:seed
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.

Thanks

46

Awesome. Happy upgrading!

47

I really appropriate your help.
Let me get this correct.
The solution is running this command in the DB node, correct?

echo "UPDATE settings SET category='Setting' WHERE category='Setting::General';" | sudo -u postgres psql foreman

I am doing this as Sandpit work before upgrading our 3.3 Production Foreman environment to latest version. I am building the practice SP environment start form scratch as same as production and upgrade to 3.3. and then applying the 3.4,3.5,…3.8. Ad I had the similar error at the start, I will redo whole 3.1 upgrade with above postgres db change.

BTW: I ran the foreman-installer, the last step again.
This time it has different error. which is I think due the foreman is down at the moment.

Example:
 foreman-installer
2023-10-23 17:49:53 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-23 17:49:57 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-23 17:49:57 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-23 17:50:06 [NOTICE] [configure] Starting system configuration.
2023-10-23 17:50:16 [NOTICE] [configure] 250 configuration steps out of 1634 steps complete.
2023-10-23 17:50:19 [NOTICE] [configure] 500 configuration steps out of 1634 steps complete.
2023-10-23 17:50:21 [NOTICE] [configure] 750 configuration steps out of 1636 steps complete.
2023-10-23 17:50:27 [NOTICE] [configure] 1000 configuration steps out of 1643 steps complete.
2023-10-23 17:50:27 [NOTICE] [configure] 1250 configuration steps out of 1644 steps complete.
2023-10-23 17:54:13 [NOTICE] [configure] 1500 configuration steps out of 1644 steps complete.
2023-10-23 17:55:01 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-23 17:55:03 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-23 17:55:03 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-23 17:55:06 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log
48

It is late in the evening here; I will continue tomorrow.
Thanks again for your support.

49

Well, ideally, you wouldn’t have to run this, as the migration I linked previously should have done exactly this statement for you.
For some reason, it did not and you had to manually poke the database.

This is most probably because the foreman service wasn’t restarted since we fixed the DB. Try a systemctl restart foreman and re-run the installer. If it clears up, you’re good.

50

HI evgeni,
I restarted the foreman with ‘systemctl restart foreman’ as you suggested and ran the ‘foreman-installer’ again. This is the result.

foreman-installer
2023-10-24 06:43:44 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-24 06:43:47 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-24 06:43:47 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-24 06:43:55 [NOTICE] [configure] Starting system configuration.
2023-10-24 06:44:04 [NOTICE] [configure] 250 configuration steps out of 1634 steps complete.
2023-10-24 06:44:07 [NOTICE] [configure] 500 configuration steps out of 1634 steps complete.
2023-10-24 06:44:09 [NOTICE] [configure] 750 configuration steps out of 1636 steps complete.
2023-10-24 06:44:10 [NOTICE] [configure] 1000 configuration steps out of 1643 steps complete.
2023-10-24 06:44:10 [NOTICE] [configure] 1250 configuration steps out of 1644 steps complete.
2023-10-24 06:45:17 [NOTICE] [configure] 1500 configuration steps out of 1644 steps complete.
2023-10-24 06:45:26 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-24 06:45:28 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-24 06:45:28 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sp-flag21.bbs.bunnings.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-flag21.bbs.bunnings.com.au/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-flag21.bbs.bunnings.com.au for detailed information
2023-10-24 06:45:30 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

These are the errors in “/var/log/foreman/production.log”

2023-10-24T06:45:28 [I|app|6cb445f7]   Rendering api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout
2023-10-24T06:45:28 [I|app|6cb445f7]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 3.5ms | Allocations: 5348)
2023-10-24T06:45:28 [I|app|6cb445f7] Completed 500 Internal Server Error in 17ms (Views: 6.5ms | ActiveRecord: 2.8ms | Allocations: 15876)
51

This is sorted now.
Thank you very much for your support.

52

I managed to upgrade to 3.3
Now trying to upgrade to 3.4, following instruction in this link: Upgrading and Updating Foreman (theforeman.org)

Doing tis step:

foreman-installer

Get this errors

 foreman-installer
2023-10-24 17:20:36 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-24 17:20:38 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-24 17:20:38 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-24 17:20:43 [NOTICE] [configure] Starting system configuration.
2023-10-24 17:20:53 [NOTICE] [configure] 250 configuration steps out of 1570 steps complete.
2023-10-24 17:20:57 [NOTICE] [configure] 500 configuration steps out of 1571 steps complete.
2023-10-24 17:20:59 [NOTICE] [configure] 750 configuration steps out of 1575 steps complete.
2023-10-24 17:20:59 [NOTICE] [configure] 1000 configuration steps out of 1580 steps complete.
2023-10-24 17:21:26 [NOTICE] [configure] 1250 configuration steps out of 1580 steps complete.
2023-10-24 17:21:32 [ERROR ] [configure] /Stage[main]/Foreman::Register/Foreman_host[foreman-sp-abc.abc.com.au]: Could not evaluate: Error making GET request to Foreman at https://abc.abc.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-abc.abc.com.au for detailed information
2023-10-24 17:21:32 [NOTICE] [configure] 1500 configuration steps out of 1580 steps complete.
2023-10-24 17:21:32 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_host[foreman-proxy-sp-abc.abc.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-abc.abc.com.au/api/v2/hosts: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-abc.abc.com.au.au for detailed information
2023-10-24 17:21:32 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sp-abc.abc.com.au]: Could not evaluate: Error making GET request to Foreman at https://sp-abc.abc.com.au/api/v2/smart_proxies: Response: 500 Internal Server Error: Check /var/log/foreman/production.log on sp-abc.abc.com.au for detailed information
2023-10-24 17:21:34 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/katello.log

Started the foreman back using ‘systemctl restart foreman’, but still the same.

53

Can you post the relevant part of /var/log/foreman/production.log where it shows the backtrace of the 500 error?

54

Here are the lines (2 lines before and 5 lines after the error)

 sudo grep -B2 -A5 "Completed 500 Internal Server Error" /var/log/foreman/production.log
2023-10-27T11:34:11 [I|app|cca6c3a0]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 1.0ms | Allocations: 263)
2023-10-27T11:34:11 [I|app|cca6c3a0]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.8ms | Allocations: 499)
2023-10-27T11:34:11 [I|app|cca6c3a0] Completed 500 Internal Server Error in 29ms (Views: 9.2ms | ActiveRecord: 1.3ms | Allocations: 25937)
2023-10-27T11:34:11 [F|app|cca6c3a0]
 cca6c3a0 | NoMethodError (undefined method `dup_value!' for #<String:0x00005648a9b95fd0>):
 cca6c3a0 |
 cca6c3a0 | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 cca6c3a0 | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:34:35 [I|app|3372c18c]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 1.1ms | Allocations: 263)
2023-10-27T11:34:35 [I|app|3372c18c]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.1ms | Allocations: 499)
2023-10-27T11:34:35 [I|app|3372c18c] Completed 500 Internal Server Error in 51ms (Views: 6.7ms | ActiveRecord: 20.0ms | Allocations: 22938)
2023-10-27T11:34:36 [F|app|3372c18c]
 3372c18c | NoMethodError (undefined method `dup_value!' for #<String:0x00005648ace352d8>):
 3372c18c |
 3372c18c | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 3372c18c | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:34:36 [I|app|1cfc8a40]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 10.0ms | Allocations: 5370)
2023-10-27T11:34:36 [I|app|1cfc8a40]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 15.1ms | Allocations: 10722)
2023-10-27T11:34:36 [I|app|1cfc8a40] Completed 500 Internal Server Error in 41ms (Views: 16.2ms | ActiveRecord: 10.6ms | Allocations: 13073)
2023-10-27T11:34:36 [F|app|1cfc8a40]
 1cfc8a40 | NoMethodError (undefined method `dup_value!' for #<String:0x00005648a69de4b0>):
 1cfc8a40 |
 1cfc8a40 | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 1cfc8a40 | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:34:36 [I|app|18cc964b]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 55.3ms | Allocations: 5371)
2023-10-27T11:34:36 [I|app|18cc964b]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 59.3ms | Allocations: 10723)
2023-10-27T11:34:36 [I|app|18cc964b] Completed 500 Internal Server Error in 77ms (Views: 59.8ms | ActiveRecord: 6.9ms | Allocations: 12616)
2023-10-27T11:34:36 [F|app|18cc964b]
 18cc964b | NoMethodError (undefined method `dup_value!' for #<String:0x00005648b4386640>):
 18cc964b |
 18cc964b | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 18cc964b | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:42:01 [I|app|5a060efe]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 1.1ms | Allocations: 263)
2023-10-27T11:42:01 [I|app|5a060efe]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.0ms | Allocations: 499)
2023-10-27T11:42:01 [I|app|5a060efe] Completed 500 Internal Server Error in 14ms (Views: 6.9ms | ActiveRecord: 0.4ms | Allocations: 3478)
2023-10-27T11:42:01 [F|app|5a060efe]
 5a060efe | NoMethodError (undefined method `dup_value!' for #<String:0x00005576232cea68>):
 5a060efe |
 5a060efe | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 5a060efe | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:42:02 [I|app|3caae935]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 0.8ms | Allocations: 263)
2023-10-27T11:42:02 [I|app|3caae935]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.4ms | Allocations: 499)
2023-10-27T11:42:02 [I|app|3caae935] Completed 500 Internal Server Error in 72ms (Views: 4.9ms | ActiveRecord: 1.7ms | Allocations: 22936)
2023-10-27T11:42:02 [F|app|3caae935]
 3caae935 | NoMethodError (undefined method `dup_value!' for #<String:0x0000557633ae9c60>):
 3caae935 |
 3caae935 | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 3caae935 | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
--
2023-10-27T11:42:02 [I|app|27cdc293]   Rendered api/v2/errors/custom_error.json.rabl within api/v2/layouts/error_layout (Duration: 4.6ms | Allocations: 5371)
2023-10-27T11:42:02 [I|app|27cdc293]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 8.0ms | Allocations: 10724)
2023-10-27T11:42:02 [I|app|27cdc293] Completed 500 Internal Server Error in 13ms (Views: 8.8ms | ActiveRecord: 0.3ms | Allocations: 13034)
2023-10-27T11:42:02 [F|app|27cdc293]
 27cdc293 | NoMethodError (undefined method `dup_value!' for #<String:0x0000557633873e90>):
 27cdc293 |
 27cdc293 | config/initializers/apipie.rb:64:in `block (3 levels) in <top (required)>'
 27cdc293 | config/initializers/apipie.rb:73:in `block (4 levels) in <top (required)>'
55

undefined method dup_value!'is usually what we've seen whenforeman_memcache` was still active.
Sure it’s gone in your env?

The installer might have installed it back, if it was installed using the installer before and you only removed the RPM?

56

How do I check it?
I ran this:

 sudo yum remove rubygem-foreman_memcache

and still ‘sudo foreman-installer’ gives the same errors.

57

can you post the output of grep memcache /etc/foreman-installer/scenarios.d/*-answers.yaml?

58 
 grep memcache /etc/foreman-installer/scenarios.d/*-answers.yaml
/etc/foreman-installer/scenarios.d/foreman-answers.yaml:foreman::plugin::memcache: false
/etc/foreman-installer/scenarios.d/katello-answers.yaml:foreman::plugin::memcache:
59

Okay, try this:

dnf remove rubygem-foreman_memcache
foreman-installer --no-enable-foreman-plugin-memcache
60

Ran ‘dnf remove rubygem-foreman_memcache’
Still ‘foreman-installer --no-enable-foreman-plugin-memcache’ gives the gives the ‘Response: 500 Internal Server Error:’ errors:

61

after you ran the two, is the rubygem-foreman_memcache package installed (again)?

does grep memcache /etc/foreman-installer/scenarios.d/katello-answers.yaml show foreman::plugin::memcache: false?

62

Ran ‘dnf remove rubygem-foreman_memcache’
Still ‘foreman-installer --no-enable-foreman-plugin-memcache’ gives the gives the ‘Response: 500 Internal Server Error:’ errors: