Candlepin probably has a problem with key reading/decrypting. Anyone have an idea how to get this working again
Expected outcome:
Foreman Katello ready to login with the existing Parameters Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Distribution and version:
Other relevant data:
foreman-maintain restore /var/katello-backup-2022-07-22-22-25-52/
Running Restore backup
================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files: [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup: [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup: [OK]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore:
WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed?, [y(yes), q(quit)] y
[OK]
--------------------------------------------------------------------------------
Setting file security:
\ Restoring SELinux context [OK]
--------------------------------------------------------------------------------
Restore configs from backup:
| Restoring configs [OK]
--------------------------------------------------------------------------------
Run installer reset:
\ Installer reset [FAIL]
Failed executing yes | foreman-installer -v --reset-data --disable-system-checks , exit status 1:
2022-07-25 09:20:12 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-07-25 09:20:17 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-07-25 09:20:17 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
2022-07-25 09:20:24 [NOTICE] [pre] Dropping foreman database!
2022-07-25 09:20:24 [ERROR ] [root] runuser -l postgres -c 'dropdb foreman' failed! Check the output for error!
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.
The following steps ended up in failing state:
[restore-installer-reset]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="restore-installer-reset"
Running Rescue Restore backup
================================================================================
[root@lxlabs0131 ~]# runuser -l postgres -c 'createdb foreman'
[root@lxlabs0131 ~]# foreman-maintain restore /var/katello-backup-2022-07-22-22-25-52/
Running Restore backup
================================================================================
Check if command is run as root user: [OK]
--------------------------------------------------------------------------------
Validate backup has appropriate files: [OK]
--------------------------------------------------------------------------------
Validate hostname is the same as backup: [OK]
--------------------------------------------------------------------------------
Validate network interfaces match the backup: [OK]
--------------------------------------------------------------------------------
Confirm dropping databases and running restore:
WARNING: This script will drop and restore your database.
Your existing installation will be replaced with the backup database.
Once this operation is complete there is no going back.
Do you want to proceed?, [y(yes), q(quit)] y
[OK]
--------------------------------------------------------------------------------
Setting file security:
/ Restoring SELinux context [OK]
--------------------------------------------------------------------------------
Restore configs from backup:
- Restoring configs [OK]
--------------------------------------------------------------------------------
Run installer reset:
- Installer reset [FAIL]
Failed executing yes | foreman-installer -v --reset-data --disable-system-checks , exit status 6:
2022-07-25 09:26:17 [NOTICE] [root] Loading installer configuration. This will take some time.
2022-07-25 09:26:21 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2022-07-25 09:26:21 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
Are you sure you want to continue? This will drop the databases, reset all configurations that you have made and bring all application data back to a fresh install. [y/n]
2022-07-25 09:26:26 [NOTICE] [pre] Dropping foreman database!
2022-07-25 09:26:26 [NOTICE] [pre] Dropping candlepin database!
2022-07-25 09:26:26 [NOTICE] [pre] Dropping pulpcore database!
2022-07-25 09:26:27 [WARN ] [pre] Pulpcore content directory not present at '/var/lib/pulp/docroot'
2022-07-25 09:26:27 [WARN ] [pre] Skipping system checks.
2022-07-25 09:26:27 [WARN ] [pre] Skipping system checks.
2022-07-25 09:26:38 [NOTICE] [configure] Starting system configuration.
2022-07-25 09:26:54 [NOTICE] [configure] 250 configuration steps out of 1364 steps complete.
2022-07-25 09:27:01 [NOTICE] [configure] 500 configuration steps out of 1366 steps complete.
2022-07-25 09:27:02 [ERROR ] [configure] Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-07-25 09:27:02 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-07-25 09:27:02 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-07-25 09:27:02 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-07-25 09:27:02 [ERROR ] [configure] ... 5 more
2022-07-25 09:27:02 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:artemis-client]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias artemis-client -file /etc/foreman/client_cert.pem -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-07-25 09:27:02 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-07-25 09:27:02 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-07-25 09:27:02 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-07-25 09:27:02 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-07-25 09:27:02 [ERROR ] [configure] ... 5 more
2022-07-25 09:27:02 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Certs::Keypair[katello-default-ca]/Private_key[/etc/candlepin/certs/candlepin-ca.key]: Could not evaluate: Execution of '/bin/openssl rsa -in /root/ssl-build/katello-default-ca.key -passin file:/etc/pki/katello/private/katello-default-ca.pwd -text' returned 1: unable to load Private Key
2022-07-25 09:27:02 [ERROR ] [configure] 140077840291648:error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt:crypto/evp/evp_enc.c:643:
2022-07-25 09:27:02 [ERROR ] [configure] 140077840291648:error:23077074:PKCS12 routines:PKCS12_pbe_crypt:pkcs12 cipherfinal error:crypto/pkcs12/p12_decr.c:63:
2022-07-25 09:27:02 [ERROR ] [configure] 140077840291648:error:2306A075:PKCS12 routines:PKCS12_item_decrypt_d2i:pkcs12 pbe crypt error:crypto/pkcs12/p12_decr.c:94:
2022-07-25 09:27:02 [ERROR ] [configure] 140077840291648:error:0907B00D:PEM routines:PEM_read_bio_PrivateKey:ASN1 lib:crypto/pem/pem_pkey.c:88:
2022-07-25 09:27:02 [ERROR ] [configure] Failed to add certificate to keystore: Execution of '/bin/keytool -importkeystore -noprompt -srckeystore /tmp/temp_keystore20220725-93372-7vb7k0 -srcstorepass:file /etc/pki/katello/keystore_password-file -destkeystore /etc/candlepin/certs/keystore -deststorepass:file /etc/pki/katello/keystore_password-file -srcalias tomcat -destalias tomcat -J-Dcom.redhat.fips=false' returned 1: Importing keystore /tmp/temp_keystore20220725-93372-7vb7k0 to /etc/candlepin/certs/keystore...
2022-07-25 09:27:02 [ERROR ] [configure] keytool error: java.io.IOException: keystore password was incorrect
2022-07-25 09:27:03 [ERROR ] [configure] Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias candlepin-ca -file /etc/candlepin/certs/candlepin-ca.crt -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-07-25 09:27:03 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-07-25 09:27:03 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-07-25 09:27:03 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-07-25 09:27:03 [ERROR ] [configure] ... 5 more
2022-07-25 09:27:03 [ERROR ] [configure] /Stage[main]/Certs::Candlepin/Truststore_certificate[/etc/candlepin/certs/truststore:candlepin-ca]/ensure: change from 'absent' to 'present' failed: Execution of '/bin/keytool -import -v -noprompt -storetype pkcs12 -keystore /etc/candlepin/certs/truststore -alias candlepin-ca -file /etc/candlepin/certs/candlepin-ca.crt -storepass:file /etc/pki/katello/truststore_password-file -J-Dcom.redhat.fips=false' returned 1: keytool error: java.io.IOException: keystore password was incorrect
2022-07-25 09:27:03 [ERROR ] [configure] java.io.IOException: keystore password was incorrect
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2079)
2022-07-25 09:27:03 [ERROR ] [configure] at java.security.KeyStore.load(KeyStore.java:1445)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.doCommands(Main.java:836)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.run(Main.java:377)
2022-07-25 09:27:03 [ERROR ] [configure] at sun.security.tools.keytool.Main.main(Main.java:370)
2022-07-25 09:27:03 [ERROR ] [configure] Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
2022-07-25 09:27:03 [ERROR ] [configure] ... 5 more
2022-07-25 09:27:08 [NOTICE] [configure] 750 configuration steps out of 1371 steps complete.
2022-07-25 09:27:09 [NOTICE] [configure] 1000 configuration steps out of 1380 steps complete.
2022-07-25 09:27:42 [NOTICE] [configure] 1250 configuration steps out of 1380 steps complete.
2022-07-25 09:29:07 [NOTICE] [configure] System configuration has finished.
There were errors detected during install.
Please address the errors and re-run the installer to ensure the system is properly configured.
Failing to do so is likely to result in broken functionality.
The full log is at /var/log/foreman-installer/katello.log
--------------------------------------------------------------------------------
Scenario [Restore backup] failed.
The following steps ended up in failing state:
[restore-installer-reset]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="restore-installer-reset"
Running Rescue Restore backup
================================================================================
Hi, any news on this? I came across the same issue when running a foreman-maintain restore
Foreman version 3.3.1 Katello 4.5.0-1. (Migrate from RHEL7.9 to RHEL8.6)
The key files from the backup will be overwritten with old passwords during the run (at least these three)
If I am overwriting these files with the right values I end up with a foreman-rake db:migrate error:
foreman-rake db:migrate --trace --verbose :(
** Invoke db:migrate (first_time)
** Invoke db:load_config (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:load_config
** Invoke plugin:refresh_migrations (first_time)
** Invoke environment ** Execute plugin:refresh_migrations
** Execute db:migrate
== 20220110223754 UpdateDisconnectedSettings: migrating ======================= rake aborted! StandardError: An error has occurred, this and all later migrations canceled:
The single-table inheritance mechanism failed to locate the subclass: 'Setting::Content'. This error is raised because the column 'category' is reserved for storing the class in case of in
heritance. Please rename this column if you didn't intend it to be used for storing the inheritance class or overwrite Setting.inheritance_column to use another column for that information
I am running into the same issue. Migrating from CentOS 7.9 to Rocky 8.
After it failed the first time I could see the keystore password files having a timestamp of “now” and are not identical to the ones from the old installation.
I’m re-running the restore right now with the suggested option --whitelist=“restore-installer-reset”. As expected, it skipped the “Run installer reset” stage and went strainht to “Stop applicable services” and now its restoring pulp data.
Gonna update this post when it has finished or failed.
Copy this directory /opt/puppetlabs/puppet/cache/foreman_cache_data from your old to your new foreman. Then the restore worked for me.
I tried to delete this directory, but it will be recreated with some random passwords, which did not work.
I took a slightly different approach, but yours is definitely preferable. I also managed to work around the issue and the F/K server plus proxy are back in business.
However, I do still believe this is something the devs should look into, if it hasn’t been solved already by a higher version of F/K.
The 20220110* would run BEFORE 20220419* but the code change was already done in foreman/katello. Therefore Setting.find(:content…) fails.
I’m currently on a broken system, because Setting with category Setting::Content exists, especially the Setting with the name “content_disconnected” and it tries to run the migration 20220110223754_update_disconnected_settings.rb but this is not possible.
