Reindexing database fails

Support
1

Problem:
I need to migrate my installation from Rocky 8 to Rocky 9. in-place is not a feasible option in this case.

I’m getting errors on duplicate keys for errata packages:

reindexdb: reindexing database "candlepin"
reindexdb: reindexing database "foreman"
reindexdb: error: processing of database "foreman" failed: ERROR:  could not create unique index "katello_erratum_packages_eid_nvrea_n_f"
DETAIL:  Key (erratum_id, nvrea, name, filename)=(51779, gcc-objc-4.8.2-16.2.el7_0.x86_64, gcc-objc, gcc-objc-4.8.2-16.2.el7_0.x86_64.rpm) is duplicated.
CONTEXT:  parallel worker

I have deleted all EL7 repositories, content-views and products.

On deleting the Products i also got another error, when i force completed that the product still disappeared from the list, but its stuck in limbo somewhere as when i run “foreman-rake katello:clean_candlepin_orphaned_products” it returns an error (same as task showed).

[root@foreman ~]# foreman-rake katello:clean_candlepin_orphaned_products
Cleaning Candlepin orphaned custom products for organization my_org
Deleting Candlepin orphaned custom product 847803058025 (and its associated pools and subscriptions)
rake aborted!
ForemanTasks::TaskError: Task fd143ecd-e065-4e9d-b3a7-4b924ba8a5f8: Katello::Errors::CandlepinError: Product "847803058025" cannot be deleted while referenced by one or more subscriptions
/usr/share/gems/gems/foreman-tasks-9.2.3/lib/foreman_tasks.rb:56:in `block in sync_task'
/usr/share/gems/gems/foreman-tasks-9.2.3/lib/foreman_tasks.rb:55:in `tap'
/usr/share/gems/gems/foreman-tasks-9.2.3/lib/foreman_tasks.rb:55:in `sync_task'
/usr/share/gems/gems/katello-4.14.0/lib/katello/tasks/clean_candlepin_orphaned_products.rake:26:in `block (4 levels) in <top (required)>'
/usr/share/gems/gems/katello-4.14.0/lib/katello/tasks/clean_candlepin_orphaned_products.rake:16:in `each'
/usr/share/gems/gems/katello-4.14.0/lib/katello/tasks/clean_candlepin_orphaned_products.rake:16:in `block (3 levels) in <top (required)>'
/usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/activerecord-6.1.7.8/lib/active_record/relation/delegation.rb:88:in `each'
/usr/share/gems/gems/katello-4.14.0/lib/katello/tasks/clean_candlepin_orphaned_products.rake:7:in `block (2 levels) in <top (required)>'
/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
Tasks: TOP => katello:clean_candlepin_orphaned_products
(See full trace by running task with --trace)

Expected outcome:
Reindex works

Foreman and Proxy versions:
Katello 4.14
foreman 3.12

Foreman and Proxy plugin versions:

Distribution and version:
Rocky 8

Other relevant data:

This might be related to another issue i’m having with removing a broken Manifest

I can start the manifest removal job, but it hangs at 0% forever and is only stoppable with warning status be restarting foreman (or more likely the postgres) through “foreman-maintain service restart”

2

When and how exactly are you performing the reindex?

If you can’t do an in-place upgrade, you can take a backup on EL8 and restore it on EL9. This will take care of re-indexing as well.

3

The when doesnt matter, i’ve tried with most services offline, everything online, while nothing is happening on the system, always the same error.

The how:
runuser -u postgres – reindexdb -a

There are 1.5 things wrong currently with backup and restore to a new instance.

When doing a full backup with this command:
foreman-maintain backup offline --assumeyes --preserve-directory /mnt/backup/

There is no pgsql_data.tar.gz created. I’ve semi-“solved” it by taking the DB offline, tar-ing it mysel with
tar --selinux -czvf /mnt/backup/pgsql_data.tar.gz /var/lib/pgsql/data/
that seemed to have worked.

But the bigger issue is that when the restore is reindexing the DB on destination host, it hits the exact same issue with reindexing, so i figured i could fix it on my old system before retrying the move.

4

It is on the EL9 system after you have copied the data there?
Did you try a re-index on EL8?

Correct, we switched to making DB dumps (so you should see foreman.dump, pulpcore.dump and candlepin.dump).

A nice side effect of dumps: they don’t need reindexing :slight_smile:

5

There is perhaps another option if its somehow possible.

Since this system has been around for a while, starting its life as OL7, migrated to CentOS 7, Leapped to Rocky8 and gone through so, so many version upgrades of Foreman/Katello

Can i somehow export all the hostgroup and puppet env/class + per host parameter changes and import that to a new instance?
if that is possible i can simply set up a brand new instance and import those settings for my ~450 hosts and then redo all the content stuff from scratch.

6

You may have switched to dumps, but did you remember to tell the restore task that? for me, at least, it says its required and stops.

I can try it again later today and see what happens if i completely skip the pgsql_data tar.

7

As a matter of fact we did.

Which version of rubygem-foreman_maintain do you have?

8 
[root@new-foreman ~]# cat /etc/system-release
Rocky Linux release 9.4 (Blue Onyx)
[root@new-foreman ~]# dnf repolist
repo id                                                                                                  repo name
appstream                                                                                                Rocky Linux 9 - AppStream
baseos                                                                                                   Rocky Linux 9 - BaseOS
candlepin                                                                                                Candlepin: an open source entitlement management system.
extras                                                                                                   Rocky Linux 9 - Extras
foreman                                                                                                  Foreman 3.12
foreman-plugins                                                                                          Foreman plugins 3.12
katello                                                                                                  Katello 4.14
pulpcore                                                                                                 pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
puppet7                                                                                                  Puppet 7 Repository el 9 - x86_64
[root@new-foreman ~]# rpm -qa | grep foreman_maintain
rubygem-foreman_maintain-1.7.4-1.el9.noarch
[root@new-foreman ~]#
9

That’s new enough (all changes for the switch landed in 1.6.13).

Would be interested in the exact error you’re getting when trying to restore a dump-only backup.

And no, there is no easy way to export groups/envs/etc and import them into a new system.

10

I’ll do a new backup and start the restore and let you know how it goes and we can debug the new system.

1 Like
11

well, i ran the backup/restore again - and by some magic it worked this time…

Going to have a poke around and see if everything is working, then finish the migration if it is.

Thanks for being an amazing rubber ducky and holding my hand, i guess hehe

Have a Fantastic Weekend!

1 Like
12

Glad it worked!