Possible to use Leapp for Oracle 7 -> Oracle 8 migration?

Support
#5

Right, I asked “which leapp version”, but what I really meant was “which leapp-upgrade-el7toel8 version”.

I guess the ol7-leapp repo you’re referring to is Oracle Linux 7 (x86_64) Leapp Upgrade Utilities | Oracle, Software. Hardware. Complete.? That contains new enough files.

You shouldn’t need to pass --enablerepo for the repositories defined in /etc/leapp/files/leapp_upgrade_repositories.repo (at least on upstream Leapp, those are automatically enabled).

Can you show me the contents of /var/log/leapp/leapp-report.txt and /var/log/leapp/leapp-preupgrade.log? Those files will contain a bit of private infromation (hostname, rpms installed), so feel free to mail them to evgeni at redhat dot com if you don’t want to post them publicly :slight_smile:

#6

Oops, sorry! leapp-upgrade-el7toel8 is version 0.17.0-1.0.3, also from the ol7-leapp repo

Yup, that’s the one.

Ah, ok. I saw in a log file a --disablerepo '*', with only the oracle repos enabled afterwards, so thought that might make a difference.

Emailed! Thank you so much for your help.

#7

Thanks a ton for the logs. They helped me to understand the problem!

So in EL8 several software is packaged as “modules” (so that the user can pick between different versions) and we’re using that to run Foreman with Ruby 2.7 (instead of the default 2.5) among other things. The error you pasted above indicates that it cannot find Ruby 2.7 in the created “target userspace” (aka: the thing it created that contains your new OL8).

The Leapp actor(s) we’ve written to aide the upgrade do have instructions to enable those modules:

And that code is also present in the RPM shipped by Oracle.

However, our actors do not implement the module enablement, but just instruct Leapp to do this for us.

The actual module handling was implemented over the course of multiple PRs (The Modularity integration by vinzenz · Pull Request #672 · oamg/leapp-repository · GitHub, modularity: Reset modules before upgrade by vinzenz · Pull Request #799 · oamg/leapp-repository · GitHub, Fix upgrade crashing due to attempting to enable non-existing modules by MichalHe · Pull Request #941 · oamg/leapp-repository · GitHub), which crucially have edited a file (leapp-repository/rhel_upgrade.py at master · oamg/leapp-repository · GitHub) which is used as a plugin to DNF to perform all necessary actions.

Why am I writing all this? Well, in the Oracle-shipped version of Leapp, there is no rhel_upgrade.py, but there is a oracle_upgrade.py (in /usr/share/leapp-repository/repositories/system_upgrade/common/files/oracle_upgrade.py) which doesn’t have any of the modularity-related changes that happened in Leapp upstream.

That means that no modules will be enabled during the upgrade and stuff breaks left and right.

I guess you gotta engage with Oracle support so that they unbreak their Leapp fork.

1 Like
#8

Wow, amazing! I grabbed the rhel_upgrade.py and merged the changes with oracle_upgrade.py. The preupgrade now passes!

I had one error during the upgrade, but that was easy to resolve, and I’m now running on Oracle Linux 8.7! Thank you so much for your help, would not have been able to do that without it.

#9

Oh wow, nice! I didn’t expect that merging those changes would be doable without many many headaches.

I just realized that the 3.3 upgrade docs are missing an important step to reindex the DB after the upgrade.

It’s documented here for 3.1: Upgrading and Updating Foreman

I’ll make sure to fix the docs, and you please run the reindex (otherwise you might get DB corruption at some point).

Upgrade Foreman 3.6 to 3.7 but Postgresql 10
#10

Hello, I upgraded recently from RHEL7 to RHEL8 without reindexing the DB, is it a big problem ?

Issue with using Leapp
#11

Do it now and we’ll find out :slight_smile:

There is a high chance you’ll see errors like this:

reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(<something>, PuppetFactName) is duplicated.

or

reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(<something>, Katello::RhsmFactName) is duplicated.

Luckily the facts table can be wiped and will be re-populated by clients once the check in again.

For the first error:

# cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'PuppetFactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'PuppetFactName').ids.count
EOF

For the second:

# cat << EOF | foreman-rake console
conf.echo = false
fact_name_ids = FactName.unscoped.where(:type => 'Katello::RhsmFactName').ids
pp fact_name_ids.count
FactValue.unscoped.where(:fact_name_id => fact_name_ids).delete_all
FactName.unscoped.where(:id => fact_name_ids).delete_all
pp FactName.unscoped.where(:type => 'Katello::RhsmFactName').ids.count
EOF

Then run the reindex command again.

Upgrade Foreman 3.6 to 3.7 but Postgresql 10
#12

Well, thank you I will try it. Should I stop the foreman service before ? Should it be run also on capsules ?

#13

No need to stop services and yes, also on the proxy.

#14

OK. It has been running more than 50 minutes:
reindexdb: reindexing database “candlepin”
and in processes I can see:
postgres: postgres candlepin [local] REINDEX waiting.

Hope it does something.

#15

huh, 50 minutes is a lot. try stopping candlepin for a moment: systemctl stop tomcat

#16

Yes, it helped, but I ended on another error:

reindexdb: error: reindexing of database “foreman” failed: ERROR: could not create unique index “katello_erratum_packages_eid_nvrea_n_f”
DETAIL: Key (erratum_id, nvrea, name, filename)=(3095639, gcc-objc-8.5.0-15.el8.s390x, gcc-objc, gcc-objc-8.5.0-15.el8.s390x.rpm) is duplicated.

#17

Okay, that’s the issue we were hoping to avoid :frowning:

I poked @iballou on IRC to look into this, stay tuned :slight_smile:

1 Like
#18

Hi @JendaVodka ,

Could you try opening up the Foreman console (foreman-rake console) and running the following:

::Katello::ErratumPackage.where.not(id: ::Katello::ErratumPackage.group(:nvrea).select("min(id)")).map {|ep| ep.erratum.repositories}.flatten.uniq.map{|r| r.relative_path}

That should return a list of repository paths. I’d be curious how many repositories are affected.

There are a couple ways to fix. The safest fix would be to destroy all errata records (and children) for your affected repositories and then reindex from Pulp. If too many large repositories are affected, then reindexing from Pulp may take too much time and we may need to fix the dupes manually.

Order of operations would be:

  1. Destroy affected errata and child records
  2. Reindex the DB
  3. Reindex/reimport the data from Pulp to the repositories (like a resync without some steps).
#19

Hi @iballou, the code you provided has been running more 15 minutes now with no output, is it correct ?

ad 1) I do not know how to do it :frowning:

Thank you very much for your attention

#20

Don’t worry about the fix that I mentioned yet, first it would be good to know the magnitude of the duplicates. If the query is really taking a long time to return your environment must be sizable and perhaps another query should be used.

If the above query seems to be taking way too long, perhaps some of these queries could give me a better idea:

::Katello::Erratum.count
::Katello::ErratumPackage.count
::Katello::Repository.count
::Katello::RootRepository.count
#21

irb(main):001:0> ::Katello::Erratum.count
=> 54983
irb(main):002:0> ::Katello::ErratumPackage.count
=> 652131
irb(main):003:0> ::Katello::Repository.count
=> 1854
irb(main):004:0> ::Katello::RootRepository.count
=> 175

#22

Busy day, but I’ve managed to start getting caught up. I started the db reindexing as you added, and had exactly the same problems as @JendaVodka. Stopping tomcat took care of the >4 hour run reindexing candlepin, then I got:

reindexdb: reindexing database "candlepin"
reindexdb: reindexing database "foreman"
reindexdb: error: reindexing of database "foreman" failed: ERROR:  could not create unique index "index_fact_names_on_name_and_type"
DETAIL:  Key (name, type)=(augeasversion, PuppetFactName) is duplicated.

I don’t know if the foreman-rake console command about errata is relevant, but it ran for a while and didn’t finish before I killed it.

What now?

If it’s helpful, here’s the size of my environment:

irb(main):001:0> ::Katello::Erratum.count
=> 18298
irb(main):002:0> ::Katello::ErratumPackage.count
=> 236017
irb(main):003:0> ::Katello::Repository.count
=> 77
irb(main):004:0> ::Katello::RootRepository.count
=> 12
#23

For the PuppetFacts ones, you can drop them as I’ve described in Possible to use Leapp for Oracle 7 -> Oracle 8 migration? - #11 by evgeni – that’s the easiest way forward.

#24

Ah! Sorry, missed that one. I used those commands and removed the facts, and got a clean, successful reindex! Thank you so much again for your help, it’s very appreciated!

1 Like