Getting this error 'clean up your database' after upgrade to foreman 2.5/katello 4.1

I am hoping someone can point me to documentation for cleaning up databases on foreman. After upgrading from foreman 2.4/katello4.0 to foreman 2.5/katello 4.1 I cannot publish new content views and foreman-rake tasks generate a warning about permissions for /usr/share/foreman.

The production log shows ‘clean up your database’ entries when I attempt either tasks.

Hey @mrstrembly

Sorry to hear you are having problems.
Can you share some of the logs, please?

Attached log entries for foremain-rake cli action…
foreman-rake_katello_upgrade_check-production.log (7.3 KB)

Attached log entries for foreman UI content-view publish action…
publish_content-view-production.log (511.9 KB)

Thanks for your response @mcorr. Please see my update about 5 minutes ago. I attached the entries from the production.log file. If there other log files that would be helpful please let me know. I am very new to foreman so I am trying to ease myself into maintaining it. This system is actually meant to replace our older server so I am trying to figure it out as I go along.

Hi,

Will start to look at this and get back to you on the next steps.

1 Like

Hi @mrstrembly

Can I have you post the output of these commands for me:

# foreman-rake console
::Katello::Repository.find(180)
::Katello::Repository.find(470)

You can then just type exit to get out of that prompt. It looks like the repo with ID 180 appears to be missing. When the upgrade completed, did the installer finish ok or error out?

Here is the output of the requested foreman-rake commands…
indent preformatted text by 4 spaces
[root@puppet tmp]# foreman-rake console
/usr/share/foreman is not writable.
Bundler will use /tmp/bundler20210614-45985-npo9u945985' as your home directory temporarily. Loading production environment (Rails 6.0.3.7) irb(main):001:0> ::Katello::Repository.find(180) => #<Katello::Repository id: 180, pulp_id: "4-CentOS_7_Content_View-Library-ff2acf25-e67c-49fd...", library_instance_id: 23, content_view_version_id: 17, relative_path: "ENSCO/Library/CentOS_7_Content_View/custom/CentOS_...", environment_id: 2, saved_checksum_type: nil, distribution_version: nil, distribution_arch: nil, distribution_bootable: nil, distribution_family: nil, distribution_variant: nil, container_repository_name: nil, root_id: 23, remote_href: nil, publication_href: "/pulp/api/v3/publications/rpm/rpm/5b83dabf-5e47-42...", version_href: "/pulp/api/v3/repositories/rpm/rpm/62fd4af9-5a5d-4a..."> irb(main):002:0> ::Katello::Repository.find(470) Traceback (most recent call last): 2: from lib/tasks/console.rake:5:in block in <top (required)>’
1: from (irb):2
ActiveRecord::RecordNotFound (Couldn’t find Katello::Repository with ‘id’=470)
irb(main):003:0> exit
rake aborted!

Ok, that confirms what we are seeing in the traceback. For this Repo:

Splunk_Forwarder/splunkforwarder-8_1_2

In the UI if you goto that repo, does it show the correct package contents etc?

Interestingly, it is missing the Upstream URL. That is an easy fix. Is that enough to cause the content-view publish failure for views that do not include that repo?

Actually, that repo is a file repo instead of a URL sourced repo and it does have the correct .rpm file.

The Splunk_Forwarder/splunkforwarder-8_1_2 Sync status as ‘Not Synced’. Is Sync an expected state for a file based repo? The file was uploaded from a copy that is local to the foreman server.

I apologize, I mispoke. The Splunk_Forwarder/splunkforwarder-8_1_2 is actually in the content view that I attempted to publish for uploading to this issue. That said, the Splunk_Forwarder/splunkforwarder-8_1_2 repo does have the expected file listed. Is the file repo type not working in foreman 2.5?

I just attempted to publish a content view that does not include the Splunk_Forwarder/splunkforwarder-8_1_2 repo…and I get the same error I have been getting for all content views… (undefined method `id’ for nil:NilClass).

Using hammer repository info to inspect the Splunk_Forwarder/splunkforwarder-8_1_2 repo shows this…

[root@puppet ~]# hammer repository info --id 134
Id:                 134
Name:               splunkforwarder-8_1_2
Label:              splunkforwarder-8_1_2
Description:
Organization:       ENSCO
Red Hat Repository: no
Content Type:       file
Mirror on Sync:     no
Url:
Publish Via HTTP:   yes
Published At:       https://localhost/pulp/content/ENSCO/Library/custom/Splunk_Forwarder/splunkforwarder-8_1_2/
Relative Path:      ENSCO/Library/custom/Splunk_Forwarder/splunkforwarder-8_1_2
Download Policy:
HTTP Proxy:
    HTTP Proxy Policy: global_default_http_proxy
Product:
    Id:   15
    Name: Splunk Forwarder
GPG Key:

Sync:
    Status: Not Synced
Created:            2021/04/29 16:16:12
Updated:            2021/04/29 16:18:19
Content Counts:
    Files: 1

Something I notice about that repo is …I cannot find

please disregard the trailing comment ending with…‘I cannot find’. That was not intended.

Can I use ‘hammer create repository --id 180’ to forcibly create a repo for ID 180 …which does not exist?

In my initial configuration I had splunkforwarder-8_1_2 in a product that did not make sense so I removed it to reorganize the structure of the products in foreman and created the Splunk_Forwarder product to contain the various files we need.

You @cintrix84 asked me about the upgrade, it appeared to run successfully. I am attaching a sanitized version of the katello.log that shows the results. It is pretty lengthy because I had debug logging enabled (still do).

This was the tail end of the output that was displayed in when it finished.

Upgrade Step 2/4: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
=============================================
Upgrade Step 3/4: katello:upgrades:4.1:sync_noarch_content. =============================================
Upgrade Step 4/4: katello:upgrades:4.1:fix_invalid_pools. I, 2021-06-10T10:15:56.016520 #131878 INFO – : Corrected 0 invalid pools
I, 2021-06-10T10:15:56.016561 #131878 INFO – : Removed 0 orphaned pools
Success!

Foreman is running at https://puppet.ensco.win
To install an additional Foreman proxy on separate machine continue by running:
foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"

Foreman Proxy is running at https://puppet.ensco.win:9090
The full log is at /var/log/foreman-installer/katello.log

katello.log (2.7 MB)

I am still looking into this and checking with others if there is a way to recover the ID’s of the repos. The repo task during the upgrade should have caught it which I am surprised it didn’t.

1 Like

I have a feeling the change I made to the repos that resulted in the missing repo occurred after the upgrade. I don’t recall which came first (upgrade or repo delete) but I don’t think the upgrade would have finished as cleanly based on what I am seeing now. I did make a backup before the upgrade but I suspect that is not helpful at this point.

Can I have you run this and see if it completes all the way:

# foreman-rake katello:correct_repositories --trace