Foreman manifest is failing

Problem:
I am trying to add a new manifest but having conflict with the old one since I removed the old manifest file from redhat portal
Expected outcome:
Foreman should upload the newly dowloaded manifest
Foreman and Proxy versions:
|2.0.3|
Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:
image
From the image attached, the Manifest_CAO_Development is still appearing after deleting manifest and removing cache. If we try to upload the new one, the result is stuck at 0% importing.

Any help would be appreciated

Hi @Siddhu1096. If you go to Monitor → Tasks, you should see that task near the top of the list (Manifest Import). If you click in to that task you can see more details about the individual steps on that task. And, by clicking directly on those individual tasks you can see even more specific information. Do you see any failed (red) tasks in the Plan or Run sections in the Manifest Import task details? Please share those details here so we can get an idea of what’s going on.

It also sounds like the first Manifest Delete task that you tried did not succeed. Definitely look into the details of that task as well.

@Jonathon_Turel This is what I could see from dynflow

Also attaching the execution history

Ok, now we’re getting somewhere. Would it be possible for you to share your /var/log/candlepin/candlepin.log with me? It will be too large to paste here, but perhaps something like Dropbox would work. And if you like you can message me privately with that log file.

1 Like

Yes I can do that. @Jonathon_Turel

Attaching it here. Not that big files. @Jonathon_Turel
error.log-2021-09-09.gz (418.4 KB)error.log-2021-09-12.gz (418.5 KB)error.log-2021-09-13.gz (207.5 KB)error.log-2021-09-14.gz (145.9 KB)

Thanks @Siddhu1096. The error seems to be in the connection between postgres and candlepin. I recommend this:

  • restart all services via foreman-maintain service restart
  • verify everything is OK via hammer ping
  • if everything is OK, then import the manifest once again

If it fails, let’s go from there!

1 Like

@Jonathon_Turel Thanks for the support. Here are the results. Attaching the results of all the checklists: -

My hunch is that since we unknowingly deleted the subscription manifest from redhat portal first rather than deleting it from the foreman server, may be it might be reaching out to the old url. So it must be some db error.

We support running Foreman+Katello in environments that are disconnected from the Internet so there shouldn’t be any issue with the scenario you described. The new manifest should get imported. Are you seeing the same problem in the new manifest import task? And can you upload the new /var/log/candlepin/error.log so I can confirm the error has not changed on the Candlepin side?

This is the error now I’m getting

Ah, right - you’re on a very old Katello. Try this:

  • Ensure the currently imported manifest is deleted (optional, but nice)
  • Log in to the Red Hat Customer Portal and download the latest manifest for your Katello
  • Import that new manifest

@Jonathon_Turel Tried everything. Imported new manifest, went inside postgres db and tried to change inside candlepin database. But that also failed. We are out of ideas now. Happy to have a call if needed. I’ve sent my email ID as direct message.

When did you last download your manifest from the Red Hat Customer Portal @Siddhu1096 ?

Couple of days back. @Jonathon_Turel

I hopped on a call with @Siddhu1096 and we found that the root cause went back to the deletion of the manifest on the Katello side. The task reported success but looking in Dynflow showed that deletion was actually stuck by some previously failed job. /var/log/candlepin/error.log showed that the system disk was full according to Artemis which by default considers full to be 90% usage. After adding <max-disk-usage>99</max-disk-usage> to the candlepin broker.xml and clearing the cp_async_jobs table, the manifest delete succeeded which unblocked import of the new manifest!

2 Likes

Thank you so much for the support

Take away is : Upgrade the version of katello to 4.1 to ease everything. :))

1 Like