Subscription-manager refresh error with katello 4.2.0.1

Support
1

Problem:
running subscription-manager refresh shows a database error:

# subscription-manager refresh
HTTP error (500 - Internal Server Error): Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433

Expected outcome:
redhat.repo file updated with repos listed in attached content view

Foreman and Proxy versions:
3.0

Foreman and Proxy plugin versions:
Katello 4.2.0.1

Distribution and version:
CentOS 8-Stream

Other relevant data:
This shows up in /var/log/foreman/production.log:

2021-10-23T21:52:45 [E|app|615f923a] RestClient::InternalServerError: Katello::Resources::Candlepin::CandlepinResource: 500 Internal Server Error {"displayMessage":"Runtime Error could not execute statement at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse:2,433","requestUuid":"f4e01985-496c-4782-95d9-f0ad2a5bfa2b"} (GET /candlepin/consumers/2b98f728-b447-4e8d-81d6-4b4f2a520a43/certificates/serials)

I had changed some of the repo listings in the CCV associated with a few of my servers, and was expecting subscription-manager refresh to adjust redhat.repo to match. I can edit redhat.repo as a workaround in the meantime, but either I’ve done something wrong or this is an issue worth reporting.

2

I looked in my postgresql logs and found this:

2021-10-24 07:27:22 CDT ERROR:  null value in column "collected" violates not-null constraint
2021-10-24 07:27:22 CDT DETAIL:  Failing row contains (3222890081946010683, 2021-10-24 07:27:22.87-05, 2021-10-24 07:27:22.87-05, null, 2049-11-30 18:00:00-06, f).
2021-10-24 07:27:22 CDT STATEMENT:  insert into cp_cert_serial (created, updated, expiration, revoked, id) values ($1, $2, $3, $4, $5)
2021-10-24 07:27:22 CDT ERROR:  current transaction is aborted, commands ignored until end of transaction block
2021-10-24 07:27:22 CDT STATEMENT:  
2021-10-24 07:27:26 CDT ERROR:  null value in column "collected" violates not-null constraint
2021-10-24 07:27:26 CDT DETAIL:  Failing row contains (4312166091278695580, 2021-10-24 07:27:26.714-05, 2021-10-24 07:27:26.714-05, null, 2049-11-30 18:00:00-06, f).
2021-10-24 07:27:26 CDT STATEMENT:  insert into cp_cert_serial (created, updated, expiration, revoked, id) values ($1, $2, $3, $4, $5)
2021-10-24 07:27:26 CDT ERROR:  current transaction is aborted, commands ignored until end of transaction block

Is it possible I’m missing a schema migration or something?

3

Running this allows subscription-manager to run without error (in the candlepin database):

alter table cp_cert_serial alter COLUMN collected SET default 'f';
4

I doubt that’s a good idea. If the database scheme isn’t right I’d say you didn’t run the installer after an update or some database migration failed.

I don’t even have the column in the database…

5

This server has only ever run the 4.2 release series, so I don’t think I would have had an upgrade in that sense.

I did try to run foreman-rake db:migrate before trying this. I looked into it further and found cpdb which did indeed perform a migration to drop that column (and it finished successfully), and my subscription-manager refresh's still run successfully, so I’m betting on a failed migration during a package upgrade.

6

Which version exactly did you start with? A nightly or RC?

7

Started with 4.2.0.rc1-1 installed from packages.

8

We’d appreciate seeing any errors from your production log or your foreman-installer logs. You would’ve seen the foreman-installer error out during installation, though, so maybe there wasn’t anything.

9

OK - I don’t have anything from production.log this far back. I think I just caught it with dnf upgrade:

# dnf history info 72
Updating Subscription Management repositories.
Transaction ID : 72
Begin time     : Thu 30 Sep 2021 06:09:14 AM CDT
Begin rpmdb    : 1267:90ca7369c31b9455472a4ab52b0ce395c73c7c22
End time       : Thu 30 Sep 2021 06:10:01 AM CDT (47 seconds)
End rpmdb      : 1267:ef9909f23074fd67e67c353970c2a8dca1c4922a
User           : Local Admin <localadmin>
Return-Code    : Success
Releasever     : 8
Command Line   : upgrade -y
Comment        : 
Packages Altered:
    Upgrade  candlepin-4.1.7-1.el8.noarch               @imladris_tfm_el8_x86_64_candlepin_42
    Upgraded candlepin-4.1.5-1.el8.noarch               @@System
    Upgrade  candlepin-selinux-4.1.7-1.el8.noarch       @imladris_tfm_el8_x86_64_candlepin_42
    Upgraded candlepin-selinux-4.1.5-1.el8.noarch       @@System
    Upgrade  katello-4.2.0.1-0.1.rc3.el8.noarch         @imladris_tfm_el8_x86_64_katello_42
    Upgraded katello-4.2.0.rc1-1.el8.noarch             @@System
    Upgrade  katello-common-4.2.0.1-0.1.rc3.el8.noarch  @imladris_tfm_el8_x86_64_katello_42
    Upgraded katello-common-4.2.0.rc1-1.el8.noarch      @@System
    Upgrade  katello-debug-4.2.0.1-0.1.rc3.el8.noarch   @imladris_tfm_el8_x86_64_katello_42
    Upgraded katello-debug-4.2.0.rc1-1.el8.noarch       @@System
    Upgrade  katello-repos-4.2.0.1-0.1.rc3.el8.noarch   @imladris_tfm_el8_x86_64_katello_42
    Upgraded katello-repos-4.2.0.rc1-1.el8.noarch       @@System
    Upgrade  rubygem-katello-4.2.0.1-0.1.rc3.el8.noarch @imladris_tfm_el8_x86_64_katello_42
    Upgraded rubygem-katello-4.2.0.rc1-1.el8.noarch     @@System

So maybe it didn’t even try to run the migration? It wasn’t a huge operational problem for me as I don’t often refresh subscriptions.

10

Ah, I didn’t realize it was the Candlepin DB that was the issue. I’ll ask around about this to see if it’s anything to worry about. Thanks for the info and debugging.

11

Apparently this is a somewhat-common issue that pops up. I’d just make sure to run foreman-installer after doing any package updates on the system.