Problem with smart proxy after upgrade

Problem:
After upgrading katello pulp isn’t working:

pulp3 OK
katello_agent OK 0 Processed, 0 Failed
pulp FAIL 404 Not Found
pulp_auth FAIL Skipped pulp_auth check after failed pulp check

I can’t update the capsule servers when trying to do yum update:
https://fqdn/pulp/repos/GO/ADM/RHEL7/custom/EPEL7/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Expected outcome:
To be able to download updates.
Foreman and Proxy versions:

  • katello-4.2.0-0.1.master.el7.noarch
    foreman-proxy-2.6.0-0.1.develop.20210623124212gita1221fd.el7.noarch
    Foreman and Proxy plugin versions:
    |foreman-tasks |5.0.0|
    |foreman_remote_execution 4.6.0

Distribution and version:
RHEL 7.9
Other relevant data:

Could you please provide details regarding your upgrade, like the old foreman/katello versions, the basic upgrade steps, etc. to help us understand your case?

Did you run into any issue during the upgrade? Any errors in the log files?

Thanks.

from katello-3.18.2-1.el7.noarch
to
4.2.0-0.1.master.el7.noarch

upgrade steps:
https://theforeman.org/plugins/katello/nightly/upgrade/index.html
Ran into some issues with the foreman service but after a reboot it was upgraded successfully:


[root@no1-psoapp-1 ~]# foreman-installer
2021-06-29 13:03:31 [NOTICE] [root] Loading installer configuration. This will take some time.
2021-06-29 13:03:36 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2021-06-29 13:03:36 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2021-06-29 13:03:48 [NOTICE] [configure] Starting system configuration.
2021-06-29 13:04:00 [NOTICE] [configure] 250 configuration steps out of 2095 steps complete.
2021-06-29 13:04:21 [NOTICE] [configure] 500 configuration steps out of 2095 steps complete.
2021-06-29 13:04:22 [NOTICE] [configure] 750 configuration steps out of 2098 steps complete.
2021-06-29 13:04:24 [NOTICE] [configure] 1000 configuration steps out of 2102 steps complete.
2021-06-29 13:04:25 [NOTICE] [configure] 1250 configuration steps out of 2106 steps complete.
2021-06-29 13:04:44 [NOTICE] [configure] 1500 configuration steps out of 2107 steps complete.
2021-06-29 13:04:45 [NOTICE] [configure] 1750 configuration steps out of 2107 steps complete.
2021-06-29 13:04:52 [NOTICE] [configure] 2000 configuration steps out of 2107 steps complete.
2021-06-29 13:04:57 [NOTICE] [configure] System configuration has finished.
Executing: foreman-rake upgrade:run
`/usr/share/foreman` is not writable.
Bundler will use `/tmp/bundler20210629-4901-hsbfj14901' as your home directory temporarily.
API controllers newer than Apipie cache! Run apipie:cache rake task to regenerate cache.
=============================================
Upgrade Step 1/3: katello:correct_repositories. This may take a long while.
Processing Repository 1/1185: Red Hat Enterprise Linux 7 Server - Extras RPMs x86_64 (1)
Failed upgrade task: katello:correct_repositories, see logs for more information.
=============================================
Upgrade Step 2/3: katello:correct_puppet_environments. This may take a long while.
Failed upgrade task: katello:correct_puppet_environments, see logs for more information.
=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
0 orphaned consumer id(s) found in candlepin.
Candlepin orphaned consumers: []
  Success!
  * Foreman is running at https://fqdn
  * To install an additional Foreman proxy on separate machine continue by running:
      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  * Foreman Proxy is running at https://fqdn:9090

  The full log is at /var/log/foreman-installer/katello.log

Services:

[root@no1-psoapp-1 ~]# hammer status
Version: 2.6.0-develop
API Version: v2
Database:
Status: ok
Server Response: Duration: 0ms
Plugins:

  1. Name: foreman-tasks
    Version: 5.0.0
  2. Name: foreman_remote_execution
    Version: 4.6.0
  3. Name: katello
    Version: 4.2.0.pre.master
    Smart Proxies:
  4. Name: capsule
    Version: 2.6.0-develop
    Status: ok
    Features:
    1. Name: pulpnode
      Version: 2.1.0
    2. Name: dynflow
      Version: 0.3.0
    3. Name: ssh
      Version: 0.3.1
    4. Name: templates
      Version: 2.6.0
    5. Name: tftp
      Version: 2.6.0
    6. Name: puppetca
      Version: 2.6.0
    7. Name: puppet
      Version: 2.6.0
    8. Name: logs
      Version: 2.6.0
    9. Name: httpboot
      Version: 2.6.0
      10)Name: registration
      Version: 2.6.0
  5. Name: name
    Version: 2.5.0
    Status: ok
    Features:
    1. Name: pulpnode
      Version: 2.1.0
    2. Name: templates
      Version: 2.5.0
    3. Name: tftp
      Version: 2.5.0
    4. Name: puppetca
      Version: 2.5.0
    5. Name: puppet
      Version: 2.5.0
    6. Name: logs
      Version: 2.5.0
    7. Name: httpboot
      Version: 2.5.0
    8. Name: registration
      Version: 2.5.0
  6. Name: katello
    Version: 2.6.0-develop
    Status: ok
    Features:
    1. Name: pulpcore
      Version: 3.1.0
    2. Name: dynflow
      Version: 0.5.2
    3. Name: ssh
      Version: 0.4.0
    4. Name: templates
      Version: 2.6.0
    5. Name: tftp
      Version: 2.6.0
    6. Name: puppetca
      Version: 2.6.0
    7. Name: puppet
      Version: 2.6.0
    8. Name: logs
      Version: 2.6.0
    9. Name: registration
      Version: 2.6.0
      Compute Resources:

katello_agent:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 14ms
candlepin_auth:
Status: ok
Server Response: Duration: 14ms
candlepin_events:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 5 Processed, 0 Failed
Server Response: Duration: 0ms
pulp3:
Status: ok
Server Response: Duration: 34ms
foreman_tasks:
Status: ok
Server Response: Duration: 2ms

I can’t find anything in the logs.

You might have upgraded to unstable nightly release version by following the nightly documentation.
Please refer to this post for upgrade procedure. Thanks.

Yes, I can see that, is it possible to downgrade or reinstall?

Sadly we don’t support downgrade.
Reinstall would be a better option if there is no data need to be preserved. Otherwise I hope there is a snapshot before your migration so you may revert back to.

There’s no way to backup the data and restore in a previous version of katello?

DB backup should be done before DB migration which would change DB schema.