Status code: 403 for Error: Failed to download metadata for repo after upgrade to 3.11

Hello community

after upgrade from Foreman 3.10.0 and Katello 4.12.1 to Foreman 3.11 and Katello 4.13, I got an error message when a new client tries to download repolist or search for packages inside the rep.
Error: Status code: 403 for …
Error: Failed to download metadata for repo …
I followed this documentation Upgrading Foreman to 3.11

dnf search tmux
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 778 B/s | 73 B 00:00
Errors during downloading metadata for repository ‘rhel-9-for-x86_64-baseos-rpms’:
- Status code: 403 for https:///pulp/content/Orga/RHEL_ENV/rhel_92_cont_view/content/dist/rhel9/9.2/x86_64/baseos/os/repodata/repomd.xml (IP: 172.17.190.5)
Error: Failed to download metadata for repo ‘rhel-9-for-x86_64-baseos-rpms’: Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

can anyone help me please?

best regards
T. Reineck

1 Like

This is usually resolved by refreshing your manifest. In the web UI: Content > Subscriptions > Manage Manifest > Refresh

Hello,

I refreshed the manifest. But the error also appears.

best regards
Tob. Rein.

Hi,
I have got the same issue with my instance. This instance is also freshly upgraded from 3.10 (I cannot confirm that it worked with 3.10).

I have refreshed the manifest. I have no problems viewing the content in a browser with the debug certificate.
Trying to curl using the entitlement cert and key gives the following error:
“403: Requested path is not a subpath of a path in the client certificate.”

Does anyone have any ideas? Thank you!

Kind regards

For folks having this issue, can you please share the ‘authorized content URLs’ and ‘Content’ sections from rct cat-cert /etc/pki/entitlement/<number>.pem ?

It’s looking like the allowed URL in the entitlement cert might be misconfigured, at least on a reproducer we found.

Also, as a workaround, try subscribing your hosts to Default Organization View / Library.

1 Like

This is appearing to be a regression with Candlepin 4.4, but an investigation is ongoing. There are escape characters erroneously showing up in the entitlement certificates’ content URLs. Candlepin 4.3 should work fine with Katello 4.13 if people are able to do a reset to downgrade. You’d just need to change the candlepin repo definition to use the 4.3 branch. Regardless, we’ll make sure to get it fixed as quickly as possible.

1 Like

exactly same here, we updated today:

rpm -q foreman katello candlepin
  foreman-3.11.0-1.el8.noarch
  katello-4.13.0-1.el8.noarch
  candlepin-4.4.10-1.el8.noarch

only rew-rhel (registered after upgrade) systems are affected.
existing systems can pull content.

Also, as a workaround, try subscribing your hosts to Default Organization View / Library.

I can confirm this workaround succeeds, if i switch to Library, it starts pulling again.

If we can help with some testing and debugging, just shout

Thanks Chris

This makes sense, as the existing ones have certificate issues by the old (4.3) Candlepin, which doesn’t have that bug.

2 Likes

ahhh, that explains a lot!
I will try to downgrade candlepin to 4.3 and come back with the documentation how to do it.
Do you know if its safe from database perspective (db scheme)?
Thanks for answering. Chris

I would not expect it to be safe to downgrade, but I am also not an authoritative source for that :wink:

1 Like

Subscribing the hosts to Default Organization View / Library does also work on my instance.

2 Likes

It’s tracked in a private RH Jira issue.
I’ve asked if it can be made public, but have no idea what the Candlepin project policies are.

1 Like

but I am also not an authoritative source for that :wink:

you are, after doing more tests, i can confirm katello does not work with downgraded candlepin-4.3, so we have to restore or wait for fix.
thanks for the support!

1 Like

What most probably should work is: take a 4.12 Katello (with CP 4.3) and before upgrading it to 4.13 (but with 4.13 repos already present) re-configure the repos to point at CP 4.3 ones, so you never upgrade/migrate to 4.4.

1 Like

2297301 – Incorrectly url encoded 'Authorized Content Url' in SCA certificates is a public BZ you can watch :slight_smile:

3 Likes

candlepin 4.4.12 should fix that issue and is on the way to our repos as we speak (give it ~20 min).

I assume you’ll have to resubscribe the affected machines tho.

3 Likes

now I can’t get candlepin to start now after updating

Do you have any logs that show why it’s not starting?

the only logs i am seeing is the tomcat service start logs.

● tomcat.service - Apache Tomcat Web Application Container
   Loaded: loaded (/usr/lib/systemd/system/tomcat.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2024-07-25 20:32:30 EDT; 13h ago
 Main PID: 105387 (java)
    Tasks: 50 (limit: 821331)
   Memory: 968.3M
   CGroup: /system.slice/tomcat.service
           └─105387 /usr/lib/jvm/jre-17/bin/java -Xms1024m -Xmx8192m -Dcom.redhat.fips=false -Djava.security.auth.login.config=/usr/share/tomcat/conf/login.config -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar: -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Jul 25 20:32:30 parton server[105387]: 25-Jul-2024 20:32:30.857 INFO [main] org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL successfully initialized [OpenSSL 1.1.1k  FIPS 25 Mar 2021]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.090 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["https-openssl-nio-127.0.0.1-23443"]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.109 WARNING [main] org.apache.tomcat.util.net.SSLUtilBase.getEnabled Tomcat interprets the [ciphers] attribute in a manner consistent with the latest OpenSSL development branch. Some of the specified [ciphers] are not supported by the configured SSL engine for this connector (which may use JSSE or an older OpenSSL version) and have been skipped: [[TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256]]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.484 INFO [main] org.apache.tomcat.util.net.AbstractEndpoint.logCertificate Connector [https-openssl-nio-127.0.0.1-23443], TLS virtual host [_default_], certificate type [UNDEFINED] configured from keystore [/etc/candlepin/certs/keystore] using alias [tomcat] with trust store [/etc/candlepin/certs/truststore]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.493 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [815] milliseconds
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.524 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.525 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.87]
Jul 25 20:32:31 parton server[105387]: 25-Jul-2024 20:32:31.530 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat/webapps/candlepin]
Jul 25 20:32:34 parton server[105387]: 25-Jul-2024 20:32:34.961 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Jul 26 08:36:58 parton server[105387]: 26-Jul-2024 08:36:58.051 INFO [main] liquibase.database.null Set default schema name to public

it also looking like foreman is getting error 500