Error while syncing Red Hat repositories

Can you try turning the download-concurrency down to 1 and trying again?

Tried with concurrency 1. Still errored out.

Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: TLSV1_ALERT_INTERNAL_ERROR] tlsv1 alert internal error (_ssl.c:877)]

Hallo,

Iā€™m facing the same error during repo sync.
(Cannot connect to host cdn.redhat.com:443 ssl:default [None]. Or ā€œServer Disconnectedā€. Depends where I look into the Foreman/Katello.)

After some tests I found that I get the same issue with Katello 3.16, 3.17 and 3.18.
But when I install Katello with Pulp v2 (foreman-installer --scenario katello ā€¦ --katello-use-pulp-2-for-yum=true) then the sync works properly.

So I try to get deeper in the changelog of Pulp v3.

Best regards.

Sorry for the delayed response. Was OOO.

@nellumhabemus

Thank you for the confirmation. Did you have to re-install Foreman from scratch to use pulp-2 or just rerun the installer with the above parameter?

@Justin_Sherrill @jeremylenz

Any idea how I can submit a bug report for this issue with pulp-3?

Hallo Schaudhary,

I installed a fresh Foreman using ā€œā€“katello-use-pulp-2-for-yum=trueā€.

(But indeed first I tried just to re-run the foreman-installer with this parameter on a running Foreman instance but this doesnā€™t work. I did not really suppose a backward-migration from Pulp3 to Pulp2 is implemented. But I can only guess. This week Iā€™m OOO. Next week I try to go a bit deeper.)

Best regards.

@schaudhary a pulp bug would be good, but i think without a reproducer it wouldnā€™t be very actionable. This new error: TLSV1_ALERT_INTERNAL_ERROR seems new? do you have a traceback in ā€˜journalctl -u pulpcore-worker@*ā€™

@nellumhabemus
Thank you for your response and efforts in trying to get this resolved.

@Justin_Sherrill
You are correct - it would not be very productive if we keep jumping from one error message to another. I think I should be able to reproduce the original error and we can focus on that to start, if it helps resolve this.

Also, is there a supported way to disable pulp-3 and enable pulp-2, without installing from scratch?

Saurabh

Thereā€™s not a way to go backwards sadly. I was actually interested in the new error and what it looked like fully. I get the error keeps changing, but weā€™re also trying different options which may be effecting what weā€™re seeing.

Iā€™ve been able to sync RH Products using pulp3 without issue, so i suspect there may be something locally within your network causing the issue or some configuration.

I tested 3 installations from the scratch:

  • Katello 3.18 with Pulp 2
  • Katello 3.18 with Pulp 3
  • Katello 4 (of course with Pulp 3)

These 3 instances have direct Internet access.

Result: All 3 instances can enable and sync several Redhat repos successfully.

(Iā€™m not sure why my problem vanished may be some bugs got fixed within the last 2 weeks?)

Now I try again Katello 4 in our company network.

1 Like

@Justin_Sherrill @nellumhabemus

I tried 3.18 with pulp2. It seemed to be working fine. I saw your message about 4 with pulp3 working, so tried that. It did not work for me.

So, Iā€™m just going to revert back to 3.18 with pulp2 and hope I donā€™t encounter any more issues.

Correction!

Unfortunately at least my test with Katello 4 from the scratch is not valid because of my test steps:

1st I did a repo sync using the default setting ā€œDefault Red Hat Repository download policyā€ == ā€œon demandā€
=> Successful.
2nd I changed the ā€œDefault Red Hat Repository download policyā€ to ā€œimmediateā€
3rd I did the repo sync again
=> Successful.

BUT: Step 2 did not work!

I can see that there are no RPMs synced indeed using:
file /var/lib/pulp/media/artifact// | grep RPM

Only after I disabled and enabled a Redhat repo again then its RPMs are synced really to my local disc.

And now this true immediate-sync fails again: This time with error ā€œResponse payload is not completedā€

This command is not the right way to check if rpm files are there. Pulp3 does not store files as .rpm. Instead is likely based of check sums. if you did
ls /var/lib/pulp/media/artifact/*/* you 'd see something like
/var/lib/pulp/media/artifact/ff/d1780fea7878ad33926d748ac61b14caeeece2eb6513c66ad3ff7ad3cfbc8a
Notice no rpm ending.

Currently I cant think of a good way to verify if the immediate rpms got downloaded other than monitoring the disk space before sync and after of du -s /var/lib/pulp.

2 Likes

Yes, I agree completely. Thatā€™s why I use the ā€˜fileā€™ command. :slight_smile:

Does anyone know of a fix for thisā€¦ I have a fresh install of Foreman 4.5.1 and Katello 4.1.0 and run into the same issue when i try and sync Index of /distribution/leap/15.3/repo/ossā€¦

Error: pulpcore-worker-1[36974]: aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected

I have changed the concurrency and the bulk value to a number of different ones,

Concurrency change from 10-1 even trying 2,5 and even going to 20
Bulk Change 2000 - 500 and even trying 1000

None of the changes i have tried allow me to sync the repo. i can do smaller repos but not large ones it seems.

@cflannigan Thanks for letting us know about this issue.

This might be related to a recent pulpcore issue Issue #8867: downloads from cdn.redhat fail with 'Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]' - Pulp, which was fixed in the pulpcore 3.14 release.

Katello 4.1.1 (released July 13) includes pulpcore 3.14, so I would suggest upgrading to that version and trying again.

Seems to me katello 4.1.1 has just arrived today in the repositories. My sync this morning didnā€™t pick it up, yet. And 4.1.1 also seems to be required to upgrade to foreman 2.5.2ā€¦

I was overly optimistic about 4.1.1 - although the gem was released, the rpm build is just now being finalized. It should be ready very soon.

Thanks for the update ill check and plan the upgrade and hopefully solves the issue for me.

Do not update to 4.1.1 at the moment. There is a serious bug in the pulp3 rpm module which prevents import of the rpm package metadata into the database causing failing dependencies on the clients.

See topic Cannot update puppet-agent to 6.24.0 on foreman servers