[ContentMigration] Failed content migration for Yum repositories

Support
1

Problem:
Content Migration doesn’t seem to have succeeded properly for yum repositories

Expected outcome:
Yum repositories should be served be pulp3

Foreman and Proxy versions:
2.3.2

Foreman and Proxy plugin versions:
foreman-tasks 3.0.2
foreman_ansible 6.1.1
foreman_remote_execution 4.2.2
katello 3.18.1

Distribution and version:
CentOS 7.9.2009

Other relevant data:
My coworker and I have been trying to figure this out for a while and we have gone through many of the steps already mentioned in other posts in this forum.

When my coworker first ran the migration instructions mentioned here (Foreman :: Plugin Manuals) he ran into an issue where the migration failed most of the way through. It was marked as success though, even those the repositories were still served via pulp2. Content was migrated to the /var/lib/pulp/media/ directory, but the apache config for /pulp/repos and the links in /var/lib/pulp/published/ did not change.

I picked it up yesterday, as I was trying to use Foreman to run some critical updates, and found that the repositories have not been updated in pulp in quite some time. Upon further inspection, it seems that the following command failed during the migration:

LANG=en_US.utf-8 foreman-installer --foreman-proxy-content-proxy-pulp-isos-to-pulpcore=true --katello-use-pulp-2-for-file=false --katello-use-pulp-2-for-docker=false --katello-use-pulp-2-for-yum=false

The reason that command was failing was because it was trying to run # pulpcore-manager in the background, which was failing because the CONTENT_ORIGIN in the /etc/pulp/settings.py file was “invalid” as it put it.

Upon even further inspection, it would seem that the pulpcore-content service is only listening on a unix socket and not on a TCP socket. The CONTENT_ORIGIN is set by the above foreman-installer command, and is being set to “https://<server_fqdn>”. For some reason, pulpcore-manager doesn’t like this, and there is no port specifier on the CONTENT_ORIGIN as is suggested in the pulp docs (Settings — Pulp Project 3.9.1 documentation)

After figuring out a way to foricibly set CONTENT_ORIGIN to a value that would allow the foreman-installer command above to succeed, we re-ran through the migration steps again, using the methods mentioned elsewhere in this forum (ie. adding things to /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/rails/configuration.rb) and the migration finally succeeded.

The issue, though, is that the smart proxies section of the foreman UI shows yum under pulpcore, but the repos are still being served via pulp2 and are no longer being updated with new content now because the foreman instance thinks it should be generated under pulpcore(pulp3) instead. I am thinking that something is missing to change the apache redirects to the right stuff, but since I had to force the template for the settings.py file to get the migration to work, something else is wrong and it was time to stop and seek further advice. If more information is needed, please let me know. We would like to use foreman to run the updates, but right now we are dead in the water.

2

More digging here, I did a foreman-installer --full-help to take a look at some of the settings, and the --foreman-proxy-plugin-pulp-pulpcore-api-url seems to be set to just https://<fqdn> where <fqdn> is the host.domain of our server. I think that is definitely wrong. But I don’t have any idea what that would be set to.

I also poked at the url: https://<fqdn>/pulp/content and that seems to house the pulp3 hosted repos! BUT the published URL for the repos listed in the foreman UI is https://<fqdn>//pulp/repos/<org>/<env>/custom/<product>/<repo>/ So I am very confused.

The layers of the onion are pealing back, but I have yet to fully understand what went wrong here and why. I don’t want to have to do a full re-install, mostly because I am not yet 100% convinced it will solve the issue.

3

I just realized that if the foreman-installer fails and pulpcore-manager spits out the error about CONTENT_ORIGIN, the issue is the permissions it seems. After a run of foreman-installer the permissions are set to
-rw-r-----. 1 root pulp 1508 Jan 29 12:47 settings.py

If I chmod it to 644, pulpcore-manager works and foreman-installer runs without issue again, but changes the perms back to 640. So changing the CONTENT_ORIGIN doesn’t seem to be necessary. This still doesn’t really explain the issue with the repository not being published to the same URL as the pulp2 repo was. Also, the pulp3 location does not require the auth certificate like the pulp2 repo did. It’s wide open. (ie I can go to https://<fqdn>/pulp/content in my browser that doesn’t have the debug cert and it works fine.)

4

This adventure continues. I made a change to the apache configuration to redirect /pulp/repos/ to the new pulp3 url. Inside of /etc/httpd/conf.d/pulp_rpm.conf I changed the following config block:

#
# This Location block replaces an `Alias` directive. In order to maintain
# backwards compatibility with existing Yum repository configurations, this
# block rewrites all requests to `/pulp/repos/` to the location of the WSGI
# application, content.wsgi, provided by the Pulp platform. The content.wsgi
# application adds support for downloading content on-demand.
<Location /pulp/repos/>
  RewriteEngine On
  RewriteCond %{HTTPS} on
  RewriteRule (.+/pulp/repos/)(.*) /pulp2/content/var/www/pub/yum/https/repos/$2 [DPI]
  RewriteCond %{HTTPS} off
  RewriteRule (.+/pulp/repos/)(.*) /pulp2/content/var/www/pub/yum/http/repos/$2 [DPI]
  RewriteCond %{HTTPS} off
</Location>

To read:

# -- Yum Repositories ---------
#
# This Location block replaces an `Alias` directive. In order to maintain
# backwards compatibility with existing Yum repository configurations, this
# block rewrites all requests to `/pulp/repos/` to the location of the WSGI
# application, content.wsgi, provided by the Pulp platform. The content.wsgi
# application adds support for downloading content on-demand.
<Location /pulp/repos/>
  RewriteEngine On
  RewriteCond %{HTTPS} on
#  RewriteRule (.+/pulp/repos/)(.*) /pulp2/content/var/www/pub/yum/https/repos/$2 [DPI]
  RewriteRule (.+/pulp/repos/)(.*) /pulp/content/$2 [DPI]
  RewriteCond %{HTTPS} off
#  RewriteRule (.+/pulp/repos/)(.*) /pulp2/content/var/www/pub/yum/http/repos/$2 [DPI]
  RewriteRule (.+/pulp/repos/)(.*) /pulp/content/$2 [DPI]
  RewriteCond %{HTTPS} off
</Location>

This pointed the old repo URL to the new pulp3 repos. Not sure if that change was supposed to happen as part of the migration or not. I will also keep looking into the authentication problems. Would be nice to have the repos be protected again.

5

Did you ever figure out the correct fix for this? I’ve got the exact same problem, pulp3 converted, but the /pulp/repos url’s still serving pulp2 content, every time I do a 3.18 minor update the apache config fix gets reverted

6

And of course I figured it out shortly after posting

foreman-installer --foreman-proxy-content-proxy-pulp-yum-to-pulpcore true --foreman-proxy-content-proxy-pulp-isos-to-pulpcore true --foreman-proxy-content-proxy-pulp-deb-to-pulpcore true

Fixed it

1 Like