Cannot use "file" type repository in Katello 4.1

caseybea · July 9, 2021, 7:58pm

So, doing some very initial testing to see if I want to start using Katello 4.

Brand new install. And right out of the gate, I have the following issue:

When creating a repository to point to an EXISTING directory for repository content, I get the following error:

Error message: the server returns an error
HTTP status code: 400
Response headers: {“date”=>“Fri, 09 Jul 2021 19:42:45 GMT”, “server”=>“gunicorn/20.0.4”, “content-type”=>“application/json”, “vary”=>“Accept,Cookie”, “allow”=>“GET, POST, HEAD, OPTIONS”, “x-frame-options”=>“SAMEORIGIN”, “content-length”=>“92”, “correlation-id”=>“60029fe5-0484-4cae-8f7d-97870613b7fd”, “access-control-expose-headers”=>“Correlation-ID”, “via”=>“1.1 my.hostname.here”, “connection”=>“close”}
Response body: {“url”:[“url ‘file://var/www/html/rocky8/BaseOS/x86_64/os/’ is not an allowed import path”]}

caseybea · July 9, 2021, 9:03pm

CLARIFICATION: This isn’t the “file” type repository, it’s a YUM repository (with file:// as the location, it’s local to the server).

gvde · July 10, 2021, 6:08am

Well, as the error says, it‘s not supported. Generally, file URLs are local client side resources and supporting those would be pretty confusing, I guess.

caseybea · July 10, 2021, 5:35pm

This works perfectly under katello 3.18. Ive been doing it for years.

gvde · July 10, 2021, 5:54pm

Interesting. It seems you have to configure that path as allowed import path with pulp3.

github.com

pulp/pulpcore/blob/81684b206e3fd8bceb104e4a77c0d717387667f9/docs/configuration/settings.rst#allowed_export_paths

.. _settings:

Settings
========

Pulp uses three types of settings:

* :ref:`Django settings <django-settings>` Pulp is configuring
* :ref:`RQ settings <rq-settings>` Pulp is using
* :ref:`Pulp defined settings <pulp-settings>`


.. _django-settings:

Django Settings
---------------

Pulp is a Django project, so any Django `Django setting
<https://docs.djangoproject.com/en/2.2/ref/settings/>`_ can also be set to configure your Pulp
deployment.

This file has been truncated. show original

caseybea · July 10, 2021, 8:44pm

Yikes. I will file a katello bug. Entering a file path should then update the pulp settings, yes? This is a deal-breaker for me.

gvde · July 11, 2021, 5:12am

I don‘t know. From security aspects it‘s good not to allow it. You‘ll never know what someone might enter there. Maybe, it could become a foreman-installer option allowing you some local import paths.

Otherwise, I think you‘ll have to change /etc/pulp/settings.py and then check after the next installer run if it sticks…

But generally, I think it‘s kind of a weird setup to mirror a distribution to the local file system and then copy it from there into the pulp directory…

caseybea · July 12, 2021, 1:32am

Not a weird setup at all. I use rsync to get my repos, then insert errata for centos.

Also, file:// access is way quicker than going to a remote repo elsewhere.

I’ll have to hack my setup to access via an internal web site as a stopgap probably.

Frustrating.

caseybea · July 12, 2021, 1:35am

And updating settings.py manually could become a nightmare. I would have to manually include every path for every single repository. MESSY.

gvde · July 12, 2021, 3:53am

Right. That’s a good reason.

How many different directories are you talking about? Above it looked like /var/www/html/ That’s all you need. Did you read the documentation for the ALLOWED_IMPORT_PATHS in the link above?

On a sidenote: you could move all that into the pub subdirectory, i.e. /var/www/html/pub On my foreman installation that is presented as simple directory through the web server, i.e. you could access everything via http/https as well, in case you need it… http://foreman.example.com/pub/

gvde · July 12, 2021, 4:28am

On another sidenote: I have the impression that katello 4 with pulp3 needs more host resources (cpu, memory, disk) than the old version. I hope that’s getting better with the time but still, personally, I wouldn’t mirror repositories into a local directory on the foreman server for staging. I am running everything in VMs and my foreman VM uses fast SSD storage while I have another mirror VM which regularly rsyncs various distributions using slower HDD storage. But again, that’s more like an impression or feeling at this time and nothing I really tested or measured…

caseybea · July 12, 2021, 1:38pm

If i can put var/www/html only in, that would work. I will experiment today

caseybea · July 12, 2021, 3:04pm

So, adding in just “/var/www/html/” does not work. It appears the FULL path has to be specified, which is ridiculous to do for every single repo I will have.

I did try moving the yum content under “pub” and set the repo to access the yum content via http: (on my katello host). It’s really sad however I now have to access the yum content via the network and not files directly.

gvde · July 12, 2021, 3:08pm

Well, the example in the docs I have linked to shows it should work. Or it’s the wrong docs.

The only thing from the docs which might make a different: the slash at the end. Try without it.

Did you stop all foreman and pulp services and started them again after the configuration change? a foreman-maintain service stop may not stop the pulpcore services even if it should.

Partha_Aji · July 13, 2021, 3:03pm

Couple of things to check:

Is the pulp user on the host able to read stuff from /var/www/html/ and below ? Try doing sudo su - pulp and ls “/var/www/html/ and directories below
As @gvde points out. You would have to restart pulpcore via foreman-maintain service restart after adding /var/www/html to the ALLOWED_IMPORT_PATHS in /etc/pulp/settings.py

Let us know if that does not work…

caseybea · July 13, 2021, 3:26pm

pulp has access. and REBOOTED. no go. I am using internal http as a short term workaround… others are gonna run into this too I suspect.

ekohl · July 13, 2021, 3:45pm

We do have Bug #29371: [puppet_pulpcore] Update pulp/settings.py to support pulpcore3.2 changes - Installer - Foreman but that isn’t really tracked as in which version, but it mentioned Pulp 3.2 so that’s old enough to be present.

The locations that are allowed are /var/lib/pulp/sync_imports and /var/lib/pulp/imports:

github.com

theforeman/puppet-foreman_proxy_content/blob/f0f477a9c0423ff48e0809b09bb39aef24b50815/manifests/init.pp#L213


      
          }
          
          if $pulpcore_mirror {
            $pulpcore_allowed_import_path    = ['/var/lib/pulp/sync_imports']
            $pulpcore_allowed_export_path    = []
          
            pulpcore::apache::fragment{'gpg_key_proxy':
              https_content => template('foreman_proxy_content/_pulp_gpg_proxy.erb'),
            }
          } else {
            $pulpcore_allowed_import_path    = ['/var/lib/pulp/sync_imports', '/var/lib/pulp/imports']
            $pulpcore_allowed_export_path    = ['/var/lib/pulp/exports']
          }
          
          if $shared_with_foreman_vhost {
            include foreman::config::apache
            $servername = $foreman::config::apache::servername
            $priority = $foreman::config::apache::priority
            $apache_http_vhost = 'foreman'
            $apache_https_vhost = 'foreman-ssl'
            $apache_https_cert = undef

Making manual changes to settings.py is not supported and will break in the next upgrade.

This doesn’t match the documentation. Settings — Pulp Project 3.53.0 documentation states:

only a subpath is needed

So if it doesn’t perhaps that’s a bug.

gvde · July 13, 2021, 4:21pm

Are you sure you made the correct change in settings.py? There is already a line for ALLOWED_IMPORT_PATHS in there, thus if you make your own setting before that it is going to be changed to the default:

ALLOWED_IMPORT_PATHS = ["/var/lib/pulp/sync_imports", "/var/lib/pulp/imports"]

What is the exact URL you have using? In your first post in the error message there is an incorrect url, but I am not sure if maybe somewhere down the line there was some text conversion involved. You seem to have file://var/www/html/rocky8/BaseOS/x86_64/os/. That equals to the file path “/www/html/rocky8/BaseOS/x86_64/os/” on host “var”!

The file URL scheme according to RFC 1738 looks like any other protocol scheme:

   A file URL takes the form:

       file://<host>/<path>

   where <host> is the fully qualified domain name of the system on
   which the <path> is accessible, and <path> is a hierarchical
   directory path of the form <directory>/<directory>/.../<name>.

Of course, the host part is empty for a local directory.

The double slash “//” is part of the prefix. I have just tested it with the predefined import path

file:///var/lib/pulp/imports/abc/def`

and I was able to set up a repository with that. Note the three (3) slashes after file:

gvde · July 13, 2021, 4:35pm

O.K. The current RFC for the file URI schema is in https://datatracker.ietf.org/doc/html/rfc8089. Compare the examples in Appendix B

So two forms are possible (and work with foreman):

file:/var/lib/pulp/imports/abc/def
file:///var/lib/pulp/imports/abc/def

Two slashes are not, even though looking at the BNF it seems it might.

caseybea · July 13, 2021, 5:22pm

OK, I’m trying it all again on a new setup. I got it to work for one of my repos (yay!), but I have a difficult issue related to doing this. THREE slashes now seems to be the key.

In the second repo I was doing, I of course accidently used the TWO shashes (that worked under foreman 3.18 btw…), and I got the same error aboout the pulp path being invalid. No big deal, went back to change it to three slashes. But now I can’t.

I am getting a conflict with the previous task (Update Repository…), and it’s in a paused state. I cannot for the life of me fgure out how to get rid of that task to prevent the conflict. I can’t kill it, and force-killing it doesnt work eiither, it’s still there. Resuming it doesn’t do anything, still in an error state.

I tried unlocking it, and that seems to have unlocked it, but then it’s back to an ERROR state, and I still gewt a task conflict with the repo when tryying to make the edit. arrggh…
Help?