Creating repo results in internal server error 500

Problem: trying to create a repo results in a internal server 500

Expected outcome: repo created from product

**Foreman and Proxy versions:
Templates, Pulp, TFTP, Puppet, Puppet CA, and Logs 1.20.3

Foreman and Proxy plugin versions:

bastion Bastion provides a UI library of AngularJS based components designed to integrate and work with Foreman. Eric D Helms and Walden Raines 6.1.16
foreman-tasks The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks. Ivan Nečas 0.14.3
foreman_docker Provision and manage Docker containers and images from Foreman. Daniel Lobato, Amos Benari 4.1.0
katello Content and Subscription Management plugin for Foreman N/A 3.10.1.1

Distribution and version:
3.10.0-2

Other relevant data:
hammer command

hammer repository create
ā€“product ā€œCentOS7ā€
ā€“name ā€œbase_x86_64ā€
ā€“label ā€œbase_x86_64ā€
ā€“content-type ā€œyumā€
ā€“download-policy ā€œon_demandā€
ā€“url ā€œhttp://mirror.centos.org/centos/7/os/x86_64/ā€
ā€“mirror-on-sync ā€œnoā€
ā€“organization ā€œDefault Organizationā€

/var/log/messages

pulp: pulp.server.webservices.middleware.exception:ERROR: (14383-87776) IOError: [Errno 13] Permission denied: ā€˜/etc/pki/pulp/content/pulp-protected-reposā€™

I tried
chown apache:apache /etc/pki/pulp/content/
chmod 644 /etc/pki/pulp/content

touch /etc/pki/pulp/content/pulp-protected-repos
chmod 644 /etc/pki/pulp/content/pulp-protected-repos
chown apache:apache /etc/pki/pulp/content/pulp-protected-repos

and
foreman-installer --scenario katello --foreman-proxy-plugin-pulp-enabled

but it keeps throwing the same error. I can create a product, just not a repo
Iā€™ve noticed I donā€™t have a pulp group

Are there any relevant looking denials in the selinux audit.log ?

I just checked by rerunning the hammer command while tailing the sexlinux file and no errors were thrown.

type=LOGIN msg=audit(1595433661.876:1980): pid=28984 uid=0 old-auid=4294967295 auid=0 tty=(none) old-ses=4294967295 ses=233 res=1
type=USER_START msg=audit(1595433661.880:1981): pid=28984 uid=0 auid=0 ses=233 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1595433661.881:1982): pid=28984 uid=0 auid=0 ses=233 msg='op=PAM:setcred grantors=pam_env,pam_pkcs11 acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_DISP msg=audit(1595433661.893:1983): pid=28984 uid=0 auid=0 ses=233 msg='op=PAM:setcred grantors=pam_env,pam_pkcs11 acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1595433661.895:1984): pid=28984 uid=0 auid=0 ses=233 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

@laferrierejc alright. are there any errors logged to journalctl or /var/log/message for pulp when this occurs?

What are the permissions on /etc/pki/pulp/content? Mine are drwxr-xr-x. 2 apache apache 80 Jul 9 2019 content

and pulp-protected-repos: -rw-r--r--. 1 apache apache 0 Jul 22 14:09 pulp-protected-repos

Lastly, just to confirm - was this working recently, and then it stopped? Can you think of something which might have influenced that? (maybe some system patches or somethingā€¦)

What are the permissions on /etc/pki/pulp/content? Mine are drwxr-xr-x. 2 apache apache 80 Jul 9 2019 content

journalctl | grep pulp when I reran hammer cli
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14383]: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296) Exception adding distributor to repo [6ae5cf8b-ab01-46d3-bce2-f40b5cffcde4]; the repo will be deleted
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296) Traceback (most recent call last):
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 434, in create_repo
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 67, in add_distributor
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     result = distributor_instance.validate_config(transfer_repo, call_config, config_conduit)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 85, in validate_config
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     return configuration.validate_config(repo, config, config_conduit)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 146, in validate_config
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     process_cert_based_auth(repo, config)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 169, in process_cert_based_auth
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     protected_repo_utils_instance.delete_protected_repo(relative_path)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 63, in delete_protected_repo
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     f.save()
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 143, in save
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296)     f = open(self.filename, 'w')
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.controllers.repository:ERROR: (14382-51296) IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296) [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296) Traceback (most recent call last):
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     return self.dispatch(request, *args, **kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     return handler(request, *args, **kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     value = method(self, *args, **kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     return func(*args, **kwargs)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 128, in post
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     distributor_list=repo_data.get('distributors')
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 434, in create_repo
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 67, in add_distributor
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     result = distributor_instance.validate_config(transfer_repo, call_config, config_conduit)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 85, in validate_config
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     return configuration.validate_config(repo, config, config_conduit)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 146, in validate_config
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     process_cert_based_auth(repo, config)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 169, in process_cert_based_auth
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     protected_repo_utils_instance.delete_protected_repo(relative_path)
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 63, in delete_protected_repo
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     f.save()
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 143, in save
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296)     f = open(self.filename, 'w')
    Jul 22 17:06:18 katellopredeploy.changed.com pulp[14382]: pulp.server.webservices.middleware.exception:ERROR: (14382-51296) IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'

Permissions

[root@katellopredeploy ~]# ls  /etc/pki/pulp/content -la
total 0
drw-r--r--. 2 apache apache 61 Jul 21 11:14 .
drwxr-xr-x. 3 root   root   83 Jul 13 18:05 ..
lrwxrwxrwx. 1 root   root   20 Jul 13 18:05 pulp-global-repo.ca -> /etc/pki/pulp/ca.crt
-rw-r--r--  1 apache apache  0 Jul 21 11:14 pulp-protected-repos

ā€œwas this working recently, and then it stopped?ā€
No. It threw that error first time I tried to create a repo. I havenā€™t done any yum updates. I read about an error in the messages to run pulp-manage-db where I had to pkill celery in order to do so. But that did not solve my problem (pulp-manage-db did run successfully).

This is supposed to be an initial setup following some tutorials.

I could wipe out and rerun. I had an outdated tutorial on repos (https://www.itzgeek.com/how-tos/linux/centos-how-tos/configure-katello-download-repositories-create-environments-and-activation-keys.html) but I was augmenting the instructions by looking at another katello server (I rewrote the instructions basically, but itā€™s simplified with the hammer cli I ran).

tail -f /var/log/messages when I run the hammer cli

    [root@katellopredeploy ~]# tail -f /var/log/messages | grep pulp
    <27>2020-07-22T17:01:51.811343-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     download_requests = _create_download_requests(deferred_content_units)
    <27>2020-07-22T17:01:51.811618-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1474, in _create_download_requests
    <27>2020-07-22T17:01:51.811850-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     working_dir = common_utils.get_working_directory()
    <27>2020-07-22T17:01:51.812075-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/_common.py", line 159, in get_working_directory
    <27>2020-07-22T17:01:51.812334-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     os.mkdir(working_dir_root)
    <27>2020-07-22T17:01:51.812635-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-0@katellopredeploy.changed.com/da09debb-8e10-45b4-811d-f77d23058582'

    [root@katellopredeploy ~]# tail -f /var/log/messages | grep pulp
    <27>2020-07-22T17:01:51.811343-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     download_requests = _create_download_requests(deferred_content_units)
    <27>2020-07-22T17:01:51.811618-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1474, in _create_download_requests
    <27>2020-07-22T17:01:51.811850-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     working_dir = common_utils.get_working_directory()
    <27>2020-07-22T17:01:51.812075-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/_common.py", line 159, in get_working_directory
    <27>2020-07-22T17:01:51.812334-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     os.mkdir(working_dir_root)
    <27>2020-07-22T17:01:51.812635-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-0@katellopredeploy.changed.com/da09debb-8e10-45b4-811d-f77d23058582'

    [root@katellopredeploy ~]# clear
    [root@katellopredeploy ~]# tail -f /var/log/messages | grep pulp
    <27>2020-07-22T17:01:51.811343-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     download_requests = _create_download_requests(deferred_content_units)
    <27>2020-07-22T17:01:51.811618-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1474, in _create_download_requests
    <27>2020-07-22T17:01:51.811850-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     working_dir = common_utils.get_working_directory()
    <27>2020-07-22T17:01:51.812075-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)   File "/usr/lib/python2.7/site-packages/pulp/server/managers/repo/_common.py", line 159, in get_working_directory
    <27>2020-07-22T17:01:51.812334-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000)     os.mkdir(working_dir_root)
    <27>2020-07-22T17:01:51.812635-04:00 katellopredeploy pulp: celery.app.trace:ERROR: [da09debb] (14296-68000) OSError: [Errno 2] No such file or directory: '/var/cache/pulp/reserved_resource_worker-0@katellopredeploy.changed.com/da09debb-8e10-45b4-811d-f77d23058582'
    <30>2020-07-22T17:05:01.624613-04:00 katellopredeploy pulp: kombu.transport.qpid:INFO: Connected to qpid with SASL mechanism ANONYMOUS
    <27>2020-07-22T17:05:01.673999-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184) Exception adding distributor to repo [1009ca7e-7e1e-4446-9804-0c3a0e91f0aa]; the repo will be deleted
    <27>2020-07-22T17:05:01.674222-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184) Traceback (most recent call last):
    <27>2020-07-22T17:05:01.674407-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 434, in create_repo
    <27>2020-07-22T17:05:01.674595-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
    <27>2020-07-22T17:05:01.674769-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 67, in add_distributor
    <27>2020-07-22T17:05:01.674939-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     result = distributor_instance.validate_config(transfer_repo, call_config, config_conduit)
    <27>2020-07-22T17:05:01.675127-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 85, in validate_config
    <27>2020-07-22T17:05:01.675298-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     return configuration.validate_config(repo, config, config_conduit)
    <27>2020-07-22T17:05:01.675468-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 146, in validate_config
    <27>2020-07-22T17:05:01.675643-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     process_cert_based_auth(repo, config)
    <27>2020-07-22T17:05:01.675812-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 169, in process_cert_based_auth
    <27>2020-07-22T17:05:01.675984-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     protected_repo_utils_instance.delete_protected_repo(relative_path)
    <27>2020-07-22T17:05:01.676164-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 63, in delete_protected_repo
    <27>2020-07-22T17:05:01.676335-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     f.save()
    <27>2020-07-22T17:05:01.676515-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 143, in save
    <27>2020-07-22T17:05:01.676685-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184)     f = open(self.filename, 'w')
    <27>2020-07-22T17:05:01.676859-04:00 katellopredeploy pulp: pulp.server.controllers.repository:ERROR: (14384-73184) IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'
    <27>2020-07-22T17:05:01.677027-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: Unhandled Exception
    <27>2020-07-22T17:05:01.677211-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184) [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'
    <27>2020-07-22T17:05:01.677393-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184) Traceback (most recent call last):
    <27>2020-07-22T17:05:01.677564-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    <27>2020-07-22T17:05:01.677733-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     response = wrapped_callback(request, *callback_args, **callback_kwargs)
    <27>2020-07-22T17:05:01.677903-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 68, in view
    <27>2020-07-22T17:05:01.678086-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     return self.dispatch(request, *args, **kwargs)
    <27>2020-07-22T17:05:01.678259-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 88, in dispatch
    <27>2020-07-22T17:05:01.678434-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     return handler(request, *args, **kwargs)
    <27>2020-07-22T17:05:01.678605-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 241, in _auth_decorator
    <27>2020-07-22T17:05:01.678773-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     return _verify_auth(self, operation, super_user_only, method, *args, **kwargs)
    <27>2020-07-22T17:05:01.678941-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/decorators.py", line 195, in _verify_auth
    <27>2020-07-22T17:05:01.679123-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     value = method(self, *args, **kwargs)
    <27>2020-07-22T17:05:01.679294-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/util.py", line 130, in wrapper
    <27>2020-07-22T17:05:01.679463-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     return func(*args, **kwargs)
    <27>2020-07-22T17:05:01.679636-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/webservices/views/repositories.py", line 128, in post
    <27>2020-07-22T17:05:01.679818-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     distributor_list=repo_data.get('distributors')
    <27>2020-07-22T17:05:01.679987-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 434, in create_repo
    <27>2020-07-22T17:05:01.680170-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     dist_controller.add_distributor(repo_id, type_id, plugin_config, auto_publish, dist_id)
    <27>2020-07-22T17:05:01.680339-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/distributor.py", line 67, in add_distributor
    <27>2020-07-22T17:05:01.680521-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     result = distributor_instance.validate_config(transfer_repo, call_config, config_conduit)
    <27>2020-07-22T17:05:01.680705-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/distributor.py", line 85, in validate_config
    <27>2020-07-22T17:05:01.680878-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     return configuration.validate_config(repo, config, config_conduit)
    <27>2020-07-22T17:05:01.681094-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 146, in validate_config
    <27>2020-07-22T17:05:01.681276-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     process_cert_based_auth(repo, config)
    <27>2020-07-22T17:05:01.681455-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/distributors/yum/configuration.py", line 169, in process_cert_based_auth
    <27>2020-07-22T17:05:01.681647-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     protected_repo_utils_instance.delete_protected_repo(relative_path)
    <27>2020-07-22T17:05:01.681837-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 63, in delete_protected_repo
    <27>2020-07-22T17:05:01.682010-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     f.save()
    <27>2020-07-22T17:05:01.682194-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)   File "/usr/lib/python2.7/site-packages/pulp/repoauth/protected_repo_utils.py", line 143, in save
    <27>2020-07-22T17:05:01.682384-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184)     f = open(self.filename, 'w')
    <27>2020-07-22T17:05:01.682570-04:00 katellopredeploy pulp: pulp.server.webservices.middleware.exception:ERROR: (14384-73184) IOError: [Errno 13] Permission denied: '/etc/pki/pulp/content/pulp-protected-repos'

Thanks for the extra information. Since you mentioned that this is your initial setup, I wonder if you could re-install Katello and use version 3.15? 3.10 is quite old, and no longer supported.

I can do that. I was installing the latest version that was available in centos 7.

I have this guide on how to install 3.15

https://theforeman.org/plugins/katello/3.15/installation/index.html

I just fear Iā€™m going to be trading one set of problems for another.

The problem Iā€™m having appears to be this documented bug since 2012: https://bugzilla.redhat.com/show_bug.cgi?id=871075

That would be best and Iā€™m glad itā€™s an option for you. I would like to think that there would be fewer issues to run into in newer versions. If you ran into that permissions error in 3.15 then we could investigate it much more readily because itā€™d be much more likely that we can reproduce the problem. Let us know how it goes :slight_smile:

Hmmā€¦ Iā€™m going to say this is resolved.

When I did the install regular like. I was able to create the repo

I then had some commands to mount an nfs store

mkdir /mnt/nfsshare
mount -t nfs dataserver0.nunya.biz:/katello /mnt/nfsshare/
rsync -vrazh /var/lib/pulp/* /mnt/nfsshare/
	
mv /var/lib/pulp /var/lib/bakpulp
	
mkdir /var/lib/pulp
	
chmod --reference=/var/lib/bakpulp /var/lib/pulp
chown --reference=/var/lib/bakpulp /var/lib/pulp

Then I mounted in fstab (and rebooted)

echo dataserver0.nunya.biz:/katello      /var/lib/pulp   nfs     defaults        0 0 >> /etc/fstab

Then I got an error 400 when I tried to create a new repoā€¦ so I compared perms on the new parent folder against bakpulp and saw bakpulp had pulp:pulp vs root:root.

So I needed to do a

chown pulp:pulp /var/lib/pulp

but itā€™s working now. Thank you.

1 Like

We are facing the same issue that pulp-protected-repos could not be created because of selinux.
This is on a foreman smart proxy 2.3 with pulp-selinux-2.21.5-1.

[root@wall-e pulp]# ls --lcontext /etc/pki/pulp/
 insgesamt 16
 rw-r----. 1 unconfined_u:object_r:cert_t:s0 root apache 1724 16. Apr 12:14 ca.crt
 rw-r----. 1 unconfined_u:object_r:cert_t:s0 root apache 3243 16. Apr 12:14 ca.key
 drwxr-xr-x. 2 system_u:object_r:cert_t:s0 apache apache 33 16. Apr 12:13 content
 drwxr-x---. 2 system_u:object_r:cert_t:s0 root pulp 38 16. Apr 12:14 qpid
 rw-r----. 1 system_u:object_r:cert_t:s0 root apache 1675 16. Apr 12:13 rsa.key
 rw-rr-. 1 unconfined_u:object_r:cert_t:s0 root apache 451 16. Apr 12:13 rsa_pub.key


 [root@wall-e pulp]# /sbin/restorecon -v -i -R /etc/pki/pulp
 /sbin/restorecon reset /etc/pki/pulp context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/content context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/content/pulp-global-repo.ca context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/qpid context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/qpid/client.crt context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/qpid/ca.crt context unconfined_u:object_r:cert_t:s0->unconfined_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/rsa.key context system_u:object_r:cert_t:s0->system_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/rsa_pub.key context unconfined_u:object_r:cert_t:s0->unconfined_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/ca.key context unconfined_u:object_r:cert_t:s0->unconfined_u:object_r:pulp_cert_t:s0
 /sbin/restorecon reset /etc/pki/pulp/ca.crt context unconfined_u:object_r:cert_t:s0->unconfined_u:object_r:pulp_cert_t:s0

 [root@wall-e pulp]# ls --lcontext /etc/pki/pulp/
 insgesamt 16
 rw-r----. 1 unconfined_u:object_r:pulp_cert_t:s0 root apache 1724 16. Apr 12:14 ca.crt
 rw-r----. 1 unconfined_u:object_r:pulp_cert_t:s0 root apache 3243 16. Apr 12:14 ca.key
 drwxr-xr-x. 2 system_u:object_r:pulp_cert_t:s0 apache apache 33 16. Apr 12:13 content
 drwxr-x---. 2 system_u:object_r:pulp_cert_t:s0 root pulp 38 16. Apr 12:14 qpid
 rw-r----. 1 system_u:object_r:pulp_cert_t:s0 root apache 1675 16. Apr 12:13 rsa.key
 rw-rr-. 1 unconfined_u:object_r:pulp_cert_t:s0 root apache 451 16. Apr 12:13 rsa_pub.key

As mentioned above, the failure can be fixed by calling /sbin/restorecon -i -R /etc/pki/pulp. This is more like a workaround.

I would expect that the context are set correctly when installing the RPM. The question for me is, why is the context for /etc/pki/pulp not set to pulp_cert_t? Which script and to which step should the context be set? (by RPM post trans)?

Do you, like @Marek_Hulan, @ekohl or @evgeni know if there was a change in foreman 2.3 / pulp / smart-proxy which would affect the smart proxy installation and the mentioned behavior that the pulp selinux rules are not configured as it need to be?

No, and given our pipelines create repos every day (and release), there needs to be something more to your setup that is missing.