Katello 3.0 No such file or directory @ rb_sysopen - /etc/pki/katello/certs/pulp-client.crt

Dylan_Baars · July 4, 2016, 11:48pm

Hi,

post my upgrade to katello 3.0
(https://groups.google.com/forum/#!topic/foreman-users/86Pt-1v0L1s) some of
my repository syncs have failed, I'm seeing this in the output

"No such file or directory @ rb_sysopen -
/etc/pki/katello/certs/pulp-client.crt"

[root@wellkatello certs]# pwd
/etc/pki/katello/certs
[root@wellkatello certs]# ls -l
total 44
-rw-r–r--. 1 root root 5496 Dec 8 2015 java-client.crt
-rw-r–r--. 1 root root 1717 Dec 8 2015 katello-apache.crt
-rw-r–r--. 1 root foreman 5431 Dec 8 2015 katello-default-ca.crt
-rw-r–r--. 1 root root 1765 Dec 8 2015 katello-default-ca-stripped.crt
-rw-r–r--. 1 root foreman 1826 Dec 8 2015 katello-server-ca.crt
-rw-r–r--. 1 root root 5390 Jul 2 15:51 pulp-client.crt
-rw-r–r--. 1 root root 5483 Dec 8 2015
wellkatello.niwa.local-qpid-broker.crt

and also

"Trying to use logger katello/pulp_rest which has not been configured."

Any ideas?

Chris_Duryee · July 5, 2016, 11:49am

The permissions for pulp-client.crt look correct at first glance. Can
you check for any selinux denials, and also send the output of "ls -lZ"
for the certs dir and for "/etc/pki/katello"?

···

On 07/04/2016 07:48 PM, Dylan Baars wrote: > Hi, > > post my upgrade to katello 3.0 > (https://groups.google.com/forum/#!topic/foreman-users/86Pt-1v0L1s) some of > my repository syncs have failed, I'm seeing this in the output > > "No such file or directory @ rb_sysopen - > /etc/pki/katello/certs/pulp-client.crt" > > [root@wellkatello certs]# pwd > /etc/pki/katello/certs > [root@wellkatello certs]# ls -l > total 44 > -rw-r--r--. 1 root root 5496 Dec 8 2015 java-client.crt > -rw-r--r--. 1 root root 1717 Dec 8 2015 katello-apache.crt > -rw-r--r--. 1 root foreman 5431 Dec 8 2015 katello-default-ca.crt > -rw-r--r--. 1 root root 1765 Dec 8 2015 katello-default-ca-stripped.crt > -rw-r--r--. 1 root foreman 1826 Dec 8 2015 katello-server-ca.crt > -rw-r--r--. 1 root root 5390 Jul 2 15:51 pulp-client.crt > -rw-r--r--. 1 root root 5483 Dec 8 2015 > wellkatello.niwa.local-qpid-broker.crt > > and also > > "Trying to use logger katello/pulp_rest which has not been configured." > > Any ideas? >

John_Mitsch · July 5, 2016, 12:53pm

> "Trying to use logger katello/pulp_rest which has not been configured."

Can you check your katello.yaml file for these lines?

You can add these in (uncommented) and also include the :loggers: line
above, which allows you to enable or disable pulp logging, which may get
rid of that error message.

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

···

On Mon, Jul 4, 2016 at 7:48 PM, Dylan Baars wrote:

Hi,

post my upgrade to katello 3.0 (
https://groups.google.com/forum/#!topic/foreman-users/86Pt-1v0L1s) some
of my repository syncs have failed, I’m seeing this in the output

“No such file or directory @ rb_sysopen -
/etc/pki/katello/certs/pulp-client.crt”

[root@wellkatello certs]# pwd
/etc/pki/katello/certs
[root@wellkatello certs]# ls -l
total 44
-rw-r–r--. 1 root root 5496 Dec 8 2015 java-client.crt
-rw-r–r--. 1 root root 1717 Dec 8 2015 katello-apache.crt
-rw-r–r--. 1 root foreman 5431 Dec 8 2015 katello-default-ca.crt
-rw-r–r--. 1 root root 1765 Dec 8 2015
katello-default-ca-stripped.crt
-rw-r–r--. 1 root foreman 1826 Dec 8 2015 katello-server-ca.crt
-rw-r–r--. 1 root root 5390 Jul 2 15:51 pulp-client.crt
-rw-r–r--. 1 root root 5483 Dec 8 2015
wellkatello.niwa.local-qpid-broker.crt

and also

“Trying to use logger katello/pulp_rest which has not been configured.”

Any ideas?

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Dylan_Baars · July 6, 2016, 1:24am

Hi guys,

I just compared the permissions to my test instance - they are the same. No
selinux errors, and here's the output of ls -lZ on the certs dir and one up
(/etc/pki/katello)

[root@wellkatello certs]# ls -lZ
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 java-client.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 katello-apache.crt
-rw-r–r--. root foreman system_u:object_r:cert_t:s0
katello-default-ca.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
katello-default-ca-stripped.crt
-rw-r–r--. root foreman system_u:object_r:cert_t:s0
katello-server-ca.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 pulp-client.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
wellkatello.niwa.local-qpid-broker.crt

[root@wellkatello katello]# ls -lZ
drwxr-xr-x. root foreman system_u:object_r:cert_t:s0 certs
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
gutterball.crt
-r--------. root root unconfined_u:object_r:cert_t:s0
gutterball.key
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 keystore
-r–r-----. root foreman system_u:object_r:cert_t:s0
keystore_password-file
-r–r-----. root foreman system_u:object_r:cert_t:s0
keystore_password-file-gutterball
drwxr-xr-x. root qpidd system_u:object_r:cert_t:s0 nssdb
drwxr-x—. root foreman system_u:object_r:cert_t:s0 private
-rw-r-----. apache apache system_u:object_r:cert_t:s0
qpid_client_striped.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_client.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_client.key
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_server.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_server.key
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
wellkatello.niwa.local-qpid-broker.pfx

I then cancelled all the paused tasks I could (some were stuck in error and
I had to resort to foreman-rake console and commands like
"ForemanTasks::Task.find("06ede169-a616-4d2c-9774-399c5ae10d32").destroy"

to get rid of the failed tasks (followed by a foreman-rake
katello:reindex). Following this, repo syncs have been OK, no errors -
except for one. The repo which refuses to the sync gives errors like the
below

An error occurred during the sync
{"url"=>"http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el7/x86_64/lmms-1.1.3-1.el7.nux.x86_64.rpm", "errors"=>["A connection error occurred"]}
Task canceled

If I use wget it takes what seems like a long time to connect, but does
eventually download the file. Is the error due to pulp having a timeout to
start downloading? I think that might be happening here, although we
haven't had a problem with this repo in the past

John, where would I find katello.yaml to
check? /etc/foreman/plugins/katello.yaml is very different to the one you
link to, so guess that isn't it…same
for /etc/foreman-installer/scenarios.d/katello.yaml

John_Mitsch · July 6, 2016, 1:06pm

Dylan,

My apologies, I should be a bit more
clear. /etc/foreman/plugins/katello.yaml is the correct file, the file I
linked to is an example of that file. You would have to add in the loggers
section. It would look like

:loggers:
:pulp_rest:
:enabled: true

About the pulp timeout, do you see anything from pulp in /var/log/messages
during that sync?

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

···

On Tue, Jul 5, 2016 at 9:24 PM, Dylan Baars wrote:

Hi guys,

I just compared the permissions to my test instance - they are the same.
No selinux errors, and here’s the output of ls -lZ on the certs dir and one
up (/etc/pki/katello)

[root@wellkatello certs]# ls -lZ
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 java-client.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
katello-apache.crt
-rw-r–r--. root foreman system_u:object_r:cert_t:s0
katello-default-ca.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
katello-default-ca-stripped.crt
-rw-r–r--. root foreman system_u:object_r:cert_t:s0
katello-server-ca.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 pulp-client.crt
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
wellkatello.niwa.local-qpid-broker.crt

[root@wellkatello katello]# ls -lZ
drwxr-xr-x. root foreman system_u:object_r:cert_t:s0 certs
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
gutterball.crt
-r--------. root root unconfined_u:object_r:cert_t:s0
gutterball.key
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0 keystore
-r–r-----. root foreman system_u:object_r:cert_t:s0
keystore_password-file
-r–r-----. root foreman system_u:object_r:cert_t:s0
keystore_password-file-gutterball
drwxr-xr-x. root qpidd system_u:object_r:cert_t:s0 nssdb
drwxr-x—. root foreman system_u:object_r:cert_t:s0 private
-rw-r-----. apache apache system_u:object_r:cert_t:s0
qpid_client_striped.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_client.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_client.key
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_server.crt
-rw-r-----. qdrouterd root system_u:object_r:cert_t:s0
qpid_router_server.key
-rw-r–r--. root root unconfined_u:object_r:cert_t:s0
wellkatello.niwa.local-qpid-broker.pfx

I then cancelled all the paused tasks I could (some were stuck in error
and I had to resort to foreman-rake console and commands like
"ForemanTasks::Task.find(“06ede169-a616-4d2c-9774-399c5ae10d32”).destroy"

to get rid of the failed tasks (followed by a foreman-rake
katello:reindex). Following this, repo syncs have been OK, no errors -
except for one. The repo which refuses to the sync gives errors like the
below

An error occurred during the sync
{“url”=>“http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el7/x86_64/lmms-1.1.3-1.el7.nux.x86_64.rpm”, “errors”=>[“A connection error occurred”]}
Task canceled

If I use wget it takes what seems like a long time to connect, but does
eventually download the file. Is the error due to pulp having a timeout to
start downloading? I think that might be happening here, although we
haven’t had a problem with this repo in the past

John, where would I find katello.yaml to
check? /etc/foreman/plugins/katello.yaml is very different to the one you
link to, so guess that isn’t it…same
for /etc/foreman-installer/scenarios.d/katello.yaml

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Dylan_Baars · July 6, 2016, 10:03pm

Hi John,

re: the timeout - this is what appears in /var/log/messages

Jul 7 10:00:07 wellkatello.niwa.local pulp: kombu.transport.qpid:INFO:
Connected to qpid with SASL mechanism ANONYMOUS
Jul 7 10:00:08 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._queue_reserved_task[79052feb-3d58-43ed-a8b4-40071f04b435]
Jul 7 10:00:08 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.managers.repo.sync.sync[72dba5e6-9ab9-4f96-b5d3-224d73fcb37d]
Jul 7 10:00:08 wellkatello.niwa.local pulp: celery.worker.strategy:INFO:
Received task:
pulp.server.async.tasks._release_resource[ab1e3f1a-a26f-4d12-86fe-fda4e3091566]
Jul 7 10:00:08 wellkatello.niwa.local pulp: celery.worker.job:INFO: Task
pulp.server.async.tasks._queue_reserved_task[79052feb-3d58-43ed-a8b4-40071f04b435]
succeeded in 0.0371223719994s: None
Jul 7 10:00:08 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el7/x86_64/.
Jul 7 10:00:08 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(1): mirror.li.nux.ro
Jul 7 10:00:10 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Jul 7 10:00:10 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el7/x86_64/.
Jul 7 10:00:10 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(1): mirror.li.nux.ro
Jul 7 10:00:11 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Jul 7 10:00:11 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata from
http://mirror.li.nux.ro/li.nux.ro/nux/dextop/el7/x86_64/.
Jul 7 10:00:11 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(1): mirror.li.nux.ro
Jul 7 10:00:12 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Parsing metadata.
Jul 7 10:00:13 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading metadata files.
Jul 7 10:00:13 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Resetting dropped
connection: mirror.li.nux.ro
Jul 7 10:00:13 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(2): mirror.li.nux.ro
Jul 7 10:00:13 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(3): mirror.li.nux.ro
Jul 7 10:00:34 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Generating metadata databases.
Jul 7 10:00:59 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Determining which units need to
be downloaded.
Jul 7 10:01:49 wellkatello.niwa.local pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 99 RPMs.
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392)
/usr/lib/python2.7/site-packages/pulp/server/managers/content/catalog.py:107:
DeprecationWarning: remove is deprecated. Use delete_one or delete_many
instead.
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392) result = collection.remove(query)
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392)
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392) /usr/lib/python2.7/site-packages/mongoengine/document.py:324:
DeprecationWarning: save is deprecated. Use insert_one or replace_one
instead
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392) object_id = collection.save(doc, **write_concern)
Jul 7 10:01:49 wellkatello.niwa.local pulp: py.warnings:WARNING:
(32562-27392)
Jul 7 10:01:49 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(1): mirror.li.nux.ro
Jul 7 10:01:49 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(2): mirror.li.nux.ro
Jul 7 10:01:50 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(3): mirror.li.nux.ro
Jul 7 10:01:50 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(4): mirror.li.nux.ro
Jul 7 10:01:50 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(5): mirror.li.nux.ro
Jul 7 10:02:16 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4,
connect=5, read=4, redirect=None)) after connection broken by
'ReadTimeoutError("HTTPConnectionPool(host='mirror.li.nux.ro', port=80):
Read timed out. (read timeout=27)",)':
/li.nux.ro/nux/dextop/el7/x86_64/xscreensaver-debuginfo-5.29-1.el7.nux.x86_64.rpm
Jul 7 10:02:16 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(6): mirror.li.nux.ro
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4,
connect=5, read=4, redirect=None)) after connection broken by
'ReadTimeoutError("HTTPConnectionPool(host='mirror.li.nux.ro', port=80):
Read timed out. (read timeout=27)",)':
/li.nux.ro/nux/dextop/el7/x86_64/python3-libs-3.3.2-12.el7.nux.x86_64.rpm
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(7): mirror.li.nux.ro
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4,
connect=5, read=4, redirect=None)) after connection broken by
'ReadTimeoutError("HTTPConnectionPool(host='mirror.li.nux.ro', port=80):
Read timed out. (read timeout=27)",)':
/li.nux.ro/nux/dextop/el7/x86_64/lmms-1.1.3-1.el7.nux.x86_64.rpm
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(8): mirror.li.nux.ro
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4,
connect=5, read=4, redirect=None)) after connection broken by
'ReadTimeoutError("HTTPConnectionPool(host='mirror.li.nux.ro', port=80):
Read timed out. (read timeout=27)",)':
/li.nux.ro/nux/dextop/el7/x86_64/vlc-core-2.1.4-5.el7.nux.x86_64.rpm
Jul 7 10:02:17 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(9): mirror.li.nux.ro
Jul 7 10:02:21 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=4,
connect=5, read=4, redirect=None)) after connection broken by
'ReadTimeoutError("HTTPConnectionPool(host='mirror.li.nux.ro', port=80):
Read timed out. (read timeout=27)",)':
/li.nux.ro/nux/dextop/el7/x86_64/gnome-games-3.4.2-2.el7.nux.x86_64.rpm
Jul 7 10:02:21 wellkatello.niwa.local pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection
(10): mirror.li.nux.ro

The ReadTimeoutError repeats for every RPM it attempts

···

On Thursday, July 7, 2016 at 1:06:30 AM UTC+12, John Mitsch wrote: > > > About the pulp timeout, do you see anything from pulp in /var/log/messages > during that sync? > >

Dylan_Baars · July 6, 2016, 11:01pm

And the after it gets through all the timeouts, the end of the process is: