Cert errors katello/foreman

Support
1

Problem:
My apologies as I am a real novice with this, but have years of unix/linux experience. Unfortunately, never dealt with certs.
I am new to katello/Foreman. I am unable to update most not all servers with error:

Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host?

Forman/katello is installed on CentOS Linux release 7.6.1810 (Core)

This installation is only used to patch/update linux servers, centOS6/7, redhat6/7

Expected outcome:
Any help would be great to enable me to update linux systems and fix cert errors. secops is on my case…
Foreman and Proxy versions:
Foreman version: 1.16.2

Active features
Templates, Pulp, TFTP, Puppet, Puppet CA, and Logs

Foreman and Proxy plugin versions:
This is what is installed:
[root@ pub]# rpm -qa|grep katello
katello-service-3.5.2-1.el7.noarch
katello-repos-3.5.2-1.el7.noarch
foreman-installer-katello-3.5.2-1.el7.noarch
katello-certs-tools-2.4.0-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.11.5-1.el7.noarch
katello-server-ca-1.0-1.noarch
katello-debug-3.5.2-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-installer-base-3.5.2-1.el7.noarch
katello-3.5.2-1.el7.noarch
tfm-rubygem-katello-3.5.2-1.el7.noarch
katello-common-3.5.2-1.el7.noarch
katello-selinux-3.0.3-1.el7.noarch

[root@ pub]# rpm -qa|grep foreman
foreman-selinux-1.16.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.1.3-5.el7.noarch
foreman-proxy-1.16.2-1.el7.noarch
foreman-release-scl-3-1.el7.noarch
foreman-installer-katello-3.5.2-1.el7.noarch
foreman-debug-1.16.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.4-2.el7.noarch
tfm-rubygem-foreman_docker-3.2.1-1.fm1_16.el7.noarch
kopas0201.vertexinc.com-foreman-proxy-client-1.0-1.noarch
foreman-installer-1.16.2-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.12-1.fm1_16.el7.noarch
tfm-rubygem-foreman_virt_who_configure-0.1.1-1.fm1_16.el7.noarch
tfm-rubygem-foreman-tasks-0.10.9-1.fm1_16.el7.noarch
foreman-compute-1.16.2-1.el7.noarch
foreman-1.16.2-1.el7.noarch
foreman-cli-1.16.2-1.el7.noarch
kopas0201.vertexinc.com-foreman-proxy-1.0-1.noarch
kopas0201.vertexinc.com-foreman-client-1.0-1.noarch
tfm-rubygem-foreman-tasks-core-0.2.4-1.fm1_16.el7.noarch
tfm-rubygem-hammer_cli_foreman-0.11.0-1.el7.noarch
foreman-release-1.16.2-1.el7.noarch
foreman-postgresql-1.16.2-1.el7.noarch
Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

logs
openssl s_client -connect xmlrpc.rhn.redhat.com:443 -CAfile /usr/share/rhn/RHNS-CA-CERT
140666848540560:error:02001002:system library:fopen:No such file or directory:bss_file.c:175:fopen('/usr/share/rhn/RHNS-CA-CERT','r')
140666848540560:error:2006D080:BIO routines:BIO_new_file:no such file:bss_file.c:182:
140666848540560:error:0B084002:x509 certificate routines:X509_load_cert_crl_file:system lib:by_file.c:258:
CONNECTED(00000003)
-------------------------------------------------------------
Fri Feb 01 03:46:44.870148 2019] [ssl:warn] [pid 47400] [client ,,,.,,,.,,,:34722] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 03:46:44.891653 2019] [ssl:warn] [pid 44290] [client ,,,.,,,.,,,:34730] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 03:46:45.158518 2019] [ssl:warn] [pid 12524] [client ,,,.,,,.,,,:34748] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 03:46:45.174559 2019] [ssl:warn] [pid 47804] [client ,,,.,,,.,,,:34750] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 03:46:46.677107 2019] [ssl:warn] [pid 12524] [client ,,,.,,,.,,,:34788] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 03:46:46.716131 2019] [ssl:warn] [pid 19821] [client ,,,.,,,.,,,:34796] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 07:54:19.972997 2019] [ssl:error] [pid 33400] [client ,,,.,,,.,,,:65305] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:54:19.977806 2019] [ssl:error] [pid 30926] [client ,,,.,,,.,,,:65306] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:54:19.982427 2019] [ssl:error] [pid 23758] [client ,,,.,,,.,,,:65307] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:54:19.986661 2019] [ssl:error] [pid 31095] [client ,,,.,,,.,,,:65297] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:54:19.994719 2019] [ssl:error] [pid 122416] [client ,,,.,,,.,,,:65304] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:54:20.002006 2019] [ssl:error] [pid 13001] [client ,,,.,,,.,,,:65303] AH02261: Re-negotiation handshake failed, referer: https://kopas0201.vertexinc.com/content_hosts/155/errata
[Fri Feb 01 07:55:56.242193 2019] [ssl:warn] [pid 31095] [client ,,,.,,,.,,,:59712] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 07:55:56.260013 2019] [ssl:warn] [pid 36448] [client ,,,.,,,.,,,:59714] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:30:32.409494 2019] [ssl:warn] [pid 37312] [client ,,,.,,,.,,,:58688] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:30:32.425444 2019] [ssl:warn] [pid 13448] [client ,,,.,,,.,,,:58692] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:42:25.508795 2019] [ssl:warn] [pid 23758] [client ,,,.,,,.,,,:39720] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:42:25.522736 2019] [ssl:warn] [pid 30926] [client ,,,.,,,.,,,:39722] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:44:12.168180 2019] [ssl:warn] [pid 60186] [client ,,,.,,,.,,,:41188] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 08:44:12.185239 2019] [ssl:warn] [pid 33400] [client ,,,.,,,.,,,:41190] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 10:11:32.961153 2019] [ssl:warn] [pid 103957] [client ,,,.,,,.,,,:53488] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 10:11:32.974662 2019] [ssl:warn] [pid 104002] [client ,,,.,,,.,,,:53490] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 10:59:08.818209 2019] [ssl:warn] [pid 101682] [client ,,,.,,,.,,,:33818] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
[Fri Feb 01 11:02:51.845809 2019] [ssl:warn] [pid 93778] [client ,,,.,,,.,,,:36772] AH02227: Failed to set r->user to 'SSL_CLIENT_S_DN_CN'
--------------------------------------------------------------------------------------------------------------------
[ 2019-02-01 09:08:26.4282 47862/7f562959c700 Pool2/SmartSpawner.h:298 ]: Preloader for /usr/share/foreman started on PID 72776, listening on unix:/tmp/passenger.1.0.47831/generation-1/backends/preloader.1jrxp66
App 73404 stdout:
App 82172 stdout:
App 82172 stderr:  --> No passenger_native_support.so found for current Ruby interpreter.
App 82172 stderr:      This library provides various optimized routines that make
App 82172 stderr:
App 82172 stderr:      Phusion Passenger faster. Please run 'sudo yum install passenger-devel-4.0.53'
App 82172 stderr:      so that Phusion Passenger can compile one on the next run.
App 82172 stderr:  --> Continuing without passenger_native_support.so.
App 82172 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
App 82172 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[ 2019-02-01 09:28:16.5174 47862/7f562c944700 Pool2/SmartSpawner.h:298 ]: Preloader for /usr/share/foreman started on PID 82172, listening on unix:/tmp/passenger.1.0.47831/generation-1/backends/preloader.13ep7cn
App 82817 stdout:
App 88310 stdout:
App 88310 stderr:  --> No passenger_native_support.so found for current Ruby interpreter.
App 88310 stderr:      This library provides various optimized routines that make
App 88310 stderr:      Phusion Passenger faster. Please run 'sudo yum install passenger-devel-4.0.53'
App 88310 stderr:      so that Phusion Passenger can compile one on the next run.
App 88310 stderr:  --> Continuing without passenger_native_support.so.
App 88310 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
App 88310 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[ 2019-02-01 09:40:43.7037 47862/7f56295dd700 Pool2/SmartSpawner.h:298 ]: Preloader for /usr/share/foreman started on PID 88310, listening on unix:/tmp/passenger.1.0.47831/generation-1/backends/preloader.139p84c
App 88931 stdout:
App 103917 stdout:
App 103917 stderr:  --> No passenger_native_support.so found for current Ruby interpreter.
App 103917 stderr:      This library provides various optimized routines that make
App 103917 stderr:
App 103917 stderr:      Phusion Passenger faster. Please run 'sudo yum install passenger-devel-4.0.53'
App 103917 stderr:      so that Phusion Passenger can compile one on the next run.
App 103917 stderr:  --> Continuing without passenger_native_support.so.
App 103917 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
App 103917 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[ 2019-02-01 10:12:36.4764 47862/7f562c985700 Pool2/SmartSpawner.h:298 ]: Preloader for /usr/share/foreman started on PID 103917, listening on unix:/tmp/passenger.1.0.47831/generation-1/backends/preloader.1sf3huz
App 104632 stdout:
App 125492 stdout:
App 125492 stderr:  --> No passenger_native_support.so found for current Ruby interpreter.
App 125492 stderr:      This library provides various optimized routines that make
App 125492 stderr:      Phusion Passenger faster. Please run 'sudo yum install passenger-devel-4.0.53'
App 125492 stderr:      so that Phusion Passenger can compile one on the next run.
App 125492 stderr:  --> Continuing without passenger_native_support.so.
App 125492 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
App 125492 stderr: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
[ 2019-02-01 10:58:14.5756 47862/7f562959c700 Pool2/SmartSpawner.h:298 ]: Preloader for /usr/share/foreman started on PID 125492, listening on unix:/tmp/passenger.1.0.47831/generation-1/backends/preloader.x4bbwm
2

Hi Mark,

This is new territory for me but I’d like to try and help, and I’ll need some more details & clarifications.

This part of the output that you gave:

openssl s_client -connect xmlrpc.rhn.redhat.com:443 -CAfile /usr/share/rhn/RHNS-CA-CERT

Can you tell me more about it - what did you do to trigger it, etc? Reason I am curious is because I can’t imagine that xmlrpc.rhn.redhat.com or that specific cert has anything to do with Foreman and Katello so it could either be the crux of the problem or a total red herring. If that’s a command you ran to troubleshoot - what led you there?

Regarding the problem in general - is this something new? If not, when was the last time this worked and are you aware of any changes on that system that could have introduced your problem?

Lastly, the output says “Is katello-agent installed and goferd running on the Host?” So now I have to ask the obvious question: have you client systems been properly bootstrapped per the documentation: Foreman :: Plugin Manuals ?

3

Hello Jonathon,
Thanks so much for replying. The ```
openssl s_client -connect xmlrpc.rhn.redhat.com:443 -CAfile /usr/share/rhn/RHNS-CA-CERT

This was taken from a google search on the issues I'm having. In my post, I see ssl errors. Also the cert in the URL shows an error.
![image|405x500](upload://qsF6HAToIRDH6YlgwHwVGxOyFvg.png) 
It started about 2 months ago, but I was not the owner then. I am now  the owner of this and need to get it back in order.
It seems even when I load katello-agent, I still cannot get patches or link the errata to the katello hosts or update/patch most systems with the error that the host didn't answer back in 20 seconds.That, I believe is a cert error, but I can't figure out how or what cert to look at.
Thanks a million for looking at this!!! I am in a crunch trying to shift this left to our NOC and am anable until it is working. :open_mouth:
Thanks again for any help,
Mark
4

Jonathon,
Please let me know if you need additional data. At this point, I will do what ever it takes to get this 100% working.

Mark

5

Thanks Mark. I think we can ignore the command you ran with openssl for now. While I’m not immediately sure of what’s wrong I think there are some next steps.

You can choose to verify the current certificates used by Foreman/Katello which might indicate what the current problem is or you can regenerate them across the board. This will involve running the foreman-installer again and installing a new client bootstrap RPM on any content hosts.

I’ve got a PR open for our documentation describing all of this: https://github.com/theforeman/theforeman.org/blob/b430e613dfb5f040f0cd5d00a82d272d5f97b52a/plugins/katello/nightly/advanced/certificates.md You should arrive at running the following command: foreman-installer --scenario katello --certs-update-server --certs-update-all

If you want to verify your existing certs I think the current values can be found at /etc/foreman-installer/scenarios.d/katello-answers.yaml I probably need to add that to the PR!

6

Thanks so much Jonathon,
Your documentation is too good. With my experience in dealing with certs I’m having a hard time trying to decipher the process… :open_mouth:
Is there a chance that you can simplify it for myself and I’m sure others at my level would appreciate it as well.That being said, thanks again for taking the time to help an old solaris guy out in dealing with katello/foreman…

7

I’d definitely like to clarify the docs if I can. They’ve been merged in now so they are here: Foreman :: Plugin Manuals Please let me know what your specific feedback is.

Before we reset all of your certificates let’s see if we can learn more:

Please provide the output of this command from your Katello server: grep SSL_CLIENT_S_DN_CN /etc/httpd/conf.d/ -A3 -B3 -R

Also, was your Katello set up using custom certificates (your orgs certs for example) or was is set up in the default way?

8

Another thing to try is restarting the goferd service on one of the clients that isn’t able to be updated. systemctl restart goferd

If there are any errors in /var/log/messages coming from gofer after doing so please provide them here

9

Jonathon, Thanks so much again for helping me with this. Here is the output of the ssl grep:

/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-### File managed with puppet ###
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-<Location /pulp/api>
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf:  SSLUsername SSL_CLIENT_S_DN_CN
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-</Location>
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-
/etc/httpd/conf.d/05-foreman-ssl.d/pulp.conf-Alias /pub /var/www/html/pub
--
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-</Directory>
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-<Location /users/extlogin>
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019:  SSLUsername SSL_CLIENT_S_DN_CN
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-</Location>
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-
/etc/httpd/conf.d/ORIGINAL-05-foreman-ssl.conf.01292019-
--
/etc/httpd/conf.d/05-foreman-ssl.conf-</Directory>
/etc/httpd/conf.d/05-foreman-ssl.conf-
/etc/httpd/conf.d/05-foreman-ssl.conf-<Location /users/extlogin>
/etc/httpd/conf.d/05-foreman-ssl.conf:  SSLUsername SSL_CLIENT_S_DN_CN
/etc/httpd/conf.d/05-foreman-ssl.conf-</Location>
/etc/httpd/conf.d/05-foreman-ssl.conf-
/etc/httpd/conf.d/05-foreman-ssl.conf-

Looked to restart the daemon as requested, but I cannot find it in the rc scripts or even running on the system
systemctl status|grep goferd
│ │ └─120423 grep --color=auto goferd
As you can see, it is not on this installation, but it was working all along without it… ? Now I’m more confused…LOL I was under the impression that goferd was an essential part of the foreman/katello installation.
To answer your question, “was your Katello set up using custom certificates (your orgs certs for example) or was is set up in the default way?”
One issue regarding my lack of understanding is that I unfortunately was not working here when it was installed, sorry. :frowning:
My messages file has no reference to goferd or katello.
Further info:
grep foreman /var/log/messages

Feb  7 08:00:01 kopas0201 systemd: Started Session 9447 of user foreman.
Feb  7 08:00:01 kopas0201 systemd: Started Session 9448 of user foreman.
Feb  7 08:00:24 kopas0201 systemd: Removed slice User Slice of foreman.
10

Hey Mark,

Yes, if goferd is not installed, then remote actions (via katello-agent) will never work. I’m not sure they ever would have? You had checked the client for goferd, correct? It does NOT need to be running on the katello server, but does need to be running on every client.

Could katello-agent or goferd have been removed recently?

Another possibility is that previously the admin was using Remote Execution to install updates? Its a newer remote-execution framework in foreman, is ssh based, and katello can use it to perform errata installations.

11

Hello Justin,
Thank you for replying. I did the following:
[root@ katello]# rpm -qa|grep gofer
python-gofer-qpid-2.7.6-1.el7.noarch
python-gofer-2.7.6-1.el7.noarch
The issue is that it is not running. Also, is gofer part of the katello agent install? I ask because I’m unable to get the agent to install on some systems. Who ever installed it seemed to not follow the good online documentation.
BTW, I never posted the OS version the server is: CentOS Linux release 7.6.1810 (Core), sorry.

12

At this point, I was thinking on a fresh install on the latest CentOS release, CentOS 7.5.1804.
Any suggestions going forward?

Cheers,
Mark

13

Based on the rpm output it looks like the gofer package itself is missing and that should be a dependency of katello-agent. Is katello-agent itself installed? You probably want to go over the steps here: Foreman :: Plugin Manuals

14

Jonathon,
I have a small window before I am scheduled to turn this over to our monitoring team. At this point I’m thinking a fresh install on CentOS 1810 would serve 2 purposes. 1, to help me better understand the whole process and the other to possibly speed this whole trial and error loop up. Any suggestion or comments would be well appreciated.

Thank you,
Mark

15

My current suggestion is to check out the documentation I linked (Foreman :: Plugin Manuals should be helpful too) and run through the client setup process which wouldn’t require a reinstall of the OS. I’m not sure of the size of your infrastructure but if reinstalling the OS and starting fresh is something you’re interested in taking on (given your small window) I’d have to say “go for it!”

16

Jonathon,

Thanks again for your help and suggestions. I hate to be a pain, but which version of foreman/katello would you suggest loading?

Grateful,
Mark

17

Ah, I misunderstood! For some reason I thought you were referring to refreshing the client OS and not the Katello server itself. In that case I’d recommend installing our latest stable release - Katello 3.10

18

Thanks again Jonathon and Justin. I will post after completion.

Cheers,
Mark

1 Like
19

Hopefully the last “stupid” question. Do I install Foreman or katello first ?
I would guess Foreman since katello is a plugin?

:open_mouth:
Mark

20

Neither! Well, it’s more like both at the same time. You can follow the steps here: https://theforeman.org/plugins/katello/3.10/installation/index.html