Problem:
I’m trying to expose only the WEB UI (port 443) to the internet. For this, i set up, on AWS, an ALB with it’s own valid certificate (*.myexample.myorgdce.net).
The LB forwards puppet.myexample.myorgdce.net HTTPS requests to my foreman/katello instance (puppet.myexample.example.eu.myorg.cloud) on port 443.
The Foreman Web UI displays everything well (puppet hosts and reports…), EXCEPT when i try to hit anything related to Katello:
If i access directly the node, on port 443 , ALL is good.
[root@puppet ~]# curl -vv [https://$(hostname)/katello/api](https://%24%28hostname%29/katello/api) -u admin:$foreman_password
* About to connect() to puppet.myexample.myorgdce.eu.MyOrg.cloud port 443 (#0)
* Trying 10.250.43.115...
* Connected to puppet.myexample.myorgdce.eu.MyOrg.cloud (10.250.43.115) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=puppet.myexample.myorgdce.eu.MyOrg.cloud,OU=BDS,O=MyOrg,ST=confusion,C=FR
* start date: Oct 07 09:37:37 2019 GMT
* expire date: Jan 18 09:37:37 2038 GMT
* common name: puppet.myexample.myorgdce.eu.MyOrg.cloud
* issuer: CN=puppet.myexample.myorgdce.eu.MyOrg.cloud,OU=BDS,O=MyOrg,L=Puteaux,ST=confusion,C=FR
* Server auth using Basic with user 'admin'
> GET /katello/api HTTP/1.1
> Authorization: Basic YWRtaW46Y2hhbmdlbWU=
> User-Agent: curl/7.29.0
> Host: puppet.myexample.myorgdce.eu.MyOrg.cloud
> Accept: */*
* NSS: client certificate not found (nickname not specified)
< HTTP/1.1 200 OK
< Date: Mon, 14 Oct 2019 11:14:56 GMT
< Server: Apache
< Foreman_version: 1.23.0
< Foreman_api_version: 2
< Foreman_current_organization: ; ANY
< Foreman_current_location: ; ANY
< Cache-Control: max-age=0, private, must-revalidate
< X-Request-Id: 65925d80-e9d2-42eb-b55b-29e8cebf2465
< X-Runtime: 0.026091
< Vary: Origin,Accept-Encoding
< Strict-Transport-Security: max-age=631139040; includeSubdomains
< X-Frame-Options: sameorigin
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Download-Options: noopen
< X-Permitted-Cross-Domain-Policies: none
< Content-Security-Policy: default-src 'self'; child-src 'self'; connect-src 'self' ws: wss:; img-src 'self' data: *.gravatar.com; script-src 'unsafe-eval' 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'
< X-Powered-By: Phusion Passenger 4.0.53
< Set-Cookie: _session_id=6d802c02667a07f38de6b19bec7c3cfe; path=/; secure; HttpOnly; SameSite=Lax
< ETag: W/"664bb5e71fe763682396198fa4a26885"
< Status: 200 OK
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=utf-8
<
[{"href":"/katello/api/capsules/","rel":"capsules"},{"href":"/katello/api/activation_keys/","rel":"activation_keys"},{"href":"/katello/api/activation_keys/","rel":"activation_keys"},{"href":"/katello/api/activation_keys/","rel":"activation_keys"},{"href":"/katello/api/activation_keys/","rel":"activation_keys"},[...]{"href":"/katello/api/sync_plans/","rel":"sync_plans"}]
When i try to access through the LB : the request times out. Performing a foreman-tail suggests that foreman is never reached by the request.
Same goes with synch_plans, views, (anything in the content tab) : page is displayed empty. And trying to create an item through the web ui results in a time out, with item not created.
[root@puppet ~]# curl -vv [https://puppet.myorgdce.MyOrgdce.net/katello/api](https://puppet.myorgdce.myorgdce.net/katello/api) -u admin:$foreman_password
* About to connect() to puppet.myorgdce.MyOrgdce.net port 443 (#0)
* Trying 54.71.4.241...
* Connected to puppet.myorgdce.MyOrgdce.net (54.71.4.241) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=*.myorgdce.myorgdce.net
* start date: Oct 14 00:00:00 2019 GMT
* expire date: Nov 14 12:00:00 2020 GMT
* common name: *.myorgdce.MyOrgdce.net
* issuer: CN=Amazon,OU=Server CA 1B,O=Amazon,C=US
* Server auth using Basic with user 'admin'
> GET /katello/api HTTP/1.1
> Authorization: Basic YWRtaW46Y2hhbmdlbWU=
> User-Agent: curl/7.29.0
> Host: puppet.myorgdce.MyOrgdce.net
> Accept: */*
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 504
< server: awselb/2.0
< date: Mon, 14 Oct 2019 11:21:20 GMT
< content-type: text/html
< content-length: 148
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
</body>
</html>
Backend System Status : Stays empty when hit through the LB, but hammer ping on the host is OK
root@puppet ~]# hammer ping
candlepin:
Status: ok
Server Response: Duration: 34ms
candlepin_auth:
Status: ok
Server Response: Duration: 13ms
pulp:
Status: ok
Server Response: Duration: 33ms
pulp_auth:
Status: ok
Server Response: Duration: 27ms
foreman_tasks:
Status: ok
Server Response: Duration: 3ms
What would be the points i need to check in order to cover my bases?
Any prerequisites i need to check to have katello work behind a Loadbalancer?
Expected outcome:
Katello contents should be displayed in the UI whether hitting it through the load balancer (NOK) or directly through the node (OK)
Foreman and Proxy versions:
foreman-proxy-1.23.0-1.el7.noarch
foreman-release-1.23.0-1.el7.noarch
foreman-release-scl-7-2.el7.noarch
Foreman and Proxy plugin versions:
Other relevant data:
#/etc/foreman-installer/scenarios.d/katello-answers.yaml (cert section) :
certs:
params:
node_fqdn: "%{::fqdn}"
cname: ['puppet.myexample.myorgdce.net']
regenerate: false
regenerate_ca: false
generate: true
deploy: true
ca_common_name: "%{::fqdn}"
country: "FR"
state: "confusion"
city: "Puteaux"
org: "MYORG"
org_unit: "BDS"
expiration: "36500"
ca_expiration: "36500"
user: "root"
group: "foreman"
#Foreman installer command:
/usr/sbin/foreman-installer -v \
--scenario katello \
--enable-foreman-compute-ec2 \
--foreman-initial-admin-password=$foreman_password \
--enable-foreman-plugin-ansible \
--enable-foreman-plugin-monitoring \
--enable-foreman-plugin-templates \
--enable-foreman-plugin-remote-execution \
--enable-foreman-plugin-openscap \
--foreman-proxy-dhcp=false \
--foreman-proxy-dns=false \
--foreman-initial-organization=$foreman_default_organization \
--foreman-cors-domains=https://puppet.myexample.myorgdce.net