[Katello 2.3] Client built with Katello receives 400 error trying to retrieve puppet yml

On Katello 2.3 server on CentOS 7, I have built a client using the Katello
default kickstart that registers to FreeIPA, installs and subscribes with
subscription manager, and registers puppet.

The server built was CentOS 6.5 with puppet 3.7.5.

In the Katello hosts list, the katello host is reporting in every 30
minutes properly. The new host is reporting with 3 errors.

Could not retrieve catalog from remote server: Error 400 on SERVER: Could
not find default node or by name with 'testhost12.mydomain.net,
testhost12.mydomain, testhost12' on node testhost12.mydomain.net
Using cached catalog
Could not retrieve catalog; skipping run

When I run puppet agent --test on the client, this is what I see in the
apache log

10.178.0.110 - - [24/Dec/2015:12:19:41 -0800] "GET
/node/testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Ruby"
10.178.0.110 - - [24/Dec/2015:12:19:44 -0800] "POST /api/hosts/facts
HTTP/1.1" 201 1030 "-" "Ruby"
10.178.0.110 - - [24/Dec/2015:12:19:44 -0800] "GET
/node/testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Ruby"
10.178.0.110 - - [24/Dec/2015:12:19:45 -0800] "POST /api/reports HTTP/1.1"
201 237 "-" "Ruby"

This is what the client shows at the command line

Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Could not find default node or by name with 'testhost12.mydomain.net,
testhost12.mydomain, testhost12' on node testhost12.mydomain.net
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

If I try retrieving that URL from my web browser I get the following apache
log entry :

10.8.134.155 - - [24/Dec/2015:12:10:12 -0800] "GET
/node/testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Mozilla/5.0
(Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0"

and my web browser shows :

··· --- classes: base: base::auditrules: base::coreusers: base::crond: base::default-directories: base::disk-partition: base::etc-hosts: base::etcresolv: base::exec-example: base::gr-domain: base::gr-yum-repos: base::hp: base::iptables: base::limits: base::networkmanager: base::nfs: base::ntp: base::openaudit: base::packages: base::puppet-agent: base::rsyslog: base::selinux: base::snmpd: base::ssh: base::sysctl: base::syslog: base::test: base::vmware-tools: limits: mail: mail::etc-aliases: selinux: selinux::config: selinux::install: selinux::params: stdlib: stdlib::stages: sysctl::base: sysctl::params: parameters: puppetmaster: katello1.mydomain.net domainname: mydomain.net realm: mydomain.net hostgroup: Old CentOS 6.5 Dev Servers location: My Location organization: My Company root_pw: $5$7YhDrE+i$t5i3j5/ItfFWHgutbd63XOdbsEjP.SxRO8lCMCeJOG7 puppet_ca: katello1.mydomain.net foreman_env: development owner_name: My Name owner_email: my_email@mydomain.net foreman_subnets: - network: 10.21.0.0 mask: 255.255.0.0 name: Dev Network vlanid: '' gateway: 10.21.0.1 dns_primary: 10.21.9.99 dns_secondary: 10.178.0.99 from: '' to: '' boot_mode: Static ipam: None foreman_interfaces: - mac: 00:50:56:b7:58:8f ip: 10.21.200.11 type: Interface name: testhost12.mydomain.net attrs: network: 10.21.0.0 mtu: '1500' netmask: 255.255.0.0 virtual: false link: true identifier: eth0 managed: true primary: true provision: true subnet: network: 10.21.0.0 mask: 255.255.0.0 name: Gastown Dev Network vlanid: '' gateway: 10.21.0.1 dns_primary: 10.21.9.99 dns_secondary: 10.178.0.99 from: '' to: '' boot_mode: Static ipam: None kt_activation_keys: Old CentOS 6.5 Dev kt_env: Old_Dev_CentOS_6_5_Clients kt_cv: Old_CentOS_6_5_View lifecycle_environment: Old_Dev_CentOS_6_5_Clients content_view: Old_CentOS_6_5_View environment: development

If I attempt to retrieve via curl on the client I get the following apache
entry :

10.21.200.11 - - [24/Dec/2015:12:09:59 -0800] “GET
/node/testhost12.mydomain.net?format=yml HTTP/1.1” 403 1 “-” “curl/7.19.7
(x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3
libidn/1.18 libssh2/1.4.2”

and the client shows on the command line :

[root@testhost12 puppet]# curl -k -v
https://katello1.mydomain.net/node/testhost12.mydomain.net?format=yml

  • About to connect() to katello1.mydomain.net port 443 (#0)
  • Trying 10.178.0.110… connected
  • Connected to katello1.mydomain.net (10.178.0.110) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • warning: ignoring value of ssl.verifyhost
  • skipping SSL peer certificate verification
  • NSS: client certificate not found (nickname not specified)
  • SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  •   subject: CN=katello1.mydomain.net,OU=SomeOrgUnit,O=Katello,ST=North 
    

Carolina,C=US

  •   start date: Dec 08 18:23:48 2015 GMT
    
  •   expire date: Dec 10 18:23:48 2035 GMT
    
  •   common name: katello1.mydomain.net
    
  •   issuer: 
    

CN=katello1.mydomain.net,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North
Carolina,C=US

GET /node/testhost12.mydomain.net?format=yml HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7
NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: katello1.mydomain.net
Accept: /

< HTTP/1.1 403 Forbidden
< Date: Thu, 24 Dec 2015 20:19:28 GMT
< Server: Apache/2.4.6 (CentOS)
< Strict-Transport-Security: max-age=631152000; includeSubdomains
< X-Frame-Options: SAMEORIGIN
< Content-Security-Policy: default-src ‘self’; connect-src ‘self’ ws: wss:;
font-src ‘self’; frame-src ‘self’; img-src ‘self’ *.gravatar.com data:;
media-src ‘self’; object-src ‘self’; script-src ‘unsafe-eval’
‘unsafe-inline’ ‘self’; style-src ‘unsafe-inline’ ‘self’;
< X-XSS-Protection: 1; mode=block
< X-Content-Type-Options: nosniff
< X-Download-Options: noopen
< X-UA-Compatible: IE=Edge,chrome=1
< Cache-Control: no-cache
< X-Request-Id: 91c073aba17766f1d3fcf228e10097cc
< X-Runtime: 0.007684
< X-Rack-Cache: miss
< X-Powered-By: Phusion Passenger 4.0.18
< Set-Cookie:
_session_id=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTFlNmY0NTgzNjgyZDUzMTZjYzc2ZTQzNzAxMjk2Nzg4BjsAVEkiC2xvY2FsZQY7AEZJIgdlbgY7AEZJIhFvcmlnaW5hbF91cmkGOwBGIjQvbm9kZS90ZXN0aG9zdDEyLmRldi1nbG9iYWxyZWxheS5uZXQ%2FZm9ybWF0PXltbA%3D%3D–83f50905bdfbae9632ca50d32c24f9d2a44d7b07;
path=/; HttpOnly
< Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
< Location: https://katello1.mydomain.net/users/login
< Status: 403 Forbidden
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: text/plain; charset=utf-8
<

  • Closing connection #0

So I have a couple of questions :

1)Why would my web browser get a yml file back but my curl request gets a
403 forbidden?
2)If Katello managed to successfully install puppet, register the client,
sign its certificate (I checked and it’s valid from 1 day ago), why would
it be refusing the agent access?
3)Is there anything I can do to fix this? Ensure that future clients I
build will be able to register properly?

It turns out that if you have any hosts manually defined in your site.pp
Katello or puppet totally ignores its ENC capabilities. Removing all hosts
from the site.pp allowed the server to once again recognize it was an ENC
and gather facts from its database instead. The run was then successful.

··· On Thursday, December 24, 2015 at 12:36:07 PM UTC-8, Nathan Peters wrote: > > On Katello 2.3 server on CentOS 7, I have built a client using the > Katello default kickstart that registers to FreeIPA, installs and > subscribes with subscription manager, and registers puppet. > > The server built was CentOS 6.5 with puppet 3.7.5. > > In the Katello hosts list, the katello host is reporting in every 30 > minutes properly. The new host is reporting with 3 errors. > > Could not retrieve catalog from remote server: Error 400 on SERVER: Could > not find default node or by name with 'testhost12.mydomain.net, > testhost12.mydomain, testhost12' on node testhost12.mydomain.net > Using cached catalog > Could not retrieve catalog; skipping run > > When I run puppet agent --test on the client, this is what I see in the > apache log > > 10.178.0.110 - - [24/Dec/2015:12:19:41 -0800] "GET /node/ > testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Ruby" > 10.178.0.110 - - [24/Dec/2015:12:19:44 -0800] "POST /api/hosts/facts > HTTP/1.1" 201 1030 "-" "Ruby" > 10.178.0.110 - - [24/Dec/2015:12:19:44 -0800] "GET /node/ > testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Ruby" > 10.178.0.110 - - [24/Dec/2015:12:19:45 -0800] "POST /api/reports HTTP/1.1" > 201 237 "-" "Ruby" > > This is what the client shows at the command line > > Info: Retrieving pluginfacts > Info: Retrieving plugin > Info: Loading facts > Error: Could not retrieve catalog from remote server: Error 400 on SERVER: > Could not find default node or by name with 'testhost12.mydomain.net, > testhost12.mydomain, testhost12' on node testhost12.mydomain.net > Warning: Not using cache on failed catalog > Error: Could not retrieve catalog; skipping run > > If I try retrieving that URL from my web browser I get the following > apache log entry : > > 10.8.134.155 - - [24/Dec/2015:12:10:12 -0800] "GET /node/ > testhost12.mydomain.net?format=yml HTTP/1.1" 200 889 "-" "Mozilla/5.0 > (Windows NT 6.1; WOW64; rv:43.0) Gecko/20100101 Firefox/43.0" > > and my web browser shows : > > --- > classes: > base: > base::auditrules: > base::coreusers: > base::crond: > base::default-directories: > base::disk-partition: > base::etc-hosts: > base::etcresolv: > base::exec-example: > base::gr-domain: > base::gr-yum-repos: > base::hp: > base::iptables: > base::limits: > base::networkmanager: > base::nfs: > base::ntp: > base::openaudit: > base::packages: > base::puppet-agent: > base::rsyslog: > base::selinux: > base::snmpd: > base::ssh: > base::sysctl: > base::syslog: > base::test: > base::vmware-tools: > limits: > mail: > mail::etc-aliases: > selinux: > selinux::config: > selinux::install: > selinux::params: > stdlib: > stdlib::stages: > sysctl::base: > sysctl::params: > parameters: > puppetmaster: katello1.mydomain.net > domainname: mydomain.net > realm: mydomain.net > hostgroup: Old CentOS 6.5 Dev Servers > location: My Location > organization: My Company > root_pw: $5$7YhDrE+i$t5i3j5/ItfFWHgutbd63XOdbsEjP.SxRO8lCMCeJOG7 > puppet_ca: katello1.mydomain.net > foreman_env: development > owner_name: My Name > owner_email: my_email@mydomain.net > foreman_subnets: > - network: 10.21.0.0 > mask: 255.255.0.0 > name: Dev Network > vlanid: '' > gateway: 10.21.0.1 > dns_primary: 10.21.9.99 > dns_secondary: 10.178.0.99 > from: '' > to: '' > boot_mode: Static > ipam: None > foreman_interfaces: > - mac: 00:50:56:b7:58:8f > ip: 10.21.200.11 > type: Interface > name: testhost12.mydomain.net > attrs: > network: 10.21.0.0 > mtu: '1500' > netmask: 255.255.0.0 > virtual: false > link: true > identifier: eth0 > managed: true > primary: true > provision: true > subnet: > network: 10.21.0.0 > mask: 255.255.0.0 > name: Gastown Dev Network > vlanid: '' > gateway: 10.21.0.1 > dns_primary: 10.21.9.99 > dns_secondary: 10.178.0.99 > from: '' > to: '' > boot_mode: Static > ipam: None > kt_activation_keys: Old CentOS 6.5 Dev > kt_env: Old_Dev_CentOS_6_5_Clients > kt_cv: Old_CentOS_6_5_View > lifecycle_environment: Old_Dev_CentOS_6_5_Clients > content_view: Old_CentOS_6_5_View > environment: development > > If I attempt to retrieve via curl on the client I get the following apache > entry : > > 10.21.200.11 - - [24/Dec/2015:12:09:59 -0800] "GET /node/ > testhost12.mydomain.net?format=yml HTTP/1.1" 403 1 "-" "curl/7.19.7 > (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 > libidn/1.18 libssh2/1.4.2" > > and the client shows on the command line : > > [root@testhost12 puppet]# curl -k -v > https://katello1.mydomain.net/node/testhost12.mydomain.net?format=yml > * About to connect() to katello1.mydomain.net port 443 (#0) > * Trying 10.178.0.110... connected > * Connected to katello1.mydomain.net (10.178.0.110) port 443 (#0) > * Initializing NSS with certpath: sql:/etc/pki/nssdb > * warning: ignoring value of ssl.verifyhost > * skipping SSL peer certificate verification > * NSS: client certificate not found (nickname not specified) > * SSL connection using TLS_DHE_RSA_WITH_AES_256_CBC_SHA > * Server certificate: > * subject: CN=katello1.mydomain.net,OU=SomeOrgUnit,O=Katello,ST=North > Carolina,C=US > * start date: Dec 08 18:23:48 2015 GMT > * expire date: Dec 10 18:23:48 2035 GMT > * common name: katello1.mydomain.net > * issuer: CN=katello1.mydomain.net,OU=SomeOrgUnit,O=Katello,L=Raleigh,ST=North > Carolina,C=US > > GET /node/testhost12.mydomain.net?format=yml HTTP/1.1 > > User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/ > 3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2 > > Host: katello1.mydomain.net > > Accept: */* > > > < HTTP/1.1 403 Forbidden > < Date: Thu, 24 Dec 2015 20:19:28 GMT > < Server: Apache/2.4.6 (CentOS) > < Strict-Transport-Security: max-age=631152000; includeSubdomains > < X-Frame-Options: SAMEORIGIN > < Content-Security-Policy: default-src 'self'; connect-src 'self' ws: > wss:; font-src 'self'; frame-src 'self'; img-src 'self' *.gravatar.com > data:; media-src 'self'; object-src 'self'; script-src 'unsafe-eval' > 'unsafe-inline' 'self'; style-src 'unsafe-inline' 'self'; > < X-XSS-Protection: 1; mode=block > < X-Content-Type-Options: nosniff > < X-Download-Options: noopen > < X-UA-Compatible: IE=Edge,chrome=1 > < Cache-Control: no-cache > < X-Request-Id: 91c073aba17766f1d3fcf228e10097cc > < X-Runtime: 0.007684 > < X-Rack-Cache: miss > < X-Powered-By: Phusion Passenger 4.0.18 > < Set-Cookie: > _session_id=BAh7CEkiD3Nlc3Npb25faWQGOgZFRkkiJTFlNmY0NTgzNjgyZDUzMTZjYzc2ZTQzNzAxMjk2Nzg4BjsAVEkiC2xvY2FsZQY7AEZJIgdlbgY7AEZJIhFvcmlnaW5hbF91cmkGOwBGIjQvbm9kZS90ZXN0aG9zdDEyLmRldi1nbG9iYWxyZWxheS5uZXQ%2FZm9ybWF0PXltbA%3D%3D--83f50905bdfbae9632ca50d32c24f9d2a44d7b07; > path=/; HttpOnly > < Set-Cookie: request_method=; path=/; expires=Thu, 01-Jan-1970 00:00:00 > GMT > < Location: https://katello1.mydomain.net/users/login > < Status: 403 Forbidden > < Connection: close > < Transfer-Encoding: chunked > < Content-Type: text/plain; charset=utf-8 > < > * Closing connection #0 > > So I have a couple of questions : > > 1)Why would my web browser get a yml file back but my curl request gets a > 403 forbidden? > 2)If Katello managed to successfully install puppet, register the client, > sign its certificate (I checked and it's valid from 1 day ago), why would > it be refusing the agent access? > 3)Is there anything I can do to fix this? Ensure that future clients I > build will be able to register properly? > >