Problem: Recently upgrade our Foreman/Katello and the external Smart Proxy to v3.9 and since the upgrade, all the hosts which are reporting to puppet server on the smart proxy fail to report the configuration, even though the check-in is fine.
Expected outcome: Similar to host reporting to Foreman server, all the hosts should report to Smart Proxy like before the upgrade
Foreman and Proxy versions: 3.9.1
Foreman and Proxy plugin versions: foreman-tasks v9.0.1, format_puppet v6.1.1, forman_remote_execution v12.0.4, katello v4.11
Distribution and version: RHEL 8.9
Other relevant data:
Puppet Server Logs on Smart Proxy, when running puppet agent on the host, shows this error which might be related:
==> /var/log/puppetlabs/puppetserver/puppetserver.log <==
2024-01-24T11:09:27.742-08:00 WARN [qtp1301971413-211] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: <HOSTNAME>: During the fact upload the server responded with: 404 Not Found. Error is ignored and the execution continues.
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
Serving cached ENC: Error retrieving node <HOSTNAME>: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.
Checked /var/log/foreman/production.log on the Foreman as suggested but it doesn’t show any error during the client check-in.
This seems like a wrong url configured in /etc/puppetlabs/puppet/foreman.yaml can you check? I don’t think anything has changed on the Foreman side at this area recently.
1 Like
Thanks for your reply, @Marek_Hulan
The configuration in /etc/puppetlabs/puppet/foreman.yaml looks fine to me and the url is accessible.
Regarding your comment about changes, just FYI, the actual upgrade was from v.3.2 > v3.9 which we had to run in multi steps.
Just some additional notes about the upgrading procedure which we’ve taken to upgrade from foreman v3.2 (katello v4.4) to foreman v3.9 (katello v4.11), as advised by the documentation, the upgrade was sequential as follows:
- Upgrade foreman server using
foreman-installer
- Generate smart proxy cert using
foreman-proxy-certs-generate
- Upgrade smart proxy to the same version using generated cert in last step and run
foreman-installer --scenario foreman-proxy-content --certs-tar-file ....
One other issue which we’ve been facing on those hosts reporting to smart proxy, is ignoring some puppet parameters for the host and treat it as empty string. Which should because of the problem with reporting the config:
___UPDATE___
About Foreman/Katello/SmartProxy/Puppet setup;
- There’s one foreman server which runs Katello and PuppetServer at Location1 and include all the contents, puppet parameters/config, etc
- Local client Hosts at Location1 pull the content from foreman server
- Local client Hosts at Location1 run puppet agent are hooked up to Puppet Master on foreman server
- There’s one Smart Proxy server which runs foreman-proxy and PuppetServer at Location2 and sync up the content from foreman server
- Local client Hosts at Location2 pull the content from Smart Proxy server
- Local client Hosts at Location2 run puppet agent are hooked up to Puppet Master running on Smart Proxy and pull the puppet parameters/groups/etc from foreman server
As @Marek_Hulan suggested, looks like during the upgrade, the :url: config in /etc/puppetlabs/puppet/foreman.yaml on Smart Proxy had been changed to Local Puppet Server (Smart Proxy) instead of foreman server. After updating the :url: the HTTPNotFound is no longer showing up and client Hosts at Location2 can now report the config to foreman and pull the puppet parameters as expected.
I’m just wondering if any extra parameter(s) needs to be passed during either of these steps when installing/upgrading/configuring Smart Proxies?
- Generating certificate tarball on foreman server:
foreman-proxy-certs-generate \
--foreman-proxy-fqdn smartproxy.example.com \
--certs-tar /root/smart-proxy_cert/smartproxy.example.com-certs.tar
- Running returned foreman-installer cmd on Smart Proxy:
foreman-installer --scenario foreman-proxy-content \
--certs-tar-file "/root/smart-proxy_cert/smartproxy.example.com-certs.tar" \
--foreman-proxy-register-in-foreman "true" \
--foreman-proxy-foreman-base-url "https://foreman.example.com" \
--foreman-proxy-trusted-hosts "foreman.example.com" \
--foreman-proxy-trusted-hosts "smartproxy.example.com" \
--foreman-proxy-oauth-consumer-key "<OAUTH_CONSUMER_KEY>" \
--foreman-proxy-oauth-consumer-secret "<OAUTH_CONSUMER_SECRET>"
Please advise
