Pulpcore-workers and postgresql trying to run and failing on proxies that has pulpcore feature disabled

tbrooks · October 9, 2024, 4:05pm

Problem:
pulpcore-workers and postgresql try to start/run on proxies that have enabled set to false

Expected outcome:
These proxies are configured to only have DHCP, DNS, remote_ execution, and a few other services, but not pulpcore. The services we use are working as expected. However, the logs and foreman-maintain health check show:
Check whether all services are running: [FAIL]
Following services are not running: postgresql, pulpcore-api, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@6.service

Foreman and Proxy versions:
Foreman 3.11.2
Katello 4.13.1

Foreman and Proxy plugin versions:

foreman-tasks 9.1.1
foreman_discovery 24.0.1
foreman_remote_execution 13.1.0
katello 4.13.1

Distribution and version:
Red Hat Enterprise Linux release 8.10 (Ootpa)

Other relevant data:

foreman-maintain health check
Running ForemanMaintain::Scenario::FilteredScenario

================================================================================

Check whether all services are running: [FAIL]

Following services are not running: postgresql, pulpcore-api, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@6.service

Continue with step [Restart applicable services]?, [y(yes), n(no)] y
Restart applicable services:

Restarting the following service(s):
postgresql, pulpcore-api, pulpcore-worker@3.service, pulpcore-worker@4.service, pulpcore-worker@6.service
/ restarting pulpcore-api
Job for postgresql.service failed because the control process exited with error code.
See “systemctl status postgresql.service” and “journalctl -xe” for details.

All services restarted
| Server responded successfully! [OK]

Rerunning the check after fix procedure
Check whether all services are running: [FAIL]

Following services are not running: postgresql, pulpcore-api, pulpcore-worker@1.service, pulpcore-worker@5.service

Continue with step [Restart applicable services]?, [y(yes), n(no)]

From /var/log/messages:
Oct 9 10:27:18 netproxy systemd[1]: pulpcore-api.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:18 netproxy systemd[1]: pulpcore-api.service: Failed with result ‘exit-code’.Oct 9 10:27:19 netproxy systemd[1]: pulpcore-worker@5.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:19 netproxy systemd[1]: pulpcore-worker@5.service: Failed with result ‘exit-code’.
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@6.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@6.service: Failed with result ‘exit-code’.
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@2.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@2.service: Failed with result ‘exit-code’.
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@3.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@3.service: Failed with result ‘exit-code’.
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@4.service: Main process exited, code=exited, status=1/FAILURE
Oct 9 10:27:21 netproxy systemd[1]: pulpcore-worker@4.service: Failed with result ‘exit-code’.

From /etc/foreman-proxy/settings.d/pulpcore.yml:
:enabled: false

gvde · October 9, 2024, 4:22pm

Sounds to me as if you have installed a content proxy scenario. If you don’t want content/pulpcore then you should have installed the proxy only using the standard scenario. I don’t know if it’s really possible to undo everything which the content proxy already installed or if you have to reinstall it again on a clean server.

tbrooks · October 9, 2024, 4:45pm

My /etc/foreman-installer/scenarios.d/foreman-proxy-content-answers.yaml has:

foreman_proxy_content: false

tbrooks · October 9, 2024, 4:57pm

I do see it set to “true” in the katello-answers.yaml on the proxy, but I was under the impression that it was reading the foreman-proxy-content-answers.yaml. I’m seeing:

foreman-answers.yaml
foreman-proxy-content-answers.yaml
foreman-proxy-content.yaml
foreman.yaml
katello-answers.yaml
katello.yaml

I’m confused as to which is taking priority. I had always made my changes in foreman-proxy-content-answers.yaml on the proxies, and foreman-proxy-content-answers.yaml on the master server.

gvde · October 9, 2024, 5:07pm

How did you run foreman-installer the very first time? Remember, that foreman-installer remembers previous configurations.
You have used the foreman-proxy-content scenario although you actually don’t want it. You have use the foreman-proxy-content package although you don’t want it. It automatically installs postgres and pulpcore as dependency.
Disabling the foreman-proxy-content module (i.e. foreman_proxy_content: false) just disabled the installer functions to install and configure the foreman_proxy_content. The foreman-installer just doesn’t do anything about it anymore. If you have configured it before disabling the module will just put a blind eye on anything about it already installed. Thus, in that case you would have to enable the module and then use the module options to configure it to disable postgres and pulpcore. However, I doubt this will really work because it’s contradictory to first install foreman-proxy-content and then tell it to actually not do it.

last_scenario.yaml contains the last scenario run. You cannot change the scenario. It’s the same problem as with disabling some module. It just won’t look at some parts anymore but that doesn’t mean those parts get removed or stopped or anything. They will just be ignored.

I would highly recommend to use foreman-installer and only make configurations with those options. It has at least some brief descriptions on each option.

Again: the best way would be to start over on a new server and NOT use the foreman-proxy-content package and scenario. The purpose of the scenario is to install a content proxy and using it to install anything else but a content proxy isn’t really working. It’s not what it’s made for.

tbrooks · October 9, 2024, 5:17pm

Yes, the first install was with "–scenario foreman-proxy-content " We have one proxy in each site that is doing the content/repo and tftp function and the other that handles primarily dhcp and dns functions. So, for the proxy that is only meant to handle dhcp, dns, remote execution. What is the correct scenario?

gvde · October 9, 2024, 5:26pm

The correct scenario is the default scenario (i.e. foreman) which is selected when you don’t specify any scenario.

The officially released docs don’t cover that, yet, however the nightly docs have a manual on installing a smart proxy in a foreman w/o katello environment at Installing a Smart Proxy Server nightly on Enterprise Linux

The guide starts with using the proxy as puppetserver proxy, but I guess it should work accordingly if you want something else and no puppetserver.

But again, this requires to start over on a new server.

tbrooks · October 9, 2024, 5:28pm

Thanks for your help. It works as is with the errors, but I’ll make plans for doing a fresh install.

gvde · October 9, 2024, 5:37pm

Well, then you could enable the foreman-proxy-content module again and check all the foreman-installer module options with --full-help and see if you can disable at least some of the functionality. foreman-proxy-content-pulpcore-mirror looks promising as does foreman-proxy-content-pulpcore-manage-postgresql. That may allow you to at least not run some of those services. But I never tried that so you have to test for yourself.

Otherwise, just let it run pulpcore and postgresql as it wants to. As long as you don’t assign any environments to the proxy it has no content to serve. But you have no errors and you can run foreman-installer again later without issues.