How to safely stop/start Foreman/Puppet services

Problem:
All the Foreman/Puppet services are on the same server. Suppose your server needs maintenance (so you need to stop the services) or there is an power outage (then you may need to restart services).

Are the following services a complete list in this case? Is there any sequence to stop/start them?

Here are all the services I can think of:

foreman-proxy
apache2
postgresql
puppetdb
puppetserver
puppet

One thing I learned is that you should not run “foreman-installer” again: it will overwrite your previous configurations.

A question related to this:

I implemented r10k and it worked fine for almost one year. Recently I found out that Foreman UI is not updating the new classes I added in. I need to stop puppetdb/puppetserver/puppet on the server and restart them. It worked for a few days, then I need to stop/start again.

I don’t know where to look at the problem. r10k works fine. The code was correctly put into the code directory on the server. It’s just it seems Foreman UI could not sense the new code there. I did “Import environments from ***” on Foreman UI but it always said “No changes to your environments detected”. So I had to stop/start puppetdb/puppetserver/puppet.

Admitted the system has been up ~250 days. I don’t want to reboot the server without knowing for sure how to bring it back safely online.

Expected outcome:

Foreman and Proxy versions:
Foreman 1.20.2
Foreman and Proxy plugin versions:
Foreman 1.20.2
Distribution and version:

Server: Ubuntu 18.04.2 LTS
Foreman 1.20.2
Puppetserver: 5.3.8
Puppet: 5.5.14
Puppetdb: 5.2.9
Postgresql: 9.6.14

Other relevant data:

1 Like

Hi,

the list you posted looks complete to me. I only have a Katello setup at hand, though, so I do have some additional services that should be specific to Katello.
If you are using remote execution plugin, there should also be smart_proxy_dynflow_core service and dynflowd service.
The “puppet” service is not directly related to Foreman, that is only the Puppet agent and not related to any serverside things.
On EL based systems, there is foreman-maintain for exactly this task. From what I’ve found, it is sadly still not supported on DEB based systems, though :frowning:

As for why your Foreman is not able to pick up new code after r10k deployments, that is hard to say. We are running r10k with Puppet for years without problems, but we do restart our Foreman/Katello stack usually once per day for backups.
The way new code is beeing picked up at import is as follows:
Foreman → foreman-proxy → puppetserver API
Maybe you can find any usefull information on this in foreman-proxy and puppetserver logs. Raising the loglevel might be helpful.

Regards

A note on the side:
If changing values in the config, it is advised to use foreman-installer for this. You can set nearly all the values using the installer itself. That way, they won’t get overridden on the next installer run.
This will save you a huge ton of headache when upgrading to a new version, since you need to run foreman-installer for that.

Let me try to understand your side note: what you mentioned there only applies when you want to do upgrade.

If you are not upgrading, then don’t run installer again, because it will overwrite your original configurations. It will bring up a new Foreman instance. I don’t know what it will do to your original database.

In this case you want to install the package, make configuration changes if needed, and restart service. For example, if you want to enable Remote Execution plugin, you need to install the package, and restart apache2 service.

Is my understanding correct?

Thanks for the information about the services.

Indeed I found there is “dynflowd.service”. Also there is another one as “ruby-foreman-tasks.service”. The contents are the same for both. The “systemctl status” output on both are the same. Both have the same alias name as “ruby-foreman-tasks.service”. They both show as “enabled” at system reboot start. So I’ll consider them as the same (though I don’t understand why two of the same service instead of one).

I also found all of the mentioned service are configured as “enabled” at system reboot start. So I don’t need to worry on this.

No, that was not what I meant. You can use foreman-installer without upgrading. That way, you can set configuration options that are managed by the installer. This is helpful since, in case of an upgrade in the future, you will not need to re-set all the options you changed manually because the installer “knows about them” already.
You can just run foreman-installer --option-xyz foobar and the the installer will set that option in the config files accordingly and “remember” that option for future runs.
If you want to know what an installer run would change (without actually changing anything) you can see that with foreman-installer -v --noop.
For a full list of the installer options, see the manual or foreman-installer --full-help (I think, foreman-installer --help will tell you the correct option).

Sorry for keeping this up: let’s say you have run foreman-installer after you install Foreman. Now you have a working Foreman and you added all the hosts and started managing them. Later you want to enable some features on Foreman. Are you saying you can stop everything then run " foreman-installer --option-xyz foobar" again? What happened to all the hosts that you have added/configured? The last time I did this way I lost those data (they were not on Foreman UI anymore).It even changed the admin credentials. Hence I said “It will bring up a new Foreman instance.”.

The following is an example of what happens if you run “foreman-installer”


ubuntu@hnubut3:/tmp$ sudo foreman-installer

Installing Done [100%] […]

Success!

  • Foreman is running at [https://hnubut3.***.com

    Initial credentials are admin / GLCYvF8YQQaAn52k

  • Foreman Proxy is running at [https://hnubut3.***.com:8443]

  • Puppetmaster is running at port 8140

The full log is at /var/log/foreman-installer/foreman.log


Hi,

Yes, this is what I’m saying, except the installer should also stop the services so you don’t have to yourself. foreman-installer should be idempotent (not doing things that had already be done).
The behaviour you experienced (losing data and resetting admin credentials) is definetly not expected behaviour. Since I don’t know the inner workings of the installer, I can not help you myself with that problem. Might very well be a Ubuntu/Deb specific issue.
I would recommend raising a seperate topic for this to give it more attention since it is not directly related to the initial question of this thread.