How to safely stop Foreman/Puppet services / reboot server

Support
1

Problem:
Hi all,

I read in some topic that after restart/reboot services/server foreman lost data! I take also some questions form other topic to create one and for all… Maybe this will help all foreman users.

It happens to me also on Test & Production Server!

This has many side effects, its annoying, it take to much time to recover the previous stand … to find an answer is also not so easy!

Are the following services a complete list in this case? Is there any sequence to stop/start them?

Here are all the services I can think of:

foreman
foreman.socket
foreman-proxy
apache2
postgresql
puppetdb
puppetserver
puppet

One thing I learned is that you should not run “foreman-installer” again: it will overwrite your previous configurations.

What I found and do until now is to create a backup like here:
https://theforeman.org/manuals3.5/index.html#5.5Backup,RecoveryandMigration

I also create a backup of all /etc/puppetlabs/puppet* and /etc/foreman* .

If you have any comments or suggestions, please, let us know…

2

I don’t know the topic you are referring to. But for me, all my foreman/katello/puppet servers and proxies work just fine and stop properly during normal shutdown. I have no lost data…

You can always use foreman-maintain service stop to stop all services.

This is completely wrong. Running foreman-installer maintains the correct configuration for the system. I don’t know what problem you may see. It will overwrite manual changes to configuration files maintained by foreman. But that’s intended. You are not supposed to modify the configuration but use the correct foreman-installer options instead…

1 Like
3

thanks for your replay @gvde

here are two of them:

I dont have the command:


root@test-srv:~# foreman-maintain
foreman-maintain: command not found
root@test-srv:~# dpkg-query -l | grep foreman
ii  foreman                                3.5.1-1+ubuntu2004                amd64        Systems management web interface
ii  foreman-cli                            3.5.1-1+ubuntu2004                all          metapackage providing hammer CLI for Foreman
ii  foreman-debug                          3.5.1-1+ubuntu2004                all          provides support utility foreman-debug.
ii  foreman-dynflow-sidekiq                3.5.1-1+ubuntu2004                all          package providing dynflow sidekiq services for Foreman
ii  foreman-installer                      3.5.1-1+ubuntu2004                all          Automated puppet-based installer for The Foreman
ii  foreman-postgresql                     3.5.1-1+ubuntu2004                all          metapackage providing PostgreSQL dependencies for Foreman
ii  foreman-proxy                          3.5.1-1+ubuntu2004                all          RESTful proxies for DNS, DHCP, TFTP, and Puppet
ii  foreman-service                        3.5.1-1+ubuntu2004                all          package providing Foreman services running via puma
ii  ruby-foreman-column-view               0.4.0-2                           all          Foreman Column View Plugin
ii  ruby-foreman-deface                    1.9.0-2                           all          Foreman Deface Plugin Dependency
ii  ruby-foreman-puppet                    5.0.0-1                           all          Foreman Puppet Plugin
ii  ruby-hammer-cli-foreman                3.5.0-1+ubuntu2004                all          Foreman commands for Hammer
ii  ruby-hammer-cli-foreman-puppet         0.0.6-1                           all          Foreman Puppet plugin for Hammer CLI

After reboot my server without stopping any services. I could not login! and all my hosts, config groups, host groups was deleted! after check the log file I saw there was a postgres error, I logged in and the foreman role and foreman database was deleted! this happens to me on test & production server!

4

Also other topic:

5

The first one, is old. From 2020 with foreman version 1.20. Very old. I don’t see anything suggesting the systems breaks during normal shutdown.

I don’t know what went wrong in the second one. That’s a rare exception. And I also don’t see that it’s caused by a normal shutdown.

You are running on ubuntu. I think that command is still missing on ubuntu.

During reboot all services are stopped automatically. Unless you mess with systemd it’ll be an orderly shutdown.

I don’t know what happened. It looks like a serious postgres error. That would have to be looked at. I suspect, not the database to be deleted but for some reason it could access it anymore and the standard postgresql boot process might automatically create a new database if it thinks it doesn’t have one. But your postgresql logs should show what happens during the last time it shut down with a working database.

But again: the postgresql database should be stopped during the normal shutdown. I would suggest to check the logs and the console output during shutdown if it’s working normal and it’s stopping all processes. If not, the systemd shutdown is broken and that needs to be fixed. It’s kind of pointless to stop all services manually before that. Fix the shutdown.

So far, I don’t see anything really related with foreman but a more general issue with the shutdown process on your server. For instance, if it kills the postgresql database during shutdown and doesn’t run the normal shutdown, chance is it breaks if it’s interrupted while it’s writing crucial data…

1 Like
6

I hope that foreman team can add the command foreman-maintain for ubuntu too.

Thank you so much @gvde you are abselutly right… because since last problem with database I stop all services manually and I create database dump foreman-rake db:dump then I reboot the server…

So we write permanent to database with 400+ Hosts. so thats right what you said its Database issue. I know now more than before… thank you a lot @gvde