Puppet agent disabled after build process

Problem: After a VM is built it boots, but the puppet service does not run. Systemd tells me, it is not enabled.
There is no journal for Puppet that indicates a problem.

How is that possible? Is there a deployment log I could check?

Expected outcome: Puppet should start (again; it did in past) after VM creation to apply the required configurations.

Foreman and Proxy versions: 2.0

Distribution and version: Installed OS is Ubuntu 18.04 ( not the latest minor release)

Other relevant data:

No updates were applied and no configuration changes (or at least we know of) were done.

As far as I understand the finish template calls the puppet_setup template.

<%#
kind: finish
name: Preseed default finish
model: ProvisioningTemplate
oses:
- Debian
- Ubuntu
%>
<%
  # safemode renderer does not support unary negation
  pm_set = @host.puppetmaster.empty? ? false : true
  puppet_enabled = pm_set || host_param_true?('force-puppet')
  salt_enabled = host_param('salt_master') ? true : false
  chef_enabled = @host.respond_to?(:chef_proxy) && @host.chef_proxy
%>

<% subnet = @host.subnet -%>
<% if @static -%>
  <%- dhcp = false -%>
<% elsif subnet.nil? -%>
  <%- dhcp = true -%>
<% else -%>
  <%- dhcp = subnet.dhcp_boot_mode? -%>
<% end -%>
<% unless dhcp -%>
# host and domain name need setting as these values may have come from dhcp if pxe booting
/bin/sed -i "s/^search.*$/search <%= @host.domain %>/g" /etc/resolv.conf
/bin/sed -i "s/.*dns-search.*/\tdns-search <%= @host.domain %>/g" /etc/network/interfaces
/bin/sed -i "s/^<%= @host.ip %>.*/<%= @host.ip %>\t<%= @host.shortname %>.<%= @host.domain %>\t<%= @host.shortname %>/g" /etc/hosts
/bin/echo <%= @host.shortname %> > /etc/hostname
/bin/hostname  <%= @host.shortname %>.<%= @host.domain %>
<% end -%>

<%= snippet_if_exists(template_name + " custom snippet") %>
<% if host_enc['parameters']['realm'] && @host.realm && @host.realm.realm_type == 'FreeIPA' -%>
<%= snippet 'freeipa_register' %>
<% end -%>

<%= snippet('remote_execution_ssh_keys') %>

<%= snippet "blacklist_kernel_modules" %>

<% if chef_enabled %>
<%= snippet 'chef_client' %>
<% end -%>

<% if puppet_enabled %>
<% if host_param_true?('enable-puppetlabs-repo') || host_param_true?('enable-puppetlabs-puppet6-repo') || host_param_true?('enable-puppetlabs-puppet5-repo') -%>
<%= snippet 'puppetlabs_repo' %>
<% end -%>
<%= snippet 'puppet_setup' %>
<% end -%>

<% if salt_enabled %>
<%= snippet 'saltstack_setup' %>
<% end -%>

<%= snippet 'preseed_networking_setup' %>
<%= snippet 'efibootmgr_netboot' %>
<%= snippet 'built' %>

And this puppet_setup template should start and enable Puppet. Shouldn’t it?

<%#
kind: snippet
name: puppet_setup
model: ProvisioningTemplate
description: this snippet will configure the Puppet agent
snippet: true
%>
<%
os_family = @host.operatingsystem.family
os_major  = @host.operatingsystem.major.to_i
os_name   = @host.operatingsystem.name

aio_enabled = host_param_true?('enable-puppetlabs-puppet6-repo') || host_param_true?('enable-puppet6') || host_param_true?('enable-puppetlabs-puppet5-repo') || host_param_true?('enable-puppet5')

if os_family == 'Freebsd'
  freebsd_package = host_param_true?('enable-puppet6') ? 'puppet6' : 'puppet5'
  etc_path = '/usr/local/etc/puppet'
  bin_path = '/usr/local/bin'
elsif os_family == 'Windows'
  windows_package = "puppet-agent-#{@host.architecture}.msi"
  etc_path = 'C:\ProgramData\PuppetLabs\puppet\etc'
  bin_path = 'C:\Program Files\Puppet Labs\Puppet\bin'
elsif aio_enabled
  linux_package = 'puppet-agent'
  etc_path = '/etc/puppetlabs/puppet'
  bin_path = '/opt/puppetlabs/bin'
else
  linux_package = os_family == 'Suse' ? 'rubygem-puppet' : 'puppet'
  etc_path = '/etc/puppet'
  bin_path = '/usr/bin'
end
%>

<% if os_family == 'Debian' -%>
apt-get update
apt-get install -y <%= linux_package %>
<% elsif os_family == 'Freebsd' -%>
pkg install -y <%= freebsd_package %>
<% elsif os_family == 'Redhat' -%>
if [ -f /usr/bin/dnf ]; then
  dnf -y install <%= linux_package %>
else
  yum -t -y install <%= linux_package %>
fi
<% elsif os_family == 'Suse' -%>
<% if host_param_true?('enable-puppetlabs-puppet6-repo') || host_param_true?('enable-puppetlabs-puppet5-repo') -%>
rpmkeys --import http://yum.puppet.com/RPM-GPG-KEY-puppetlabs
rpmkeys --import http://yum.puppet.com/RPM-GPG-KEY-puppet
<% end -%>
<% if @host.provision_method == 'image' -%>
/usr/bin/zypper -n install <%= linux_package %>
<% end -%>
<% elsif os_family == 'Windows' -%>
$puppet_agent_msi = "${env:TEMP}\<%= windows_package %>"
$puppet_install_args = @(
  '/qn',
  '/norestart',
  '/i',
  "${puppet_agent_msi}",
  <%- if @host.puppet_ca_server.strip -%>
  "PUPPET_CA_SERVER=<%= @host.puppet_ca_server %>",
  <%- end -%>
  "PUPPET_MASTER_SERVER=<%= @host.puppetmaster %>"
)

Write-Host "Installing ${puppet_agent_msi} with args ${puppet_install_args}"
Start-Process 'msiexec.exe' -ArgumentList $puppet_install_args -Wait -NoNewWindow
<% end -%>

<% if os_family == 'Windows' -%>
$puppet_conf = @("<%= snippet 'puppet.conf' %>".Replace("`n","`r`n"))
Out-File -FilePath <%= etc_path %>\puppet.conf -InputObject $puppet_conf
<% else -%>
cat > <%= etc_path %>/puppet.conf << EOF
<%= snippet 'puppet.conf' %>
EOF
<% end -%>

<% if @host.puppetca_token.present? -%>
<% if os_family == 'Windows' -%>
$csr_attributes = @("<%= snippet 'csr_attributes.yaml' %>".Replace("`n","`r`n"))
Out-File -FilePath <%= etc_path %>\csr_attributes.yaml -InputObject $csr_attributes
<% else -%>
cat > <%= etc_path %>/csr_attributes.yaml << EOF
<%= snippet 'csr_attributes.yaml' %>
EOF
<% end -%>
<% end -%>

<% if os_family == 'Redhat' -%>
<% if os_major > 6 -%>
puppet_unit=puppet
/usr/bin/systemctl list-unit-files | grep -q puppetagent && puppet_unit=puppetagent
/usr/bin/systemctl enable ${puppet_unit}
<% else -%>
/sbin/chkconfig --level 345 puppet on
<% end -%>
<% end -%>
<% if os_family == 'Freebsd' -%>
echo 'puppet_enable="YES"' >>/etc/rc.conf
<% end -%>
<% unless aio_enabled -%>
<% if os_family == 'Debian' -%>
if [ -f "/etc/default/puppet" ]
then
/bin/sed -i 's/^START=no/START=yes/' /etc/default/puppet
fi
<%= bin_path %>/puppet agent --enable
<% elsif os_family == 'Suse' -%>
if [ -f "/etc/sysconfig/puppet" ]
then
/usr/bin/sed -ie s/^PUPPET_SERVER=.*/PUPPET_SERVER=<%= @host.puppetmaster.blank? ? '' : @host.puppetmaster %>/ /etc/sysconfig/puppet
fi
<% end -%>
<% end -%>
<%#
IMPORTANT NOTE: Setting "run-puppet-in-installer" is UNSUPPORTED!

The default mode of operation in Foreman is only to set up Puppet, but to not run it inside the installer environment.
Running Puppet inside the installer can cause various hard to diagnose errors, many of them resulting from the fact that
services are not started inside the installer.

If you are aware of the downsides, you can trigger a Puppet run inside the installer by setting the variable
run-puppet-in-installer to true.

Note, that this is an *unsupported mode of operation* and not supported by Foreman at all. You have been warned!
%>
<% if host_param_true?('run-puppet-in-installer') -%>
<% if (os_name == 'Ubuntu' && os_major >= 15) || (os_name == 'Debian' && os_major >= 8) -%>
# Puppet tries to detect the init system by checking the presence of the directory /run/systemd/system. That detection
# fails in a chroot environment like the one the installer provides. See Puppet tickets PA-136 and PUP-5577
#
# We work around that here until it gets fixed in Puppet (probably never for Puppet 3.x)
mkdir -p /run/systemd/system
<% end -%>
<% end -%>
# export a custom fact called 'is_installer' to allow detection of the installer environment in Puppet modules
<% if os_family == 'Windows' -%>
$env:FACTER_is_installer = $TRUE

# passing a non-existent tag like "no_such_tag" to the puppet agent only initializes the node
$puppet_agent_args = @(
  "agent",
  "--config", "<%= etc_path %>\puppet.conf",
  "--onetime",
  <%= host_param_true?('run-puppet-in-installer') || @full_puppet_run ? '' : '"--tags no_such_tag",' %>
  <%= @host.puppetmaster.blank? ? '' : "\"--server #{@host.puppetmaster}\"," %>
  "--no-daemonize"
)
Start-Process '<%= bin_path %>\puppet' -ArgumentList $puppet_agent_args -Wait -NoNewWindow
<% else -%>
export FACTER_is_installer=true
# passing a non-existent tag like "no_such_tag" to the puppet agent only initializes the node
<%= bin_path %>/puppet agent --config <%= etc_path %>/puppet.conf --onetime <%= host_param_true?('run-puppet-in-installer') || @full_puppet_run ? '' : '--tags no_such_tag' %> <%= @host.puppetmaster.blank? ? '' : "--server #{@host.puppetmaster}" %> --no-daemonize
<% if os_family == 'Suse' || (os_name == 'Debian' && os_major > 8) || (os_name == 'Ubuntu' && os_major >= 15) -%>
<%= bin_path %>/puppet resource service puppet enable=true
<% end -%>
<% if @host.provision_method == 'image' -%>
<%= bin_path %>/puppet resource service puppet ensure=running
<% end -%>
<% end -%>
1 Like

Since the repository for Puppet6 is set up, I claim the finish template is running. At least partly.

The docs tell:

Finish templates are available for all hypervisors that support image based installs where the foreman server can reach the newly installed machine via ssh and scp. The script generated from the finish template is copied by the Foreman to the newly installed system via scp using username and password specified with the image. It is then executed by connecting again via ssh, making the script executable and either executing it directly or via sudo if the specified user is not root. Standard output and standard

But I can’t find any log files like this on Foreman server and proxys as well.

I am still not able to find the issue here :frowning:

But a workaround is to clone both templates and modify them. The finish template to call the new snippet, and the second one to run different commands:

In the new finish template, instead of:

<% if puppet_enabled %>
<% if host_param_true?('enable-puppetlabs-repo') || host_param_true?('enable-puppetlabs-puppet6-repo') || host_param_true?('enable-puppetlabs-puppet5-repo') -%>
<%= snippet 'puppetlabs_repo' %>
<% end -%>
<%= snippet 'puppet_setup' %>
<% end -%>

Use

<% if puppet_enabled %>
<% if host_param_true?('enable-puppetlabs-repo') || host_param_true?('enable-puppetlabs-puppet6-repo') || host_param_true?('enable-puppetlabs-puppet5-repo') -%>
<%= snippet 'puppetlabs_repo' %>
<% end -%>
<%= snippet 'puppet_setup_NEWNAME' %>
<% end -%>

And in the puppet_config_NEWNAME:
Instead of:

<%= bin_path %>/puppet resource service puppet enable=true

now:

/bin/systemctl enable puppet
<%= bin_path %>/puppet agent -t

Additionally, I’ve changed the names in both new templates in row 3 accordingly since I think it is required.

Old:

<%#
kind: finish
name: Preseed default finish 
model: ProvisioningTemplate
oses:
...
...

New:

<%#
kind: finish
name: Preseed default finish NEWNAME
model: ProvisioningTemplate
oses:
...
...

Also the operating system configuration needs to point the the new finish template.

I also found the error messages about the missing /etc/init.d/puppet file which is mentioned here.
This is really strange because this file is provided by the package puppet. But we install puppet-agent.
I don’t think this error is related to my problem described above.

I’m actually having the same problem with the default templates. This is happening on Ubuntu 18.04 and Ubuntu 20.04 on puppet 6 with theforeman 2.3.2.

I found the following in /var/log/installer/syslog after a successful install that did not enable puppet after boot.

Mar  3 21:28:32 in-target: Running in chroot, ignoring request.
Mar  3 21:28:32 in-target: ^[[1;31mError: Execution of '/usr/sbin/update-rc.d puppet defaults' returned 1: update-rc.d: error: unable to read /etc/init.d/puppet^[[0m
Mar  3 21:28:32 in-target: ^[[1;31mError: /Service[puppet]/enable: change from 'false' to 'true' failed: Execution of '/usr/sbin/update-rc.d puppet defaults' returned 1: update-rc.d: error: unable to read /etc/init.d/puppet^[[0m
Mar  3 21:28:32 in-target: Running in chroot, ignoring request.
Mar  3 21:28:32 in-target: service { 'puppet':
Mar  3 21:28:32 in-target:   enable   => 'false',
Mar  3 21:28:32 in-target:   provider => 'debian',
Mar  3 21:28:32 in-target: }

I found https://tickets.puppetlabs.com/browse/PUP-9935 which seems to be a similar issue where puppet did not recognize a new Debian version as needing the systemd provider. My best guess is that puppet can’t correctly identify systemd as the provider for the service while in the installer environment. I can override the templates, but I think either the default provisioning templates should include the provider in the puppet command or that I should report this as a bug to puppet. I don’t know that they officially support puppet commands from inside installation environments though.

We are encountering the same problem on both 18.04 and 20.04, running Puppet 6 and Foreman 1.24.3.

I have opened a bug report with Puppet, since this seems to be an upstream issue.
https://tickets.puppetlabs.com/browse/PUP-10963

2 Likes

An update: we merged a provision template workaround:
https://github.com/theforeman/foreman/pull/8440

@ekohl That change will ship with the 2.4.1 release if I’m reading that correctly?