Failed to retrieve provision template

Most of my set up was working great, until the past day. What’s happening is, during deployments, whether virtual or baremetal, the tftp file is created for the given MAC address on the smart proxy, and the target deployment starts booting into initrd / etc., gets all the way to reading the kickstart file via http, and errors being unable to find the provision template.

I can see more by curling the given http URL:

$ curl http://foreman.example.com/unattended/provision?token=fde4dd56-c3ea-4093-a07b-27041df34e0e
\# unable to find provision template for dev-testvm409.example.com running CentOS 7

When forcing /etc/foreman-proxy/settings.d/templates.yaml to use the proxy instead of the main foreman host, I get a different error on curl:

Failed to retrieve provision template for {"token"=>"fde4dd56-c3ea-4093-a07b-27041df34e0e", "splat"=>[], "captures"=>["provision"], "kind"=>"provision"}: SSL_connect returned=1 errno=0 state=error: certificate verify failed

There hasn’t been many changes in the past few days, the only thing I can think of is a plugin that sync’s to github our provisioning templates. It’s been working all week, for the most part.

Using Foreman 1.15 and Foreman Proxy 1.15

Foreman master plugins:

tfm-rubygem-fast_gettext-1.1.0-1.el7.noarch
tfm-rubygem-gettext_i18n_rails-1.2.1-3.el7.noarch
tfm-rubygem-unf-0.1.3-5.el7.noarch
tfm-rubygem-excon-0.51.0-1.el7.noarch
tfm-rubygem-sshkey-1.9.0-1.el7.noarch
tfm-rubygem-css_parser-1.4.7-1.el7.noarch
tfm-rubygem-rest-client-1.8.0-1.el7.noarch
tfm-rubygem-ancestry-2.2.1-1.el7.noarch
tfm-rubygem-will_paginate-3.1.5-1.el7.noarch
tfm-rubygem-activerecord-session_store-0.1.2-2.el7.noarch
tfm-rubygem-apipie-rails-0.4.0-2.el7.noarch
tfm-rubygem-passenger-native-libs-4.0.18-9.11.el7.x86_64
tfm-rubygem-rbvmomi-1.10.0-1.el7.noarch
tfm-rubygem-fog-libvirt-0.4.1-1.el7.noarch
tfm-rubygem-apipie-bindings-0.2.0-1.el7.noarch
tfm-rubygem-locale-2.0.9-11.el7.noarch
tfm-rubygem-trollop-2.1.2-1.el7.noarch
tfm-rubygem-clamp-1.0.0-5.el7.noarch
tfm-rubygem-unicode-display_width-1.0.5-1.el7.noarch
tfm-rubygem-highline-1.7.8-2.el7.noarch
tfm-runtime-3.2-9.el7.x86_64
tfm-rubygem-sexp_processor-4.4.4-3.el7.noarch
tfm-rubygem-ruby_parser-3.6.3-4.el7.noarch
tfm-rubygem-net-ssh-4.0.1-2.el7.noarch
tfm-rubygem-net-scp-1.1.0-6.el7.noarch
tfm-rubygem-safemode-1.3.2-1.el7.noarch
tfm-rubygem-rack-jsonp-1.3.1-5.el7.noarch
tfm-rubygem-unf_ext-0.0.6-6.el7.x86_64
tfm-rubygem-domain_name-0.5.20160310-1.el7.noarch
tfm-rubygem-net-ping-2.0.1-1.el7.noarch
tfm-rubygem-useragent-0.16.8-1.el7.noarch
tfm-rubygem-mysql2-0.4.5-1.el7.x86_64
tfm-rubygem-oauth-0.5.1-1.el7.noarch
tfm-rubygem-addressable-2.3.6-4.el7.noarch
tfm-rubygem-formatador-0.2.1-9.el7.noarch
tfm-rubygem-netrc-0.7.7-7.el7.noarch
tfm-rubygem-deacon-1.0.0-1.el7.noarch
tfm-rubygem-logging-1.8.2-4.el7.noarch
tfm-rubygem-bundler_ext-0.4.1-1.el7.noarch
tfm-rubygem-scoped_search-4.1.0-1.el7.noarch
tfm-rubygem-validates_lengths_from_database-0.5.0-3.el7.noarch
tfm-rubygem-friendly_id-5.1.0-3.el7.noarch
tfm-rubygem-ldap_fluff-0.4.6-1.el7.noarch
tfm-rubygem-x-editable-rails-1.5.5-1.el7.noarch
tfm-rubygem-rails-i18n-4.0.9-1.el7.noarch
tfm-rubygem-webpack-rails-0.9.8-1.el7.noarch
tfm-rubygem-roadie-rails-1.1.1-1.el7.noarch
tfm-rubygem-passenger-4.0.18-9.11.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-9.11.el7.x86_64
tfm-rubygem-fog-xml-0.1.2-4.el7.noarch
tfm-rubygem-fog-vsphere-1.7.0-1.el7.noarch
tfm-rubygem-rainbow-2.2.1-1.el7.noarch
tfm-rubygem-foreman_vmwareannotations-0.0.1-1.fm1_15.el7.noarch
tfm-rubygem-ruby-libvirt-0.7.0-1.el7.x86_64
tfm-rubygem-hammer_cli_foreman-0.10.2-1.el7.noarch
tfm-rubygem-diffy-3.0.1-3.el7.noarch
tfm-rubygem-foreman_templates-5.0.1-1.fm1_15.el7.noarch
tfm-rubygem-net-ldap-0.15.0-1.el7.noarch
tfm-rubygem-ruby2ruby-2.1.3-4.el7.noarch
tfm-rubygem-little-plugger-1.1.3-21.el7.noarch
tfm-rubygem-http-cookie-1.0.2-1.el7.noarch
tfm-rubygem-secure_headers-3.4.1-1.el7.noarch
tfm-rubygem-facter-2.4.0-3.el7.x86_64
tfm-rubygem-fog-core-1.42.0-1.el7.noarch
tfm-rubygem-deep_cloneable-2.2.2-1.el7.noarch
tfm-rubygem-audited-4.4.1-1.el7.noarch
tfm-rubygem-rabl-0.12.0-2.el7.noarch
tfm-rubygem-responders-2.3.0-1.el7.noarch
tfm-rubygem-roadie-3.2.1-1.el7.noarch
tfm-rubygem-fog-json-1.0.2-4.el7.noarch
tfm-rubygem-deface-1.2.0-1.el7.noarch
tfm-rubygem-foreman_snapshot_management-1.3.0-1.fm1_15.el7.noarch
tfm-rubygem-awesome_print-1.7.0-1.el7.noarch
tfm-rubygem-hammer_cli-0.10.2-1.el7.noarch
tfm-rubygem-git-1.2.5-7.el7.noarch

Foreman Proxy plugins:

rubygem-smart_proxy_dns_infoblox-0.0.6-1.fm1_15.el7.noarch
rubygem-logging-1.8.2-4.el7.noarch
rubygem-faraday_middleware-0.10.0-1.el7.noarch
rubygem-rubyipmi-0.10.0-2.el7.noarch
rubygem-json-1.7.7-30.el7.x86_64
rubygem-ffi-1.9.10-2.el7.x86_64
rubygem-tilt-2.0.7-1.el7.noarch
rubygem-bundler_ext-0.4.1-1.el7.noarch
rubygem-ansi-1.4.3-2.el7.noarch
rubygem-multi_json-1.10.1-3.el7.noarch
rubygem-rack-1.6.4-2.el7.noarch
rubygem-rack-protection-1.5.3-3.el7.noarch
rubygem-bundler-1.7.8-3.el7.noarch
rubygem-infoblox-2.0.4-1.el7.noarch
rubygem-smart_proxy_dhcp_infoblox-0.0.7-1.fm1_15.el7.noarch
rubygem-kafo_wizards-0.0.1-2.el7.noarch
rubygem-kafo_parsers-0.1.6-1.el7.noarch
rubygem-hashie-2.0.5-4.el7.noarch
rubygem-kafo-2.0.0-1.el7.noarch
rubygem-rb-inotify-0.9.7-2.el7.noarch
rubygem-concurrent-ruby-1.0.3-1.el7.noarch
rubygem-thor-0.19.1-1.el7.noarch
rubygem-psych-2.0.0-30.el7.x86_64
rubygem-rdoc-4.0.0-30.el7.noarch
rubygem-rake-0.9.6-30.el7.noarch
rubygem-gssapi-1.1.2-3.el7.noarch
rubygem-sinatra-1.4.8-2.el7.noarch
rubygem-bigdecimal-1.2.0-30.el7.x86_64
rubygem-little-plugger-1.1.3-21.el7.noarch
rubygem-faraday-0.9.1-4.el7.noarch
rubygem-net-http-persistent-2.8-5.el7.noarch
rubygems-2.0.14.1-30.el7.noarch
rubygem-highline-1.7.8-2.el7.noarch
rubygem-clamp-1.0.0-5.el7.noarch
rubygem-powerbar-1.0.17-1.el7.noarch
rubygem-multipart-post-2.0.0-4.el7.noarch
rubygem-rkerberos-0.1.3-5.el7.x86_64
rubygem-io-console-0.4.2-30.el7.x86_64

Hey, what is your token duration in Administer - Setting? By default it is 6 hours, you may want to doublecheck that. In the host detail page, it should be visible if token is still valid, or you can use foreman-rake console:

# Host.find(123).token

After grepping our codebase, it looks like you don’t have “provision” kind associated with your OS. Import had to reassociate or unassociate it, that’s your problem.

Hi thanks for your help, the token duration is set to 360 mins.

I’m not sure how to access foreman-rake console, the command itself asks for a bunch of options it doesn’t drop me to a CLI.

Something strange, I uninstalled the tfm-rubygem-foreman_templates-5.0.1-1.fm1_15.el7.noarch plugin, and now when I look in Operating System templates, my usual Provisioning, iPXE, and PXELinux templates are there, then there are these new ones I’ve never seen before:

Finish template *
PXEGrub template *
PXEGrub2 template *
User data template *

The pull down allows for some type of community templates to assign. However, the error still persists during deployment.

I’m going to try starting another instance from scratch and excluding this plugin, I fear it has something to do with this issue.

Turns out you were right, a syntax issue across the templates was messing everything up.