Cannot import Modules/Classes on 1.12 with Puppet 4

Hi All,

I have a strange issue with the puppetclasses which don't want to be
imported into the environments with puppet 4

My modules are in /etc/puppetlabs/code/environments/common which normally
imported them in all available environments which they don't. As this is an
upgrade/migration the classes were already in the database and I was able
to import one to test. I saw that my agents were not able to find the
classes on a run where I tried to import again to see what happened. All
classes were removed from all environments in Foreman but they still exist
in Foreman, just not assigned to environments.

In general:

  • My modules are in: /etc/puppetlabs/code/environments/common
  • All module folders and manifests are owned by puppet:puppet
  • I don't see any errors on debug level.

foreman-proxy.log

D, [2016-10-16T23:08:06.837427 #2139] DEBUG – : close: 172.16.3.211:54628
D, [2016-10-16T23:29:39.917682 #2139] DEBUG – : accept: 172.16.3.211:54704
D, [2016-10-16T23:29:39.921191 #2139] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-16T23:29:39.923338 #2139] DEBUG – : verifying remote client
172.16.3.211 against trusted_hosts ["foreman-01.my.domain"]
I, [2016-10-16T23:29:40.128618 #2139] INFO – : 172.16.3.211 - -
[16/Oct/2016:23:29:40 +0200] "GET /puppet/environments HTTP/1.1" 200 37
0.2055

D, [2016-10-16T23:29:40.170844 #2139] DEBUG – : close: 172.16.3.211:54704
D, [2016-10-16T23:29:40.250892 #2139] DEBUG – : accept: 172.16.3.211:54708
D, [2016-10-16T23:29:40.254132 #2139] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-16T23:29:40.256609 #2139] DEBUG – : verifying remote client
172.16.3.211 against trusted_hosts ["foreman-01.my.domain"]
I, [2016-10-16T23:29:40.462696 #2139] INFO – : 172.16.3.211 - -
[16/Oct/2016:23:29:40 +0200] "GET /puppet/environments/production/classes
HTTP/1.1" 200 2 0.2063

D, [2016-10-16T23:29:40.503884 #2139] DEBUG – : close: 172.16.3.211:54708
D, [2016-10-16T23:29:40.581616 #2139] DEBUG – : accept: 172.16.3.211:54712
D, [2016-10-16T23:29:40.584482 #2139] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-16T23:29:40.586922 #2139] DEBUG – : verifying remote client
172.16.3.211 against trusted_hosts ["foreman-01.my.domain"]
I, [2016-10-16T23:29:40.799533 #2139] INFO – : 172.16.3.211 - -
[16/Oct/2016:23:29:40 +0200] "GET /puppet/environments/development/classes
HTTP/1.1" 200 2 0.2129

D, [2016-10-16T23:29:40.841018 #2139] DEBUG – : close: 172.16.3.211:54712
D, [2016-10-16T23:29:40.925290 #2139] DEBUG – : accept: 172.16.3.211:54716
D, [2016-10-16T23:29:40.929084 #2139] DEBUG – : Rack::Handler::WEBrick is
invoked.
D, [2016-10-16T23:29:40.931537 #2139] DEBUG – : verifying remote client
172.16.3.211 against trusted_hosts ["foreman-01.my.domain"]
I, [2016-10-16T23:29:41.126534 #2139] INFO – : 172.16.3.211 - -
[16/Oct/2016:23:29:41 +0200] "GET /puppet/environments/common/classes
HTTP/1.1" 200 2 0.1953

D, [2016-10-16T23:29:41.168023 #2139] DEBUG – : close: 172.16.3.211:54716

puppetserver.log
2016-10-16 23:29:40,457 INFO [qtp85066979-69] [puppetserver] Puppet Not
Found: Could not find instances in resource_type with ''
2016-10-16 23:29:40,793 INFO [qtp85066979-69] [puppetserver] Puppet Not
Found: Could not find instances in resource_type with '
'
2016-10-16 23:29:41,121 INFO [qtp85066979-69] [puppetserver] Puppet Not
Found: Could not find instances in resource_type with '*'

But what I do see is:

puppetserver-access.log
172.16.3.250 - - [16/Oct/2016:23:33:11 +0200] "PUT
/puppet/v3/report/foreman-01.my.domain?environment=production& HTTP/1.1" 200
11 "-" "Ruby" 805
172.16.3.250 - - [16/Oct/2016:23:33:15 +0200] "GET /puppet/v3/environments
HTTP/1.1" 200 918 "-" "Ruby" 21
172.16.3.250 - - [16/Oct/2016:23:33:15 +0200] "GET
/puppet/v3/resource_types/?kind=class&&environment=production HTTP/1.1" 404
109 "-" "Ruby" 24
172.16.3.250 - - [16/Oct/2016:23:33:16 +0200] "GET
/puppet/v3/resource_types/
?kind=class&&environment=development HTTP/1.1"
404 109 "-" "Ruby" 24
172.16.3.250 - - [16/Oct/2016:23:33:16 +0200] "GET
/puppet/v3/resource_types/?kind=class&&environment=common HTTP/1.1" 404 109
"-" "Ruby" 20
172.16.3.250 - - [16/Oct/2016:23:34:34 +0200] "GET
/puppet/v3/node/puppetca-01.my.domain?environment=production&configured_environment=production&transaction_uuid=07d74bf0-46e1-4695-b156-2dedbf172a22&fail_on_404=true
HTTP/1.1" 200 12466 "-" "Ruby" 1414
172.16.3.250 - - [16/Oct/2016:23:34:34 +0200] "GET
/puppet/v3/file_metadatas/pluginfacts?environment=production&links=follow&recurse=true&source_permissions=use&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5
HTTP/1.1" 200 219 "-" "Ruby" 22
172.16.3.250 - - [16/Oct/2016:23:34:34 +0200] "GET
/puppet/v3/file_metadatas/plugins?environment=production&links=follow&recurse=true&source_permissions=ignore&ignore=.svn&ignore=CVS&ignore=.git&checksum_type=md5
HTTP/1.1" 200 220 "-" "Ruby" 18
172.16.3.250 - - [16/Oct/2016:23:34:37 +0200] "POST
/puppet/v3/catalog/puppetca-01.my.domain?environment=production HTTP/1.1"
200 608 "-" "Ruby" 2006
172.16.3.250 - - [16/Oct/2016:23:34:38 +0200] "PUT
/puppet/v3/report/puppetca-01.my.domain?environment=production& HTTP/1.1"
200 11 "-" "Ruby" 486
172.16.3.250 - - [16/Oct/2016:23:35:14 +0200] "GET /puppet/v3/environments
HTTP/1.1" 200 918 "-" "Ruby" 18
172.16.3.250 - - [16/Oct/2016:23:35:14 +0200] "GET
/puppet/v3/resource_types/
?kind=class&&environment=production HTTP/1.1" 404
109 "-" "Ruby" 21
172.16.3.250 - - [16/Oct/2016:23:35:15 +0200] "GET
/puppet/v3/resource_types/?kind=class&&environment=development HTTP/1.1"
404 109 "-" "Ruby" 20
172.16.3.250 - - [16/Oct/2016:23:35:15 +0200] "GET
/puppet/v3/resource_types/
?kind=class&&environment=common HTTP/1.1" 404 109
"-" "Ruby" 20

There must be some misconfiguration I guess and I'm not sure what.

It's hard to say what might be happening without seeing the structure. Can
you share your puppet.conf on the master, and the output of "tree
/etc/puppetlabs/code/environments -d -L 3"?

Hi Greg,

I wasn't sure about them, here they are:

tree /etc/puppetlabs/code/environments -d -L 3

/etc/puppetlabs/code/environments
├── common
│ ├── accounts
│ │ ├── files
│ │ ├── lib
│ │ ├── manifests
│ │ ├── spec
│ │ └── templates
│ ├── apache
│ │ ├── files
│ │ ├── manifests
│ │ └── templates
│ ├── apt
│ │ ├── lib
│ │ ├── manifests
│ │ ├── spec
│ │ ├── templates
│ │ └── tests
│ ├── stdlib
│ │ ├── examples
│ │ ├── lib
│ │ ├── manifests
│ │ └── spec
│ ├── tests
│ │ ├── manifests
│ │ └── templates
│ ├── timezone
│ │ ├── manifests
│ │ ├── spec
│ │ └── tests
│ ├── vscrepo
│ │ ├── examples
│ │ ├── lib
│ │ └── spec
│ └── yum
│ ├── manifests
│ ├── spec
│ └── tests
├── development
│ ├── manifests
│ └── modules
└── production

puppet.conf:

cat /etc/puppetlabs/puppet/puppet.conf

File managed with puppet

Module: 'puppet'

[main]
# Where Puppet's general dynamic and/or growing data is kept
vardir = /opt/puppetlabs/puppet/cache

# The Puppet log directory.
# The default value is '$vardir/log'.
logdir = /var/log/puppetlabs/puppet

# Where Puppet PID files are kept.
# The default value is '$vardir/run'.
rundir = /var/run/puppetlabs

# Where SSL certificates are kept.
# The default value is '$confdir/ssl'.
ssldir = /etc/puppetlabs/puppet/ssl

# Allow services in the 'puppet' group to access key (Foreman + proxy)
privatekeydir = $ssldir/private_keys { group = service }
hostprivkey = $privatekeydir/$certname.pem { mode = 640 }

show_diff     = false

log_level = debug

Server config

reports          = foreman,puppetdb

environmentpath  = /etc/puppetlabs/code/environments
basemodulepath   = /etc/puppetlabs/code/environments/common:/etc/

puppetlabs/code/modules:/usr/share/puppet/modules

hiera_config     = $confdir/hiera.yaml

Next part of the file is managed by a different template

Module: 'puppet'

[agent]
# The file in which puppetd stores a list of the classes
# associated with the retrieved configuration. Can be loaded in
# the separate puppet executable using the --loadclasses
# option.
# The default value is '$statedir/classes.txt'.
classfile = $statedir/classes.txt

# Where puppetd caches the local configuration.  An
# extension indicating the cache format is added automatically.
# The default value is '$confdir/localconfig'.
localconfig = $vardir/localconfig

# Disable the default schedules as they cause continual skipped
# resources to be displayed in Foreman - only for Puppet >= 3.4
default_schedules = false

report            = true
pluginsync        = true
masterport        = 8140

environment = production

certname          = foreman-01.my.domain
server            = foreman.my.domain
listen            = false
splay             = false
splaylimit        = 1800
runinterval       = 1800
noop              = false
usecacheonfailure = true
ca_server         = puppetca.my.domain

Next part of the file is managed by a different template

Module: 'puppet'

[master]

autosign = /etc/puppetlabs/puppet/autosign.conf { mode = 0664 }

external_nodes = /etc/puppetlabs/puppet/node.rb
node_terminus  = exec
ca             = false
ssldir         = /etc/puppetlabs/puppet/ssl
certname       = foreman-01.my.domain
parser         = current
ca_server      = puppetca.my.domain
strict_variables = false

pluginsync = true

storeconfigs = true
storeconfigs_backend = puppetdb
log_level = debug
reports          = foreman

environmentpath=/etc/puppetlabs/code/environments

··· Op maandag 17 oktober 2016 10:04:11 UTC+2 schreef Greg Sutcliffe: > > It's hard to say what might be happening without seeing the structure. Can > you share your puppet.conf on the master, and the output of "tree > /etc/puppetlabs/code/environments -d -L 3"? >

So that looks ok, I think, but I do wonder about how "common" is inside the
environmentpath. Does "common" have an environment.conf as well (so that
puppet would regard it as a valid environment)?

Also this is interesting:

> … I saw that my agents were not able to find the classes on a run …

To me that says Puppet can't find your classes either. Since Foreman
queries Puppet for it's class listing in Puppet4 (via the puppetserver's
API) then I would expect Foreman to think the classes had gone if Puppet
can't find them either.

It feels like a misconfig of some kind, but I can't quite spot it.
Perhaps try moving the common modules to "/etc/puppetlabs/code/modules"
(since thats also in the basemodulepath) as a test?

Greg

··· On 17 October 2016 at 09:40, Matt wrote:

Hi,

common didn't had an environment.conf in it, I placed it in, didn't fix it.

When I did a mv * …/…/modules foreman found all my classes, this is
strange to me.

So, your last line is indeed a fix, but it doens't solve the problem for
environments/common

Do you want me more to chec/test ?

··· Op dinsdag 18 oktober 2016 10:48:24 UTC+2 schreef Greg Sutcliffe: > > So that looks ok, I think, but I do wonder about how "common" is inside > the environmentpath. Does "common" have an environment.conf as well (so > that puppet would regard it as a valid environment)? > > Also this is interesting: > > On 17 October 2016 at 09:40, Matt <yamaka...@gmail.com > > wrote: > > ... I saw that my agents were not able to find the classes on a run ... > > To me that says Puppet can't find your classes either. Since Foreman > queries Puppet for it's class listing in Puppet4 (via the puppetserver's > API) then I would expect Foreman to think the classes had gone if Puppet > can't find them either. > > It *feels* like a misconfig of some kind, but I can't quite spot it. > Perhaps try moving the common modules to "/etc/puppetlabs/code/modules" > (since thats also in the basemodulepath) as a test? > > Greg >

I wanted to update this thread as it seems that on puppet 4 the overall, so
known common manifests are now in /etc/puppetlabs/code/modules.

Seperate environments are only picked up in
/etc/puppetlabs/code/environments/<environmentname>/modules

I think it's how the API works and searches.

··· Op dinsdag 18 oktober 2016 15:23:41 UTC+2 schreef Matt: > > Hi, > > common didn't had an environment.conf in it, I placed it in, didn't fix it. > > When I did a mv * ../../modules foreman found all my classes, this is > strange to me. > > So, your last line is indeed a fix, but it doens't solve the problem for > environments/common > > Do you want me more to chec/test ? > > > Op dinsdag 18 oktober 2016 10:48:24 UTC+2 schreef Greg Sutcliffe: >> >> So that looks ok, I think, but I do wonder about how "common" is inside >> the environmentpath. Does "common" have an environment.conf as well (so >> that puppet would regard it as a valid environment)? >> >> Also this is interesting: >> >> On 17 October 2016 at 09:40, Matt wrote: >> > ... I saw that my agents were not able to find the classes on a run ... >> >> To me that says Puppet can't find your classes either. Since Foreman >> queries Puppet for it's class listing in Puppet4 (via the puppetserver's >> API) then I would expect Foreman to think the classes had gone if Puppet >> can't find them either. >> >> It *feels* like a misconfig of some kind, but I can't quite spot it. >> Perhaps try moving the common modules to "/etc/puppetlabs/code/modules" >> (since thats also in the basemodulepath) as a test? >> >> Greg >> >