Issues on upgrade foreman 1.20 to 1.21

Support
1

Problem:
After the upgrade of foreman from 1.20 to 1.21, the puppetserver will not start anymore.
The logfile shows a Error in the async-dispatch-2 thread (logging below)

Foreman and Proxy versions:
Foreman 1.21.0-1
Puppetserver 5.3.7

Other relevant data:
Given commands during the upgrade :

sudo apache2 stop
sudo vi /etc/apt/sources.list.d/foreman.list
sudo apt update
sudo apt --only-upgrade install ruby\* foreman\*
sudo foreman-installer

The /var/log/puppetlabs/puppetserver/puppetserver.log shows the following error :

2019-03-11 11:19:06,198 INFO  [main] [o.e.j.u.log] Logging initialized @12913ms to org.eclipse.jetty.util.log.Slf4jLog
2019-03-11 11:19:08,413 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2019-03-11 11:19:08,430 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
clojure.lang.ExceptionInfo: Value does not match schema: (not (map? nil))
        at schema.core$validator$fn__2823.invoke(core.clj:155)
        at schema.core$validate.invokeStatic(core.clj:164)
        at schema.core$validate.invoke(core.clj:159)
        at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__42045$service_fnk__5266__auto___positional$reify__42052.init(metrics_service.clj:24)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5069$G__5056__5072.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5069$G__5055__5076.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842$fn__13843.invoke(internal.clj:198)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842.invoke(internal.clj:181)
        at puppetlabs.trapperkeeper.internal$eval13864$run_lifecycle_fns__13869$fn__13870.invoke(internal.clj:231)
        at puppetlabs.trapperkeeper.internal$eval13864$run_lifecycle_fns__13869.invoke(internal.clj:208)
        at puppetlabs.trapperkeeper.internal$eval14346$build_app_STAR___14355$fn$reify__14365.init(internal.clj:583)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399$fn__14400$fn__14402.invoke(internal.clj:611)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399$fn__14400.invoke(internal.clj:610)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399.invoke(internal.clj:604)
        at clojure.core$partial$fn__4761.invoke(core.clj:2521)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014$fn__14016.invoke(internal.clj:251)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014.invoke(internal.clj:251)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:1012)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:1011)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:1016)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:1014)
        at clojure.core.async$ioc_alts_BANG_$fn__12043.invoke(async.clj:383)
        at clojure.core.async$do_alts$fn__11989$fn__11992.invoke(async.clj:252)
        at clojure.core.async.impl.channels.ManyToManyChannel$fn__6799$fn__6800.invoke(channels.clj:95)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2019-03-11 11:19:08,439 INFO  [main] [p.t.internal] Beginning shutdown sequence
2

Which puppet agent version are you using? We try to derive the Puppet server version from the agent version but this isn’t perfect.

You could explicitly set the server version if you intend to stay at 5.3.7. Another is to upgrade to the latest 5.x version. I’d also verify the JRE version. It should be Java 8.

3

We have agents running on 4.10 (have to upgrade those) and on 5.5.10.
Java version is 1.8.0_191.

I will check what happens when I upgrade the puppetserver to the latest 5.x release.

4

It seems puppetserver is already at the highest 5.x version, so no solution to be expected from that side…
(https://puppet.com/docs/puppetserver/5.3/install_from_packages.html)

About the jre version… a bit more specific :

 $ dpkg -l | grep jre
ii  openjdk-8-jre-headless:amd64        8u191-b12-2ubuntu0.16.04.1

So, it looks like the prerequisites you mentioned are met…

  • JRE on version 8
  • Puppet server on highest supported version for foreman

A small thing for me to do is to make sure the 4.10 agents are upgraded as well. My suspicion is that the agent version is not the issue, since the puppetserver already fails at startup… before any evil agent did get a chance to do it’s dirty work…

5

I suspect that the issue is with using a 5.x server and a 4.x agent package on the same system. Note we don’t require all agents to be 5.x. It comes from:

Note that you can pass in the puppetserver version explicitly as well (--puppet-puppetserver-version 5.3.7) in case our logic is incorrect.

If this doesn’t work, I’m a bit out of ideas and I’d suggest the Puppet bugtracker or their slack channel.

6

No luck yet…
I did try with option --puppet-server-version 5.3.7, but this resulted in a error :

 /Stage[main]/Puppet::Server::Install/Package[puppetserver]/ensure: change from '5.3.7-1xenial' to '5.3.7' failed

(option --puppet-puppetserver-version die not exist)

So… I took the liberty to set the value to 5.3.7-1xenial :

sudo foreman-installer --puppet-server-version 5.3.7-1xenial

This did not show any error on the foreman installer run about the version, but start of the puppetserver service still failed with the clojure.lang.ExceptionInfo as described in the initial issue.

7

Apologies, it’s actually --puppet-server-puppetserver-version. That sets the logic on which version is to be installed.

The option --puppet-server-version is the directive for the puppet package directive. That can also be installed, latest or absent. Because of this difference in format we actually have two parameters. Maybe we could unify them, but right now we don’t.

8

Did try with the correct parameter :

sudo foreman-installer --puppet-server-puppetserver-version 5.3.7

But… still the same issue on the restart of the puppet server :

2019-03-12 14:19:27,650 INFO  [main] [o.e.j.u.log] Logging initialized @12636ms to org.eclipse.jetty.util.log.Slf4jLog
2019-03-12 14:19:30,037 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2019-03-12 14:19:30,064 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
clojure.lang.ExceptionInfo: Value does not match schema: (not (map? nil))
        at schema.core$validator$fn__2823.invoke(core.clj:155)
        at schema.core$validate.invokeStatic(core.clj:164)
        at schema.core$validate.invoke(core.clj:159)
        at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__42045$service_fnk__5266__auto___positional$reify__42052.init(metrics_service.clj:24)
9

I found out that this issue can be fixed with one parameter in the answers file (/etc/foreman-installer/scenarios.d/foreman-answers.yaml).
Change server_puppetserver_metrics: false to true, run the foreman-installer again and your done.

Costed me a couple of hours to find out so it would have been nice if the installer warned us that this setting was wrong.

1 Like
10

Your couple of hours of work paid off :slight_smile:
This solution did the trick and our server is now happily running on 1.21.2.

Thanks a lot everyone for the effort of finding a solution.