Update to Foreman 1.24 with Katello 3.14 bricks puppet

Support
1

Problem:
Upgrading to Foreman 1.24 with Katello 3.14 bricks pupppet. While running command
“foreman-installer --scenario katello --upgrade” the service for puppet.server throws an error.

Expected outcome:
Puppetserver should start and “foreman-installer --scenario katello --upgrade” exit successfully.

Foreman and Proxy versions:
Foreman 1.24 with Katello 3.14

Foreman and Proxy plugin versions:
foreman-tasks 0.17.5
foreman_ansible 4.0.3
foreman_discovery 16.0.1
foreman_docker 5.0.0
foreman_hooks 0.3.15
foreman_remote_execution 2.0.6
katello 3.14.0

Distribution and version:
CentOS 7.6

Other relevant data:

puppetserver.log 
2020-01-03 13:50:29,892 INFO  [main] [o.e.j.u.log] Logging initialized @13523ms to org.eclipse.jetty.util.log.Slf4jLog
2020-01-03 13:50:32,289 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2020-01-03 13:50:32,313 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
clojure.lang.ExceptionInfo: Value does not match schema: (not (map? nil))
        at schema.core$validator$fn__2823.invoke(core.clj:155)
        at schema.core$validate.invokeStatic(core.clj:164)
        at schema.core$validate.invoke(core.clj:159)
        at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__42253$service_fnk__5266__auto___positional$reify__42260.init(metrics_service.clj:24)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5069$G__5056__5072.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5069$G__5055__5076.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842$fn__13843.invoke(internal.clj:198)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842.invoke(internal.clj:181)
        at puppetlabs.trapperkeeper.internal$eval13864$run_lifecycle_fns__13869$fn__13870.invoke(internal.clj:231)
        at puppetlabs.trapperkeeper.internal$eval13864$run_lifecycle_fns__13869.invoke(internal.clj:208)
        at puppetlabs.trapperkeeper.internal$eval14346$build_app_STAR___14355$fn$reify__14365.init(internal.clj:583)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399$fn__14400$fn__14402.invoke(internal.clj:611)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399$fn__14400.invoke(internal.clj:610)
        at puppetlabs.trapperkeeper.internal$eval14392$boot_services_for_app_STAR__STAR___14399.invoke(internal.clj:604)
        at clojure.core$partial$fn__4761.invoke(core.clj:2521)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014$fn__14016.invoke(internal.clj:251)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014.invoke(internal.clj:251)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:1012)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:1011)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:1016)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:1014)
        at clojure.core.async$ioc_alts_BANG_$fn__12043.invoke(async.clj:383)
        at clojure.core.async$do_alts$fn__11989$fn__11992.invoke(async.clj:252)
        at clojure.core.async.impl.channels.ManyToManyChannel$fn__6799$fn__6800.invoke(channels.clj:95)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-01-03 13:50:32,321 INFO  [main] [p.t.internal] Beginning shutdown sequence
2020-01-03 13:50:32,332 INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopping metrics sampler job
2020-01-03 13:50:32,333 INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopped metrics sampler job
2020-01-03 13:50:32,343 ERROR [async-dispatch-3] [p.t.internal] Encountered error during shutdown sequence
clojure.lang.ExceptionInfo: Input to stop-all does not match schema: [(named {:registries missing-required-key, :can-update-registry-settings? missing-required-key, :registry-settings missing-required-key, :metrics-config missing-required-key} service-context)]
        at puppetlabs.trapperkeeper.services.metrics.metrics_core$eval42202$stop_all__42207.invoke(metrics_core.clj:295)
        at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__42253$service_fnk__5266__auto___positional$reify__42260.stop(metrics_service.clj:36)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5095$G__5060__5098.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.services$eval5068$fn__5095$G__5059__5102.invoke(services.clj:9)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842$fn__13843.invoke(internal.clj:198)
        at puppetlabs.trapperkeeper.internal$eval13835$run_lifecycle_fn_BANG___13842.invoke(internal.clj:181)
        at puppetlabs.trapperkeeper.internal$eval14258$shutdown_BANG___14263$fn__14264$shutdown_fn__14266$fn__14277.invoke(internal.clj:440)
        at puppetlabs.trapperkeeper.internal$eval14258$shutdown_BANG___14263$fn__14264$shutdown_fn__14266.invoke(internal.clj:439)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014$fn__14016$fn__14029.invoke(internal.clj:276)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014$fn__14016.invoke(internal.clj:260)
        at puppetlabs.trapperkeeper.internal$eval13905$initialize_lifecycle_worker__13916$fn__13917$fn__14013$state_machine__11875__auto____14014.invoke(internal.clj:251)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:1012)
        at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:1011)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:1016)
        at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:1014)
        at clojure.core.async$ioc_alts_BANG_$fn__12043.invoke(async.clj:383)
        at clojure.core.async$do_alts$fn__11989$fn__11992.invoke(async.clj:252)
        at clojure.core.async.impl.channels.ManyToManyChannel$fn__6813.invoke(channels.clj:135)
        at clojure.lang.AFn.run(AFn.java:22)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
2020-01-03 13:50:32,347 INFO  [async-dispatch-3] [p.t.s.w.jetty9-service] Shutting down web server(s).
2020-01-03 13:50:32,360 INFO  [main] [p.t.internal] Finished shutdown sequence
2

Can you post the Puppetserver version?

3

Hi and welcome to the forum :slight_smile:

I’ve found an old thread with the same problem and a solution to it:

From what I understand, this problem occures if puppetserver and puppetagent versions do not match during installation/upgrade. In that case, you would have to force the desired puppetserver version.
Is that the case for you? If so, I would recommend checking the described solution from the linked toppic and report back.

Regards

4

The versions are:
puppetserver-5.3.10-1.el7.noarch
puppet-agent-5.5.17-1.el7.x86_64

But will take first a look on areyus solution.

5

Well, will this work?
Because “–puppet-server-puppetserver-version” does not appear as an option if I call “foreman-installer --help”.

6

foreman-installer --help only shows the most common/usefull options. There is foreman-installer --full-help (as mentioned by --help in the last line of output), that should show the option. I do not have a 1.24 box at hand to check, but I doubt that option was removed.

7

You are right, I forgot that.
As in the link you posted, I tried this command: foreman-installer --puppet-server-puppetserver-version 5.3.10
Unfortunately, I have still the same problem.

8

Do you have metrics enabled? It may be that we don’t test with metrics and generate an invalid config for Puppet 5.

9

As far as I know, I did not enable metrics.
How do I check the status of metrics?

10

I know, community driven software, but bump.
Want to know what causes this issue and how to avoid it in the future.

11

Hi,

here in the Puppet docs is a description of how to enable and configure metrics.
No clue if that causes your problem, but since the error references metrics and it is not tested against as @ekohl mentioned, it is worth a look.

12

Thanks, but no luck with that.
Searching for metrics.conf gave zero hits and the only value for metrics I found is in the
puppetserver.conf but disabled “metrics-enabled: false”.
Any other ideas?

13

I personally have no good idea, but in the old thread I linked, the final solution was to actually enable metrics via the foreman-installer answers file.
This seems counter-intuitive to me, but maybe Puppetserver requires them under certain circumstances? I just checked and it looks like we have metrics “somewhat” enabled. At least we have a metrics.conf and jmx metrics are enabled. In the installer answers file we have:

 server_puppetserver_metrics: false
  server_metrics_jmx_enable: true
  server_metrics_graphite_enable: false

Maybe this helps.