Upgrade katello 3.12 - puppet server error

Ondrej_Levy · June 11, 2019, 4:18pm

Problem:
During upgrade to Katello 3.12 i stuck with following error:

[DEBUG 2019-06-11T18:13:17 main] Hook /usr/share/foreman-installer/katello/hooks/post/09-version_locking.rb returned nil
[DEBUG 2019-06-11T18:13:17 main] Hook /usr/share/foreman-installer/katello/hooks/post/10-post_install.rb returned nil
[ERROR 2019-06-11T18:13:17 main] Upgrade failed during the installation phase. Fix the error and re-run the upgrade.
[DEBUG 2019-06-11T18:13:17 main] Hook /usr/share/foreman-installer/katello/hooks/post/30-upgrade.rb returned [#<Logging::Logger:0x00000002e01e68 @name="main", @parent=#<Logging::RootLogger:0x000000022c8038 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::RollingFile:0x00000002e05fe0 @roller=#<Logging::Appenders::RollingFile::Roller:0x00000002e05fb8 @fn="/var/log/foreman-installer/katello{{.%d}}.log", @roll_by=:number, @filename="/var/log/foreman-installer/katello.log", @roll=false, @keep=nil, @copy_file="/var/log/foreman-installer/katello.log._copy_", @glob="/var/log/foreman-installer/katello.*.log", @number_rgxp=/\/var\/log\/foreman-installer\/katello.(\d+).log/, @format="/var/log/foreman-installer/katello.%d.log">, @size=nil, @age_fn="/var/log/foreman-installer/katello.log.age", @age_fn_mtime=nil, @age=nil, @encoding=#<Encoding:UTF-8>, @mode="a+:UTF-8", @io=#<File:/var/log/foreman-installer/katello.log>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="configure", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x00000001c93f78 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x000000022dec98 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2019-06-11 18:08:20 +0200, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=nil>, @level=0, @write_size=500>], @additive=true, @level=0, @caller_tracing=false>, #<Logging::Logger:0x00000001cb1b68 @name="fatal", @parent=#<Logging::RootLogger:0x000000022c8038 @name="root", @appenders=[], @additive=false, @caller_tracing=false, @level=0>, @appenders=[#<Logging::Appenders::Stderr:0x00000001cc3278 @io=#<IO:<STDERR>>, @close_method=:close, @buffer=[], @immediate=[], @auto_flushing=1, @async=false, @async_flusher=nil, @flush_period=nil, @name="stderr", @closed=false, @filters=[], @mutex=#<ReentrantMutex:0x00000001cc2e40 @locker=nil>, @layout=#<Logging::Layouts::Pattern:0x000000022bd0e8 @obj_format=:string, @backtrace=true, @utc_offset=nil, @cause_depth=8, @created_at=2019-06-11 18:08:20 +0200, @date_pattern="%Y-%m-%dT%H:%M:%S", @date_method=nil, @pattern="[%5l %d %c] %m\n", @color_scheme=#<Logging::ColorScheme:0x000000022bea10 @scheme={"date"=>"\e[34m", "logger"=>"\e[36m", "line"=>"\e[33m", "file"=>"\e[33m", "method"=>"\e[33m", "info"=>"\e[32m", "warn"=>"\e[33m", "error"=>"\e[31m", "fatal"=>"\e[37m\e[41m"}, @lines=false, @levels=true>, @name_map_0=["DEBUG", "\e[32m INFO\e[0m", "\e[33m WARN\e[0m", "\e[31mERROR\e[0m", "\e[37m\e[41mFATAL\e[0m"]>, @level=0, @encoding=nil, @write_size=500>], @additive=true, @level=4, @caller_tracing=false>]
[DEBUG 2019-06-11T18:13:18 main] Hook /usr/share/foreman-installer/katello/hooks/post/31-mongo_syspath.rb returned nil
[ INFO 2019-06-11T18:13:18 main] All hooks in group post finished
[DEBUG 2019-06-11T18:13:18 main] Exit with status code: 6 (signal was 6)
[ERROR 2019-06-11T18:13:18 main] Errors encountered during run:
[ERROR 2019-06-11T18:13:18 main]  Systemd start for puppetserver failed!
[ERROR 2019-06-11T18:13:18 main] journalctl log for puppetserver:
[ERROR 2019-06-11T18:13:18 main] -- Logs begin at Tue 2019-06-11 17:39:26 CEST, end at Tue 2019-06-11 18:12:55 CEST. --
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: Exception in thread "main" java.lang.IllegalArgumentException: No matching clause: :error
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.core$main.invokeStatic(core.clj:178)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:160)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invoke(core.clj:660)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.main$_main.invokeStatic(main.clj:7)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:4)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main_opt.invokeStatic(main.clj:491)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main_opt.invoke(main.clj:487)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main.invokeStatic(main.clj:598)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main.doInvoke(main.clj:561)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main.main(main.java:37)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service: control process exited, code=exited status=1
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Failed to start puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Unit puppetserver.service entered failed state.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service failed.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service holdoff time over, scheduling restart.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Stopped puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Starting puppetserver Service...
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: Exception in thread "main" java.lang.IllegalArgumentException: No matching clause: :error
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.core$main.invokeStatic(core.clj:178)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:160)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invoke(core.clj:660)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.main$_main.invokeStatic(main.clj:7)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:4)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main_opt.invokeStatic(main.clj:491)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main_opt.invoke(main.clj:487)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main.invokeStatic(main.clj:598)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main.doInvoke(main.clj:561)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main.main(main.java:37)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service: control process exited, code=exited status=1
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: Failed to start puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: Unit puppetserver.service entered failed state.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service failed.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service holdoff time over, scheduling restart.
[ERROR 2019-06-11T18:13:18 main]
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb:188:in `rescue in start'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb:183:in `start'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:105:in `block (3 levels) in <module:Puppet>'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:491:in `set'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:568:in `sync'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:116:in `sync'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:239:in `sync'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:80:in `perform_changes'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:259:in `apply'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:279:in `eval_resource'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `call'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block (2 levels) in evaluate'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:183:in `block in evaluate'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:173:in `evaluate'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:161:in `with_destination'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:192:in `block (2 levels) in apply_catalog'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:191:in `block in apply_catalog'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `block in benchmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:230:in `benchmark'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:190:in `apply_catalog'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:376:in `run_internal'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:240:in `block in run'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:264:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:217:in `run'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:343:in `apply_catalog'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:260:in `block (2 levels) in main'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:264:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:243:in `block in main'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:264:in `override'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:207:in `main'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:177:in `run_command'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:671:in `exit_on_fail'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:139:in `run'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:77:in `execute'
[ERROR 2019-06-11T18:13:18 main] /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
[ERROR 2019-06-11T18:13:18 main]  /Stage[main]/Puppet::Server::Service/Service[puppetserver]/ensure: change from 'stopped' to 'running' failed: Systemd start for puppetserver failed!
[ERROR 2019-06-11T18:13:18 main] journalctl log for puppetserver:
[ERROR 2019-06-11T18:13:18 main] -- Logs begin at Tue 2019-06-11 17:39:26 CEST, end at Tue 2019-06-11 18:12:55 CEST. --
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: Exception in thread "main" java.lang.IllegalArgumentException: No matching clause: :error
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.core$main.invokeStatic(core.clj:178)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:160)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invoke(core.clj:660)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.main$_main.invokeStatic(main.clj:7)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:4)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main_opt.invokeStatic(main.clj:491)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main_opt.invoke(main.clj:487)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main.invokeStatic(main.clj:598)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main$main.doInvoke(main.clj:561)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:02 foreman-server.corp puppetserver[8500]: at clojure.main.main(main.java:37)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service: control process exited, code=exited status=1
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Failed to start puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Unit puppetserver.service entered failed state.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service failed.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: puppetserver.service holdoff time over, scheduling restart.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Stopped puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:03 foreman-server.corp systemd[1]: Starting puppetserver Service...
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: Exception in thread "main" java.lang.IllegalArgumentException: No matching clause: :error
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.core$main.invokeStatic(core.clj:178)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.core$main.doInvoke(core.clj:160)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invoke(core.clj:660)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.main$_main.invokeStatic(main.clj:7)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at puppetlabs.trapperkeeper.main$_main.doInvoke(main.clj:4)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.core$apply.invokeStatic(core.clj:665)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main_opt.invokeStatic(main.clj:491)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main_opt.invoke(main.clj:487)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main.invokeStatic(main.clj:598)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main$main.doInvoke(main.clj:561)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.RestFn.applyTo(RestFn.java:137)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.lang.Var.applyTo(Var.java:705)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:15 foreman-server.corp puppetserver[8638]: at clojure.main.main(main.java:37)
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service: control process exited, code=exited status=1
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: Failed to start puppetserver Service.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: Unit puppetserver.service entered failed state.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service failed.
[ERROR 2019-06-11T18:13:18 main] Jun 11 18:08:16 foreman-server.corp systemd[1]: puppetserver.service holdoff time over, scheduling restart.
[ERROR 2019-06-11T18:13:18 main] Upgrade failed during the installation phase. Fix the error and re-run the upgrade.
[DEBUG 2019-06-11T18:13:18 main] Cleaning /tmp/kafo_puppet20190611-8828-ghk7ox.conf
[DEBUG 2019-06-11T18:13:18 main] Cleaning /tmp/kafo_hiera20190611-8828-lip75
[DEBUG 2019-06-11T18:13:18 main] Cleaning /tmp/kafo_puppet20190611-8828-1il2uqc.conf
[DEBUG 2019-06-11T18:13:18 main] Cleaning /tmp/default_values.yaml
[ INFO 2019-06-11T18:13:18 main] Installer finished in 291.534776723 seconds

Foreman and Proxy versions:

Upgrade from 3.11.1 to 3.12

ekohl · June 11, 2019, 4:54pm

I took the liberty of adding code comments to your post for readability.

Which Puppet version are you using and is there anything in the Puppet logs? IIRC it’s /var/log/puppetserver/puppetserver.log.

Ondrej_Levy · June 11, 2019, 6:42pm

Hi,
versions of puppet packages are:

pulp-puppet-plugins.noarch 2.19.1-1.el7
pulp-puppet-tools.noarch 2.19.1-1.el7
puppet-agent.x86_64 6.4.2-1.el7
puppet-agent-oauth.noarch 0.5.1-3.el7
puppet-foreman_scap_client.noarch 0.3.19-1.el7
puppet6-release.noarch 6.0.0-4.el7
puppetlabs-release-pc1.noarch 1.1.0-5.el7
puppetlabs-stdlib.noarch 4.25.1-1.el7
puppetserver.noarch 6.3.0-1.el7
python-pulp-puppet-common.noarch 2.19.1-1.el7

in puppetserver log are following errors:

2019-06-11T18:13:19.376+02:00 INFO  [main] [o.e.j.u.log] Logging initialized @10308ms to org.eclipse.jetty.util.log.Slf4jLog
2019-06-11T18:13:20.592+02:00 INFO  [async-dispatch-2] [p.t.s.s.scheduler-service] Initializing Scheduler Service
2019-06-11T18:13:20.619+02:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Using default implementation for ThreadExecutor
2019-06-11T18:13:20.621+02:00 INFO  [async-dispatch-2] [o.q.s.SimpleThreadPool] Job execution threads will use class loader of thread: async-dispatch-2
2019-06-11T18:13:20.634+02:00 INFO  [async-dispatch-2] [o.q.c.SchedulerSignalerImpl] Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
2019-06-11T18:13:20.634+02:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Quartz Scheduler v.2.2.3 created.
2019-06-11T18:13:20.635+02:00 INFO  [async-dispatch-2] [o.q.s.RAMJobStore] RAMJobStore initialized.
2019-06-11T18:13:20.635+02:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Scheduler meta-data: Quartz Scheduler (v2.2.3) 'DefaultQuartzScheduler' with instanceId 'NON_CLUSTERED'
  Scheduler class: 'org.quartz.core.QuartzScheduler' - running locally.
  NOT STARTED.
  Currently in standby mode.
  Number of jobs executed: 0
  Using thread pool 'org.quartz.simpl.SimpleThreadPool' - with 10 threads.
  Using job-store 'org.quartz.simpl.RAMJobStore' - which does not support persistence. and is not clustered.

2019-06-11T18:13:20.635+02:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
2019-06-11T18:13:20.635+02:00 INFO  [async-dispatch-2] [o.q.i.StdSchedulerFactory] Quartz scheduler version: 2.2.3
2019-06-11T18:13:20.635+02:00 INFO  [async-dispatch-2] [o.q.c.QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
2019-06-11T18:13:20.638+02:00 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2019-06-11T18:13:20.657+02:00 ERROR [async-dispatch-2] [p.t.internal] Error during service init!!!
clojure.lang.ExceptionInfo: Value does not match schema: (not (map? nil))
	at schema.core$validator$fn__2529.invoke(core.clj:155)
	at schema.core$validate.invokeStatic(core.clj:164)
	at schema.core$validate.invoke(core.clj:159)
	at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__26168$service_fnk__4991__auto___positional$reify__26175.init(metrics_service.clj:24)
	at puppetlabs.trapperkeeper.services$fn__4815$G__4807__4818.invoke(services.clj:9)
	at puppetlabs.trapperkeeper.services$fn__4815$G__4806__4822.invoke(services.clj:9)
	at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560$fn__13561.invoke(internal.clj:198)
	at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560.invoke(internal.clj:181)
	at puppetlabs.trapperkeeper.internal$fn__13582$run_lifecycle_fns__13587$fn__13588.invoke(internal.clj:231)
	at puppetlabs.trapperkeeper.internal$fn__13582$run_lifecycle_fns__13587.invoke(internal.clj:208)
	at puppetlabs.trapperkeeper.internal$fn__14155$build_app_STAR___14164$fn$reify__14176.init(internal.clj:583)
	at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210$fn__14211$fn__14213.invoke(internal.clj:611)
	at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210$fn__14211.invoke(internal.clj:610)
	at puppetlabs.trapperkeeper.internal$fn__14203$boot_services_for_app_STAR__STAR___14210.invoke(internal.clj:604)
	at clojure.core$partial$fn__5826.invoke(core.clj:2630)
	at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817.invoke(internal.clj:251)
	at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814.invoke(internal.clj:251)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at clojure.core.async$ioc_alts_BANG_$fn__11006.invoke(async.clj:384)
	at clojure.core.async$do_alts$fn__10946$fn__10949.invoke(async.clj:253)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__6246$fn__6247.invoke(channels.clj:95)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-06-11T18:13:20.663+02:00 INFO  [main] [p.t.internal] Beginning shutdown sequence
2019-06-11T18:13:20.673+02:00 INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopping metrics sampler job
2019-06-11T18:13:20.674+02:00 INFO  [async-dispatch-3] [p.s.j.jruby-metrics-service] JRuby Metrics Service: stopped metrics sampler job
2019-06-11T18:13:20.688+02:00 ERROR [async-dispatch-3] [p.t.internal] Encountered error during shutdown sequence
clojure.lang.ExceptionInfo: Input to stop-all does not match schema:

	   [(named {:registries missing-required-key, :can-update-registry-settings? missing-required-key, :registry-settings missing-required-key, :metrics-config missing-required-key} service-context)]


	at puppetlabs.trapperkeeper.services.metrics.metrics_core$fn__26115$stop_all__26120.invoke(metrics_core.clj:295)
	at puppetlabs.trapperkeeper.services.metrics.metrics_service$reify__26168$service_fnk__4991__auto___positional$reify__26175.stop(metrics_service.clj:36)
	at puppetlabs.trapperkeeper.services$fn__4841$G__4811__4844.invoke(services.clj:9)
	at puppetlabs.trapperkeeper.services$fn__4841$G__4810__4848.invoke(services.clj:9)
	at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560$fn__13561.invoke(internal.clj:198)
	at puppetlabs.trapperkeeper.internal$fn__13553$run_lifecycle_fn_BANG___13560.invoke(internal.clj:181)
	at puppetlabs.trapperkeeper.internal$fn__14063$shutdown_BANG___14068$fn__14069$shutdown_fn__14071$fn__14086.invoke(internal.clj:440)
	at puppetlabs.trapperkeeper.internal$fn__14063$shutdown_BANG___14068$fn__14069$shutdown_fn__14071.invoke(internal.clj:439)
	at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817$fn__13831.invoke(internal.clj:276)
	at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814$fn__13817.invoke(internal.clj:260)
	at puppetlabs.trapperkeeper.internal$fn__13627$initialize_lifecycle_worker__13638$fn__13639$fn__13789$state_machine__10791__auto____13814.invoke(internal.clj:251)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invokeStatic(ioc_macros.clj:973)
	at clojure.core.async.impl.ioc_macros$run_state_machine.invoke(ioc_macros.clj:972)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invokeStatic(ioc_macros.clj:977)
	at clojure.core.async.impl.ioc_macros$run_state_machine_wrapped.invoke(ioc_macros.clj:975)
	at clojure.core.async$ioc_alts_BANG_$fn__11006.invoke(async.clj:384)
	at clojure.core.async$do_alts$fn__10946$fn__10949.invoke(async.clj:253)
	at clojure.core.async.impl.channels.ManyToManyChannel$fn__6262.invoke(channels.clj:135)
	at clojure.lang.AFn.run(AFn.java:22)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2019-06-11T18:13:20.692+02:00 INFO  [async-dispatch-3] [p.t.s.w.jetty9-service] Shutting down web server(s).
2019-06-11T18:13:20.698+02:00 INFO  [async-dispatch-3] [p.t.s.s.scheduler-service] Shutting down Scheduler Service
2019-06-11T18:13:20.700+02:00 INFO  [clojure-agent-send-off-pool-0] [o.q.c.QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutting down.
2019-06-11T18:13:20.700+02:00 INFO  [clojure-agent-send-off-pool-0] [o.q.c.QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED paused.
2019-06-11T18:13:21.124+02:00 INFO  [clojure-agent-send-off-pool-0] [o.q.c.QuartzScheduler] Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED shutdown complete.
2019-06-11T18:13:21.124+02:00 INFO  [async-dispatch-3] [p.t.s.s.scheduler-service] Scheduler Service shutdown complete.
2019-06-11T18:13:21.128+02:00 INFO  [main] [p.t.internal] Finished shutdown sequence

ekohl · June 11, 2019, 6:59pm

You can remove this, but shouldn’t be an issue

Can you check the content of /etc/puppetlabs/puppetserver/services.d/ca.cfg? I would expect puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service to be there, but if it’s not, then that’s likely the issue.

This should be guarded by the puppetserver version. It should be undef or correct. Can you check what the value of server_puppetserver_version is in your answers file (/etc/foreman-installer/scenarios.d/katello-answers.yaml).

Ondrej_Levy · June 12, 2019, 6:44am

Hello,
/etc/puppetlabs/puppetserver/services.d/ca.cfg seems to be ok

#
# Managed by Puppet
#
# To enable the CA service, leave the following line uncommented
puppetlabs.services.ca.certificate-authority-service/certificate-authority-service
# To disable the CA service, comment out the above line and uncomment the line below
#puppetlabs.services.ca.certificate-authority-disabled-service/certificate-authority-disabled-service
puppetlabs.trapperkeeper.services.watcher.filesystem-watch-service/filesystem-watch-service

server_puppetserver_version in /etc/foreman-installer/scenarios.d/katello-answers.yaml was

 server_puppetserver_dir: /etc/puppetlabs/puppetserver
  server_puppetserver_vardir: /opt/puppetlabs/server/data/puppetserver
  server_puppetserver_rundir: /var/run/puppetlabs/puppetserver
  server_puppetserver_logdir: /var/log/puppetlabs/puppetserver
  server_puppetserver_version:

i chage it to
server_puppetserver_version: 6.3.0

but re-run of “foreman-installer --scenario katello --upgrade” retrun same errors.

ekohl · June 12, 2019, 7:00am

That does look correct. If the version is not specified, it’s derived from the installed puppetversion so it’d be best to keep it empty for future upgrades.

You’re also sure foreman-installer is version 1.22?

senetm · June 12, 2019, 7:05am

I had the same problem and fixed it with:
Erase puppetlabs repo … yum erase -y puppetlabs*
Install puppet6… yum -y localinstall https://yum.puppet.com/puppet6-release-el-7.noarch.rpm

After update i have resetted the parameters for puppet:
yum reinstall puppet-agent-oauth

foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental

And after that i have successfully upgraded Foreman-katello.

Ondrej_Levy · June 12, 2019, 8:53am

@senetm Thanks. It helps a lot. Now upgrade works without problem.

svenvogel · June 25, 2019, 9:19am

@senetm thanks for the info. it seems its okay when we run only the command without upgrade of puppet or reinstall of the oauth agent. it fixed my problem too.

foreman-installer --reset-puppet-server-jvm-extra-args --reset-puppet-server-puppetserver-version --reset-puppet-server-puppetserver-metrics --reset-puppet-server-puppetserver-experimental

foreman-installer --scenario katello --upgrade

Netbulae1 · July 23, 2019, 9:11am

Same issue here, thanks for the fix! I didn’t need to reinstall puppet-agent-oauth.

Maybe the upgrade documentation needs to be enhanced with this info?

Netbulae1 · July 23, 2019, 10:34am

Also the same issue with the capsule upgrade, same fix.

Reedler · January 22, 2020, 2:17pm

Thanks! This worked for me!