Foreman-installer fails to upgrade foreman. rh-mongodb34-mongod can't start

Problem:
rh-mongodb34-mongod fails to start.

May 25 07:20:03 foreman.bne-home.net mongod.27017[9357]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: ERROR: child process failed, exited with error number 1

Expected outcome:
rh-mongodb34-mongod would start

Foreman and Proxy versions:

[root@foreman run]# rpm -qa | grep foreman-release
foreman-release-scl-4-1.el7.noarch
foreman-release-1.18.0-0.develop.201805221751gita653e38.el7.noarch

Foreman and Proxy plugin versions:
katello-3.7.0-4.nightly.el7.noarch

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]

[root@foreman run]# foreman-installer
Resetting puppet server version param...
 Systemd start for rh-mongodb34-mongod failed!
journalctl log for rh-mongodb34-mongod:
-- Logs begin at Thu 2018-05-24 20:39:44 AEST, end at Fri 2018-05-25 07:20:03 AEST. --
May 25 07:20:03 foreman.bne-home.net systemd[1]: Starting High-performance, schema-free document-oriented database...
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: about to fork child process, waiting until server is ready for connections.
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: forked process: 9357
May 25 07:20:03 foreman.bne-home.net mongod.27017[9357]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: ERROR: child process failed, exited with error number 1
May 25 07:20:03 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service: control process exited, code=exited status=1
May 25 07:20:03 foreman.bne-home.net systemd[1]: Failed to start High-performance, schema-free document-oriented database.
May 25 07:20:03 foreman.bne-home.net systemd[1]: Unit rh-mongodb34-mongod.service entered failed state.
May 25 07:20:03 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service failed.

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb:167:in `rescue in start'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/systemd.rb:163:in `start'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:103:in `block (3 levels) in <module:Puppet>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:491:in `set'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property.rb:568:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:114:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:236:in `sync'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:134:in `sync_if_needed'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:80:in `perform_changes'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:21:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
 /Stage[main]/Mongodb::Server::Service/Service[mongodb]/ensure: change from 'stopped' to 'running' failed: Systemd start for rh-mongodb34-mongod failed!
journalctl log for rh-mongodb34-mongod:
-- Logs begin at Thu 2018-05-24 20:39:44 AEST, end at Fri 2018-05-25 07:20:03 AEST. --
May 25 07:20:03 foreman.bne-home.net systemd[1]: Starting High-performance, schema-free document-oriented database...
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: about to fork child process, waiting until server is ready for connections.
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: forked process: 9357
May 25 07:20:03 foreman.bne-home.net mongod.27017[9357]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
May 25 07:20:03 foreman.bne-home.net mongodb-scl-helper[9352]: ERROR: child process failed, exited with error number 1
May 25 07:20:03 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service: control process exited, code=exited status=1
May 25 07:20:03 foreman.bne-home.net systemd[1]: Failed to start High-performance, schema-free document-oriented database.
May 25 07:20:03 foreman.bne-home.net systemd[1]: Unit rh-mongodb34-mongod.service entered failed state.
May 25 07:20:03 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service failed.
 /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[foreman.bne-home.net]: Could not evaluate: Exception Failed to open TCP connection to foreman.bne-home.net:443 (Connection refused - connect(2) for "foreman.bne-home.net" port 443) in get request to: https://foreman.bne-home.net/api/v2/smart_proxies?search=name=%22foreman.bne-home.net%22
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:89:in `rescue in request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:71:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1070:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1098:in `retrieve_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:302:in `from_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Wrapped exception:
Failed to open TCP connection to foreman.bne-home.net:443 (Connection refused - connect(2) for "foreman.bne-home.net" port 443)
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:906:in `rescue in block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:903:in `block in connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/timeout.rb:93:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/timeout.rb:103:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:902:in `connect'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:887:in `do_start'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:876:in `start'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1407:in `request'
/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/oauth-0.5.1/lib/oauth/consumer.rb:161:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_resource/rest_v3.rb:76:in `request'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:6:in `proxy'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
/usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1070:in `retrieve'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1098:in `retrieve_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:302:in `from_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/resource_harness.rb:20:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:257:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:277:in `eval_resource'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block (2 levels) in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:181:in `block in evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/graph/relationship_graph.rb:121:in `traverse'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction.rb:171:in `evaluate'
/usr/share/gems/gems/kafo-2.1.0/modules/kafo_configure/lib/puppet/parser/functions/add_progress.rb:30:in `evaluate_with_trigger'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:239:in `block (2 levels) in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:238:in `block in apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/log.rb:156:in `with_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/transaction/report.rb:146:in `as_logging_destination'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/catalog.rb:237:in `apply'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:186:in `block (2 levels) in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:513:in `block in thinmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:512:in `thinmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:185:in `block in apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:232:in `block in benchmark'
/opt/puppetlabs/puppet/lib/ruby/2.4.0/benchmark.rb:308:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:231:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:184:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:369:in `run_internal'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:237:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:211:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:354:in `apply_catalog'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block (2 levels) in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:280:in `block in main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:260:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:233:in `main'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:174:in `run_command'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `block in run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:661:in `exit_on_fail'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:375:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:137:in `run'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'
/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'
Installing             Done                                               [100%] [..........................................................................................................................................................]
  Something went wrong! Check the log for ERROR-level output
  The full log is at /var/log/foreman-installer/katello.log

Also tried changing the permissions of the directory it can’t write the PID to

[root@foreman run]# ls -al /var/opt/rh/rh-mongodb34/run/mongodb/
total 0
drwxr-xr-x. 2 mongodb root     6 May 25 06:43 .
drw-r--r--. 3 mongodb mongodb 21 May 16 20:51 ..

It’s still unable to write the PID with full permissions to the directory

[root@foreman run]# sudo -u mongodb touch /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid
touch: cannot touch ‘/var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid’: Permission denied

Making it 777 obviously works for that issue, which isn’t a long-term fix, but then I get a new issue:

[root@foreman run]# systemctl status rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: failed (Result: exit-code) since Fri 2018-05-25 07:30:36 AEST; 4s ago
  Process: 12385 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=100)

May 25 07:30:36 foreman.bne-home.net mongodb-scl-helper[12385]: about to fork child process, waiting until server is ready for connections.
May 25 07:30:36 foreman.bne-home.net mongodb-scl-helper[12385]: forked process: 12390
May 25 07:30:36 foreman.bne-home.net mongod.27017[12390]: [initandlisten] MongoDB starting : pid=12390 port=27017 dbpath=/var/lib/mongodb 64-bit host=foreman.bne-home.net
May 25 07:30:36 foreman.bne-home.net mongod.27017[12390]: [initandlisten] db version v3.4.9
May 25 07:30:36 foreman.bne-home.net mongod.27017[12390]: [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
May 25 07:30:36 foreman.bne-home.net mongodb-scl-helper[12385]: ERROR: child process failed, exited with error number 100
May 25 07:30:36 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service: control process exited, code=exited status=100
May 25 07:30:36 foreman.bne-home.net systemd[1]: Failed to start High-performance, schema-free document-oriented database.
May 25 07:30:36 foreman.bne-home.net systemd[1]: Unit rh-mongodb34-mongod.service entered failed state.
May 25 07:30:36 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service failed.

After setting the $OPTIONS and $RH_MONGODB34_SCLS_ENABLED variables and manually executing the ExecStart line:

[root@foreman run]# echo $OPTIONS
-f /etc/opt/rh/rh-mongodb34/mongod.conf
[root@foreman run]# echo $RH_MONGODB34_SCLS_ENABLED
rh-mongodb34
[root@foreman run]# /opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run
about to fork child process, waiting until server is ready for connections.
forked process: 12834
ERROR: child process failed, exited with error number 100
[root@foreman run]# 

Trying to run mongod on it’s own:

[root@foreman run]# /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run
/opt/rh/rh-mongodb34/root/usr/bin/mongod: error while loading shared libraries: libyaml-cpp.so.rh-mongodb34-0.5: cannot open shared object file: No such file or directory

This library does exist within the chroot environment for mongo:

[root@foreman run]# find / -name libyaml-cpp.so.rh-mongodb34-0.5
/opt/rh/rh-mongodb34/root/usr/lib64/libyaml-cpp.so.rh-mongodb34-0.5

Howdy,

Can you clarify what scenario you are doing? This reads like you are attempting to upgrade from a previous version of Foreman and Katello to the unstable nightly versions? This is generally not a supported or recommended path. What version are you attempting to upgrade from?

Hey Eric. It was installed initially as the nightly version. To upgrade, I just run yum update -y and then run through the upgrade steps.

Is the additional foreman-release rpm what you’re seeing as an issue? I thought it looked a bit odd and i may have taken a wrong turn somewhere to end up with that package.

Did you run foreman-installer --upgrade ?

I did, yeah. I can grab you the output for that as well. It’s very similar to the output of foreman-installer on it’s own. I’ll run it now and post the output for you.

Is mongod running?

It is:

[root@foreman ~]# systemctl status mongod
● mongod.service - High-performance, schema-free document-oriented database
Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2018-05-24 20:40:22 AEST; 13h ago
Main PID: 1238 (mongod)
CGroup: /system.slice/mongod.service
└─1238 /usr/bin/mongod --quiet -f /etc/mongod.conf run

May 24 20:40:21 foreman.bne-home.net systemd[1]: Starting High-performance, schema-free…
May 24 20:40:21 foreman.bne-home.net mongod[1117]: about to fork child process, waiting…s.
May 24 20:40:21 foreman.bne-home.net mongod[1117]: forked process: 1238
May 24 20:40:22 foreman.bne-home.net systemd[1]: Started High-performance, schema-free …e.
Hint: Some lines were ellipsized, use -l to show in full.

foreman-installer–upgrade.logs.tar (2.2 MB)

Try stopping mongod and re-running the installer. I can’t say for sure if all necessary packages were in nightly yet to support an upgrade.

Gave it a yum update which grabbed a new copy of foreman-installer. Still having the same problem though. Would you like the logs from the new attempt? They appear to be the same as the previous ones though.

Do you think this has anything to do with it? Or am I just grasping at straws here.

[root@foreman ~]# /opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable rh-mongodb34 – /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run
about to fork child process, waiting until server is ready for connections.
forked process: 4751
ERROR: child process failed, exited with error number 100
[root@foreman ~]# /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run
/opt/rh/rh-mongodb34/root/usr/bin/mongod: error while loading shared libraries: libyaml-cpp.so.rh-mongodb34-0.5: cannot open shared object file: No such file or directory

Probably grasping at straws. I just set systemLog.quiet: false
in /etc/opt/rh/rh-mongodb34/mongod.conf and tried again. I got this:

[root@foreman ~]# journalctl -f
-- Logs begin at Thu 2018-05-24 20:39:44 AEST. --
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] modules: none
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] build environment:
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten]     distarch: x86_64
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten]     target_arch: x86_64
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] options: { command: [ "run" ], config: "/etc/opt/rh/rh-mongodb34/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", engine: "mmapv1", journal: { enabled: true } }, systemLog: { destination: "syslog", quiet: false } }
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] exception in initAndListen: 98 Unable to lock file: /var/lib/mongodb/mongod.lock Resource temporarily unavailable. Is a mongod instance already running?, terminating
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] shutdown: going to close listening sockets...
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] shutdown: going to flush diaglog...
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] now exiting
May 25 18:19:07 foreman.bne-home.net mongod.27017[5209]: [initandlisten] shutting down with code:100
^C
[root@foreman ~]# 
[root@foreman ~]# 
[root@foreman ~]# systemctl status mongod
● mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-05-25 17:53:04 AEST; 26min ago
 Main PID: 12473 (mongod)
   CGroup: /system.slice/mongod.service
           └─12473 /usr/bin/mongod --quiet -f /etc/mongod.conf run

May 25 17:53:03 foreman.bne-home.net systemd[1]: Starting High-performance, schema-free document-oriented database...
May 25 17:53:04 foreman.bne-home.net mongod[12471]: about to fork child process, waiting until server is ready for connections.
May 25 17:53:04 foreman.bne-home.net mongod[12471]: forked process: 12473
May 25 17:53:04 foreman.bne-home.net mongod[12471]: child process started successfully, parent exiting
May 25 17:53:04 foreman.bne-home.net systemd[1]: Started High-performance, schema-free document-oriented database.
[root@foreman ~]# systemctl stop mongod
[root@foreman ~]# /opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable rh-mongodb34 -- /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run
about to fork child process, waiting until server is ready for connections.
forked process: 5307
child process started successfully, parent exiting
[root@foreman ~]# 

Makes sense why you asked about mongod running. It must have been started again during the foreman-installer process. Still getting the same error when I try to run foreman-installer though. But it’s progress.

Looks like it broke the permissions again for it’s .pid directory

-- Unit rh-mongodb34-mongod.service has begun starting up.
May 25 18:23:18 foreman.bne-home.net mongodb-scl-helper[7162]: about to fork child process, waiting until server is ready for connections.
May 25 18:23:18 foreman.bne-home.net mongodb-scl-helper[7162]: forked process: 7167
May 25 18:23:18 foreman.bne-home.net mongod.27017[7167]: [main] ERROR: Cannot write pid file to /var/opt/rh/rh-mongodb34/run/mongodb/mongod.pid: Permission denied
May 25 18:23:18 foreman.bne-home.net mongodb-scl-helper[7162]: ERROR: child process failed, exited with error number 1
May 25 18:23:18 foreman.bne-home.net systemd[1]: rh-mongodb34-mongod.service: control process exited, code=exited status=1
May 25 18:23:18 foreman.bne-home.net systemd[1]: Failed to start High-performance, schema-free document-oriented database.
-- Subject: Unit rh-mongodb34-mongod.service has failed

Definitely seems to be related to permissions during the foreman-installer process. The permissions are being changed during that process that stops it being able to write to the places it needs to write to. It seems a bit odd as all directories are owned by mongodb.

May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] MongoDB starting : pid=11586 port=27017 dbpath=/var/lib/mongodb 64-bit host=foreman.bne-home.net
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] db version v3.4.9
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] git version: 876ebee8c7dd0e2d992f36a848ff4dc50ee6603e
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips  26 Jan 2017
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] allocator: tcmalloc
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] modules: none
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] build environment:
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten]     distarch: x86_64
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten]     target_arch: x86_64
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] options: { command: [ "run" ], config: "/etc/opt/rh/rh-mongodb34/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/op
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty.
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten]
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 4096KB
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] **          We suggest setting it to 256KB (512 sectors) or less
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] **          http://dochub.mongodb.org/core/readahead
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] journal dir=/var/lib/mongodb/journal
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] recover begin
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] info no lsn file in journal/ directory
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] recover lsn: 0
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] recover /var/lib/mongodb/journal/j._0
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] couldn't open /var/lib/mongodb/journal/j._0 Permission denied
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] Assertion: 13544:recover error couldn't open /var/lib/mongodb/journal/j._0 src/mongo/db/storage/mmap_v1/dur_recover.cpp 561
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] 
                                                           0x56199a31235a 0x56199a297d79 0x561999572dcb 0x561999572eb9 0x561999f8277b 0x561999f82e34 0x561999f843df 0x561999f84c90 0x561999f62790 0x561999edb865 0x561999558984 0x56199957ce4
                                                          ----- BEGIN BACKTRACE -----
                                                          {"backtrace":[{"b":"56199936F000","o":"FA335A"},{"b":"56199936F000","o":"F28D79"},{"b":"56199936F000","o":"203DCB"},{"b":"56199936F000","o":"203EB9"},{"b":"56199936F000","o":"C137
                                                           mongod(+0xFA335A) [0x56199a31235a]
                                                           mongod(+0xF28D79) [0x56199a297d79]
                                                           mongod(+0x203DCB) [0x561999572dcb]
                                                           mongod(+0x203EB9) [0x561999572eb9]
                                                           mongod(+0xC1377B) [0x561999f8277b]
                                                           mongod(+0xC13E34) [0x561999f82e34]
                                                           mongod(+0xC153DF) [0x561999f843df]
                                                           mongod(+0xC15C90) [0x561999f84c90]
                                                           mongod(+0xBF3790) [0x561999f62790]
                                                           mongod(+0xB6C865) [0x561999edb865]
                                                           mongod(+0x1E9984) [0x561999558984]
                                                           mongod(+0x20DE44) [0x56199957ce44]
                                                           libc.so.6(__libc_start_main+0xF5) [0x7f5981843445]
                                                           mongod(+0x26B615) [0x5619995da615]
                                                          -----  END BACKTRACE  -----
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] dbexception during recovery: 13544 recover error couldn't open /var/lib/mongodb/journal/j._0
May 25 18:40:10 foreman.bne-home.net mongodb-scl-helper[11581]: ERROR: child process failed, exited with error number 100
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] exception in initAndListen: 13544 recover error couldn't open /var/lib/mongodb/journal/j._0, terminating
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] shutdown: going to close listening sockets...
May 25 18:40:10 foreman.bne-home.net mongod.27017[11586]: [initandlisten] removing socket file: /tmp/mongodb-27017.sock

I gave the directory a chown -R mongodb:mongodb and it looks like it has resolved the issue:

[root@foreman ~]# sudo chown -R mongodb:mongodb /var/lib/mongodb
[root@foreman ~]# systemctl start rh-mongodb34-mongod
[root@foreman ~]# systemctl status rh-mongodb34-mongod
● rh-mongodb34-mongod.service - High-performance, schema-free document-oriented database
   Loaded: loaded (/usr/lib/systemd/system/rh-mongodb34-mongod.service; enabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-05-25 18:48:28 AEST; 7s ago
  Process: 12866 ExecStart=/opt/rh/rh-mongodb34/root/usr/libexec/mongodb-scl-helper enable $RH_MONGODB34_SCLS_ENABLED -- /opt/rh/rh-mongodb34/root/usr/bin/mongod $OPTIONS run (code=exited, status=0/SUCCESS)
 Main PID: 12871 (mongod)
   CGroup: /system.slice/rh-mongodb34-mongod.service
           └─12871 /opt/rh/rh-mongodb34/root/usr/bin/mongod -f /etc/opt/rh/rh-mongodb34/mongod.conf run

May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten] **        We suggest setting it to 'never'
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten]
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten] **        We suggest setting it to 'never'
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten]
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [initandlisten] Initializing full-time diagnostic data capture with directory '/var/lib/mongodb/diagnostic.data'
May 25 18:48:28 foreman.bne-home.net mongod.27017[12871]: [thread1] waiting for connections on port 27017
May 25 18:48:28 foreman.bne-home.net mongodb-scl-helper[12866]: child process started successfully, parent exiting
May 25 18:48:28 foreman.bne-home.net systemd[1]: Started High-performance, schema-free document-oriented database.
May 25 18:48:29 foreman.bne-home.net mongod.27017[12871]: [ftdc] Unclean full-time diagnostic data capture shutdown detected, found interim file, some metrics may have been lost. OK
[root@foreman ~]# 
[root@foreman ~]# 
[root@foreman ~]# 
[root@foreman ~]# 
[root@foreman ~]# foreman-installer 
Resetting puppet server version param...
Installing             Done                                               [100%] [..........................................................................................................................................................]
  Success!
  * Katello is running at https://foreman.bne-home.net
  * To install an additional Foreman proxy on separate machine continue by running:

      foreman-proxy-certs-generate --foreman-proxy-fqdn "$FOREMAN_PROXY" --certs-tar "/root/$FOREMAN_PROXY-certs.tar"
  The full log is at /var/log/foreman-installer/katello.log

This allowed the services to all start again. However the UI never loads any of the dashboards or lets me go to any of the tabs.

Running foreman-installer --upgrade breaks the permissions again.