Why is forklift / foreman-install most of the time broken?

Development
1

I tried now yet again to use forklift to setup a katello environment and yet, the installer does not even run through. This time it is definitely some other error then last time that did not work, but basically, it is always failing then I try it.
Working with forklift is a pain anyways, I can’t understand how you endure this. Ruby is always slow, but the way you use vagrant together with yaml, makes is just unbearable slow.
time vagrant help
real 0m32.467s
That is the penalty for each vagrant command. Seriously ?

I tried to follow the original documentation and all I get is a error like this:

TASK [foreman_installer : Join options] ****************************************
ok: [centos8-katello-devel2]
 [started TASK: foreman_installer : Replace REX Script with REX SSH options for pre-3.3 on centos8-katello-devel2]
 [started TASK: foreman_installer : Run installer on centos8-katello-devel2]

TASK [foreman_installer : Run installer] ***************************************
fatal: [centos8-katello-devel2]: FAILED! => changed=true 
  cmd: |-
    foreman-installer -v --no-colors --disable-system-checks --scenario katello-devel --katello-devel-extra-plugins theforeman/foreman_remote_execution --katello-devel-extra-plugins theforeman/foreman_discovery --foreman-proxy-content-enable-ostree=true  --katello-devel-admin-password changeme --katello-devel-github-username=poelzi --katello-devel-extra-plugins theforeman/foreman_remote_execution
  delta: '0:03:57.398939'
  end: '2022-04-26 19:52:34.796125'
  msg: non-zero return code
  rc: 1
  start: '2022-04-26 19:48:37.397186'
  stderr: ''
  stderr_lines: <omitted>
  stdout: |-
    2022-04-26 19:48:42 [NOTICE] [root] Loading installer configuration. This will take some time.
    2022-04-26 19:48:47 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
    2022-04-26 19:48:47 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
    2022-04-26 19:48:49 [WARN  ] [pre] Skipping system checks.
    2022-04-26 19:48:49 [WARN  ] [pre] Skipping system checks.
    2022-04-26 19:52:28 [NOTICE] [configure] Starting system configuration.
    2022-04-26 19:52:34 [ERROR ] [configure] Evaluation Error: The title 'theforeman/foreman_remote_execution' has already been used in this resource expression (file: /usr/share/foreman-installer/modules/katello_devel/manifests/config.pp, line: 38, column: 3) on node centos8-katello-devel2.galaxy.example.com
    2022-04-26 19:52:34 [NOTICE] [configure] System configuration has finished.
  
      There were errors detected during install.
      Please address the errors and re-run the installer to ensure the system is properly configured.
      Failing to do so is likely to result in broken functionality.
  
      The full log is at /var/log/foreman-installer/katello-devel.log
  stdout_lines: <omitted>

PLAY RECAP *********************************************************************
centos8-katello-devel2     : ok=36   changed=20   unreachable=0    failed=1    skipped=21   rescued=0    ignored=0   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Removing the remote execution plugin installer argument. maybe this is broken…

TASK [foreman_installer : Run installer] ***************************************
fatal: [centos8-katello-devel2]: FAILED! => changed=true 
  cmd: |-
    foreman-installer -v --no-colors --disable-system-checks --scenario katello-devel   --katello-devel-admin-password changeme --katello-devel-github-username=poelzi
  delta: '0:00:23.497818'
  end: '2022-04-26 20:44:47.880279'
  msg: non-zero return code
  rc: 1
  start: '2022-04-26 20:44:24.382461'
  stderr: ''
  stderr_lines: <omitted>
  stdout: |-
    2022-04-26 20:44:29 [NOTICE] [root] Loading installer configuration. This will take some time.
    2022-04-26 20:44:34 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
    2022-04-26 20:44:34 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
    2022-04-26 20:44:36 [WARN  ] [pre] Skipping system checks.
    2022-04-26 20:44:36 [WARN  ] [pre] Skipping system checks.
    2022-04-26 20:44:41 [NOTICE] [configure] Starting system configuration.
    2022-04-26 20:44:47 [ERROR ] [configure] Evaluation Error: Error while evaluating a Function Call, Duplicate declaration: Package[cyrus-sasl-plain] is already declared at (file: /usr/share/foreman-installer/modules/katello_devel/manifests/install.pp, line: 5); cannot redeclare (file: /usr/share/foreman-installer/modules/qpid/manifests/install.pp, line: 21) (file: /usr/share/foreman-installer/modules/qpid/manifests/install.pp, line: 21, column: 5) on node centos8-katello-devel2.galaxy.example.com
    2022-04-26 20:44:47 [NOTICE] [configure] System configuration has finished.
  
      There were errors detected during install.
      Please address the errors and re-run the installer to ensure the system is properly configured.
      Failing to do so is likely to result in broken functionality.
  
      The full log is at /var/log/foreman-installer/katello-devel.log
  stdout_lines: <omitted>

Ahh, yes, I saw this error before. removing…

A new day and I dance in cycles to maybe satisfy the goods of foreman for a working dev environment.

Saldy, I could not please the gods:

TASK [foreman_installer : Run installer] ***************************************
fatal: [centos8-katello-devel2]: FAILED! => changed=true 
  cmd: |-
    foreman-installer -v --no-colors --disable-system-checks --scenario katello-devel   --katello-devel-admin-password changeme --katello-devel-github-username=poelzi
  delta: '0:01:10.298266'
  end: '2022-04-27 11:07:11.754088'
  msg: non-zero return code
  rc: 4
  start: '2022-04-27 11:06:01.455822'
  stderr: ''
  stderr_lines: <omitted>
  stdout: |-
    2022-04-27 11:06:06 [NOTICE] [root] Loading installer configuration. This will take some time.
    2022-04-27 11:06:13 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
    2022-04-27 11:06:13 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
    2022-04-27 11:06:15 [WARN  ] [pre] Skipping system checks.
    2022-04-27 11:06:15 [WARN  ] [pre] Skipping system checks.
    2022-04-27 11:06:20 [NOTICE] [configure] Starting system configuration.
    2022-04-27 11:06:35 [NOTICE] [configure] 250 configuration steps out of 1410 steps complete.
    2022-04-27 11:06:40 [NOTICE] [configure] 500 configuration steps out of 1415 steps complete.
    2022-04-27 11:06:43 [NOTICE] [configure] 750 configuration steps out of 1419 steps complete.
    2022-04-27 11:06:44 [NOTICE] [configure] 1000 configuration steps out of 1423 steps complete.
    2022-04-27 11:06:44 [ERROR ] [configure] Systemd start for httpd failed!
    2022-04-27 11:06:44 [ERROR ] [configure] journalctl log for httpd:
    2022-04-27 11:06:44 [ERROR ] [configure] -- Logs begin at Tue 2022-04-26 21:52:27 UTC, end at Wed 2022-04-27 11:06:44 UTC. --
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: Starting The Apache HTTP Server...
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: AH00112: Warning: DocumentRoot [/usr/share/foreman/public] does not exist
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: AH00526: Syntax error on line 4 of /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf:
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: Invalid command 'SetEnvIf', perhaps misspelled or defined by a module not included in the server configuration
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: httpd.service: Failed with result 'exit-code'.
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: Failed to start The Apache HTTP Server.
    2022-04-27 11:06:44 [ERROR ] [configure] /Stage[main]/Apache::Service/Service[httpd]/ensure: change from 'stopped' to 'running' failed: Systemd start for httpd failed!
    2022-04-27 11:06:44 [ERROR ] [configure] journalctl log for httpd:
    2022-04-27 11:06:44 [ERROR ] [configure] -- Logs begin at Tue 2022-04-26 21:52:27 UTC, end at Wed 2022-04-27 11:06:44 UTC. --
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: Starting The Apache HTTP Server...
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: AH00112: Warning: DocumentRoot [/usr/share/foreman/public] does not exist
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: AH00526: Syntax error on line 4 of /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf:
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com httpd[32696]: Invalid command 'SetEnvIf', perhaps misspelled or defined by a module not included in the server configuration
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: httpd.service: Main process exited, code=exited, status=1/FAILURE
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: httpd.service: Failed with result 'exit-code'.
    2022-04-27 11:06:44 [ERROR ] [configure] Apr 27 11:06:44 centos8-katello-devel2.galaxy.example.com systemd[1]: Failed to start The Apache HTTP Server.
    2022-04-27 11:06:59 [ERROR ] [configure] 'bundle install --retry 3 --jobs 3 --path .vendor' returned 5 instead of one of [0]
    2022-04-27 11:06:59 [ERROR ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install --retry 3 --jobs 3 --path .vendor]/returns: change from 'notrun' to ['0'] failed: 'bundle install --retry 3 --jobs 3 --path .vendor' returned 5 instead of one of [0]
    2022-04-27 11:06:59 [NOTICE] [configure] 1250 configuration steps out of 1424 steps complete.
    2022-04-27 11:07:11 [NOTICE] [configure] System configuration has finished.
  
      There were errors detected during install.
      Please address the errors and re-run the installer to ensure the system is properly configured.
      Failing to do so is likely to result in broken functionality.
  
      The full log is at /var/log/foreman-installer/katello-devel.log
  stdout_lines: <omitted>

PLAY RECAP *********************************************************************
centos8-katello-devel2     : ok=36   changed=7    unreachable=0    failed=1    skipped=21   rescued=0    ignored=0   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Looking into the log:

2022-04-27 11:06:59 [INFO  ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install -
-retry 3 --jobs 3 --path .vendor]/returns:     qpid_proton
2022-04-27 11:06:59 [ERROR ] [configure] 'bundle install --retry 3 --jobs 3 --path .vendor' returned 5 instead of one of [0]
2022-04-27 11:06:59 [ERROR ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install -
-retry 3 --jobs 3 --path .vendor]/returns: change from 'notrun' to ['0'] failed: 'bundle install --retry 3 --jobs 3 --path .vendor' returned 5 instead of one of 
[0]
2022-04-27 11:06:59 [DEBUG ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install -
-retry 3 --jobs 3 --path .vendor]: Evaluated in 12.08 seconds
2022-04-27 11:06:59 [DEBUG ] [configure] Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]: Starting to evaluate the resource (1195 of 1424)

Nice, right, I remember now, qpid_proton proto is broken on ruby 2.4.

In conclusion, the VM you suggest in the README does not even work after long setup and wrokaround time. Lets then try the centos7 box…

BOOOOMM

   2022-04-27 11:30:41 [ERROR ] [configure] Evaluation Error: Error while evaluating a Function Call, Duplicate declaration: Package[cyrus-sasl-plain] is already declared at (file: /usr/share/foreman-installer/modules/katello_devel/manifests/install.pp, line: 5); cannot redeclare (file: /usr/share/foreman-installer/modules/qpid/manifests/install.pp, line: 21) (file: /usr/share/foreman-installer/modules/qpid/manifests/install.pp, line: 21, column: 5) on node centos7-katello-devel.galaxy.local
    2022-04-27 11:30:41 [NOTICE] [configure] System configuration has finished.
  
      There were errors detected during install.
      Please address the errors and re-run the installer to ensure the system is properly configured.
      Failing to do so is likely to result in broken functionality.
  
      The full log is at /var/log/foreman-installer/katello-devel.log

I know that one, so, not only is master for alt least 2-3 days broken on your suggested environment, also on older ones. Ok, did not work with

TASK [foreman_installer : Run installer] ***************************************
fatal: [centos7-katello-devel]: FAILED! => changed=true 
  cmd: |-
    foreman-installer -v --no-colors --disable-system-checks --scenario katello-devel --foreman-proxy-content-enable-ostree=true --katello-devel-scl-ruby=rh-ruby27 --katello-devel-admin-password changeme --katello-devel-github-username=poelzi
  delta: '0:02:12.538965'
  end: '2022-04-27 11:42:58.732164'
  msg: non-zero return code
  rc: 6
  start: '2022-04-27 11:40:46.193199'
  stderr: ''
  stderr_lines: <omitted>
  stdout: |-
    2022-04-27 11:40:52 [NOTICE] [root] Loading installer configuration. This will take some time.
    2022-04-27 11:40:58 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
    2022-04-27 11:40:58 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
    2022-04-27 11:41:00 [WARN  ] [pre] Skipping system checks.
    2022-04-27 11:41:00 [WARN  ] [pre] Skipping system checks.
    2022-04-27 11:41:05 [NOTICE] [configure] Starting system configuration.
    2022-04-27 11:41:18 [NOTICE] [configure] 250 configuration steps out of 1411 steps complete.
    2022-04-27 11:41:33 [NOTICE] [configure] 500 configuration steps out of 1416 steps complete.
    2022-04-27 11:41:44 [ERROR ] [configure] Execution of '/bin/yum -d 0 -e 0 -y install pulpcore-plugin(ostree)' returned 1: Error: Nothing to do
    2022-04-27 11:41:44 [ERROR ] [configure] /Stage[main]/Pulpcore::Plugin::Ostree/Pulpcore::Plugin[ostree]/Package[pulpcore-plugin(ostree)]/ensure: change from 'purged' to 'present' failed: Execution of '/bin/yum -d 0 -e 0 -y install pulpcore-plugin(ostree)' returned 1: Error: Nothing to do
    2022-04-27 11:41:44 [NOTICE] [configure] 750 configuration steps out of 1420 steps complete.
    2022-04-27 11:41:45 [NOTICE] [configure] 1000 configuration steps out of 1464 steps complete.
    2022-04-27 11:41:47 [ERROR ] [configure] /Service[httpd]: Failed to call refresh: Systemd restart for httpd failed!
    2022-04-27 11:41:47 [ERROR ] [configure] journalctl log for httpd:
    2022-04-27 11:41:47 [ERROR ] [configure] -- Logs begin at Wed 2022-04-27 11:19:45 UTC, end at Wed 2022-04-27 11:41:47 UTC. --
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:46 centos7-katello-devel.galaxy.local systemd[1]: Stopping The Apache HTTP Server...
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Stopped The Apache HTTP Server.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Starting The Apache HTTP Server...
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: AH00112: Warning: DocumentRoot [/usr/share/foreman/public] does not exist
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: AH00526: Syntax error on line 4 of /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf:
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: Invalid command 'SetEnvIf', perhaps misspelled or defined by a module not included in the server configuration
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Failed to start The Apache HTTP Server.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Unit httpd.service entered failed state.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: httpd.service failed.
    2022-04-27 11:41:47 [ERROR ] [configure] /Service[httpd]: Systemd restart for httpd failed!
    2022-04-27 11:41:47 [ERROR ] [configure] journalctl log for httpd:
    2022-04-27 11:41:47 [ERROR ] [configure] -- Logs begin at Wed 2022-04-27 11:19:45 UTC, end at Wed 2022-04-27 11:41:47 UTC. --
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:46 centos7-katello-devel.galaxy.local systemd[1]: Stopping The Apache HTTP Server...
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Stopped The Apache HTTP Server.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Starting The Apache HTTP Server...
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: AH00112: Warning: DocumentRoot [/usr/share/foreman/public] does not exist
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: AH00526: Syntax error on line 4 of /etc/httpd/conf.d/05-foreman-ssl.d/katello.conf:
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local httpd[20177]: Invalid command 'SetEnvIf', perhaps misspelled or defined by a module not included in the server configuration
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Failed to start The Apache HTTP Server.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: Unit httpd.service entered failed state.
    2022-04-27 11:41:47 [ERROR ] [configure] Apr 27 11:41:47 centos7-katello-devel.galaxy.local systemd[1]: httpd.service failed.
    2022-04-27 11:42:41 [ERROR ] [configure] 'scl enable rh-ruby27 rh-nodejs12 rh-postgresql12 'bundle install --retry 3 --jobs 3 --path .vendor'' returned 5 instead of one of [0]
    2022-04-27 11:42:41 [ERROR ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install --retry 3 --jobs 3 --path .vendor]/returns: change from 'notrun' to ['0'] failed: 'scl enable rh-ruby27 rh-nodejs12 rh-postgresql12 'bundle install --retry 3 --jobs 3 --path .vendor'' returned 5 instead of one of [0]
    2022-04-27 11:42:54 [ERROR ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install --retry 3 --jobs 3 --path .vendor]: Failed to call refresh: 'scl enable rh-ruby27 rh-nodejs12 rh-postgresql12 'bundle install --retry 3 --jobs 3 --path .vendor'' returned 5 instead of one of [0]
    2022-04-27 11:42:54 [ERROR ] [configure] /Stage[main]/Katello_devel::Setup/Katello_devel::Bundle[install --retry 3 --jobs 3 --path .vendor]/Exec[bundle-install --retry 3 --jobs 3 --path .vendor]: 'scl enable rh-ruby27 rh-nodejs12 rh-postgresql12 'bundle install --retry 3 --jobs 3 --path .vendor'' returned 5 instead of one of [0]
    2022-04-27 11:42:54 [NOTICE] [configure] 1250 configuration steps out of 1465 steps complete.
    2022-04-27 11:42:58 [NOTICE] [configure] System configuration has finished.
  
      There were errors detected during install.
      Please address the errors and re-run the installer to ensure the system is properly configured.
      Failing to do so is likely to result in broken functionality.
  
      The full log is at /var/log/foreman-installer/katello-devel.log
  stdout_lines: <omitted>

The machine is some month old, lets try it from scratch.
Lol on the apache error. Maybe this will be gone on the next run. After some failed runs, lets try this
vagrant box update vagrant does not update stuff and there is no version pinning anywhere, so.

At least the duplicated package error is still there, so it was not that this was the ultimate culpit.
Lets try from scratch, removing old VM, .vagrant folder, etc. Ahh, a new error:

TASK [foreman_installer : Run installer] ***************************************
fatal: [my-centos7-katello-devel]: FAILED! => changed=true 
  cmd: |-
    foreman-installer -v --no-colors --disable-system-checks --scenario katello-devel --foreman-proxy-content-enable-ostree=true --katello-devel-scl-ruby=rh-ruby27 --katello-devel-admin-password changeme
  delta: '0:00:14.304883'
  end: '2022-04-28 23:13:00.142669'
  msg: non-zero return code
  rc: 20
  start: '2022-04-28 23:12:45.837786'
  stderr: Forward DNS points to 127.0.1.1 which is not configured on this server
  stderr_lines: <omitted>
  stdout: |-
    2022-04-28 23:12:52 [NOTICE] [root] Loading installer configuration. This will take some time.
    2022-04-28 23:12:58 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
    2022-04-28 23:12:58 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
    Your system does not meet configuration criteria
  stdout_lines: <omitted>

After some days:
For $reasons I had to mkdir /etc/mosquitto/ because the puppet module assumes it exists, which doesn’t on a new install.

Apache still fails with

2022-05-02 16:24:18 [ERROR ] [configure] May 02 16:24:18 my-centos7-katello-devel.galaxy.local httpd[9104]: Invalid command 'SetEnvIf', perhaps misspelled or defined by a module not included in the server configuration

Despite by afc08e30 William 2022-4-4 which is over 1 month old.
WTF

[root@centos7-katello-devel foreman-installer]# yum info foreman-installer
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.fra10.de.leaseweb.net
 * centos-sclo-rh: de.mirrors.clouvider.net
 * epel: mirror.dogado.de
 * extras: mirror.fra10.de.leaseweb.net
 * updates: mirror.imt-systems.com
Installed Packages
Name        : foreman-installer
Arch        : noarch
Epoch       : 1
Version     : 3.3.0
Release     : 0.2.develop.20220503134331git566b293.el7
Size        : 6.9 M
Repo        : installed
From repo   : foreman-koji
Summary     : Puppet-based installer for The Foreman
URL         : https://theforeman.org
License     : GPLv3+ and ASL 2.0
Description : Complete installer for The Foreman life-cycle management system based on Puppet.

I have no Idea how you package your sources together, but it is definitly not tested and working.
So, editing the katello_devel puppet module seems to have helped, pr opened.
Now the next problem:

2022-05-04 12:35:54 [ERROR ] [configure] Proxy my-centos7-katello-devel.galaxy.local has failed to load one or more features (Scrip
t), check /var/log/foreman-proxy/proxy.log for configuration errors

 2022-05-04 12:35:54 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[my-centos7-katello-devel.galaxy.lo
cal]/ensure: change from 'absent' to 'present' failed: Proxy my-centos7-katello-devel.galaxy.local has failed to load one or more featu
res (Script), check /var/log/foreman-proxy/proxy.log for configuration errors                                                           
   2022-05-04 12:35:57 [NOTICE] [configure] System configuration has finished.

No errors in proxylog.

I’m a developer since 94 or so, I have seen countless dev environments and languages.

forklift and foreman is, sorry to say this so flarkly, the worst dev environment I have ever seen so far. Maybe I should look into COBOL and mainframes to get my brain straight, but this dev experience is pure garbage. I can’t imagine somebody voluntary going through this and not giving up after a day.

1 Like
2

Although I agree that it’s not quite obvious why forklift fails during a dev env setup, have you tried stable [1] boxes?

[1] - forklift/stable_boxes.md at master · theforeman/forklift · GitHub

3

I have a job that requires a development env.

2022-05-04 13:02:53 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Collected {"fe
atures"=>["Logs", "Pulpcore", "Dynflow", "Script"]}
2022-05-04 13:02:53 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Selecting sour
ce_key features
2022-05-04 13:02:53 [DEBUG ] [configure] Datacat_collector[foreman_proxy::enabled_features](provider=datacat_collector): Now setting fi
eld :features
2022-05-04 13:02:53 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Datacat_collector[foreman_proxy::enabled_features]: Evalu
ated in 0.02 seconds
2022-05-04 13:02:53 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[my-centos7-katello-devel.galaxy.local]
: Starting to evaluate the resource (1428 of 1443)
2022-05-04 13:02:53 [DEBUG ] [configure] Foreman_smartproxy[my-centos7-katello-devel.galaxy.local](provider=rest_v3): Making get reques
t to https://my-centos7-katello-devel.galaxy.local/api/v2/smart_proxies?search=name%3D%22my-centos7-katello-devel.galaxy.local%22
2022-05-04 13:02:53 [DEBUG ] [configure] Foreman_smartproxy[my-centos7-katello-devel.galaxy.local](provider=rest_v3): Received response
 200 from request to https://my-centos7-katello-devel.galaxy.local/api/v2/smart_proxies?search=name%3D%22my-centos7-katello-devel.galax
y.local%22
2022-05-04 13:02:53 [DEBUG ] [configure] Foreman_smartproxy[my-centos7-katello-devel.galaxy.local](provider=rest_v3): Making put reques
t to https://my-centos7-katello-devel.galaxy.local/api/v2/smart_proxies/1/refresh
2022-05-04 13:02:54 [DEBUG ] [configure] Foreman_smartproxy[my-centos7-katello-devel.galaxy.local](provider=rest_v3): Received response
 200 from request to https://my-centos7-katello-devel.galaxy.local/api/v2/smart_proxies/1/refresh
2022-05-04 13:02:54 [ERROR ] [configure] Proxy my-centos7-katello-devel.galaxy.local has failed to load one or more features (Script), 
check /var/log/foreman-proxy/proxy.log for configuration errors
2022-05-04 13:02:54 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[my-centos7-katello-devel.galaxy.local]
/features: change from ["Dynflow", "Logs", "Pulpcore"] to ["Dynflow", "Logs", "Pulpcore", "Script"] failed: Proxy my-centos7-katello-de
vel.galaxy.local has failed to load one or more features (Script), check /var/log/foreman-proxy/proxy.log for configuration errors
2022-05-04 13:02:54 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[my-centos7-katello-devel.galaxy.local]
: Evaluated in 1.25 seconds
2022-05-04 13:02:54 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-my-centos7-katello-
devel.galaxy.local]: Starting to evaluate the resource (1429 of 1443)
2022-05-04 13:02:54 [INFO  ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-my-centos7-katello-
devel.galaxy.local]: Dependency Foreman_smartproxy[my-centos7-katello-devel.galaxy.local] has failures: true
2022-05-04 13:02:54 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-my-centos7-katello-
devel.galaxy.local]: Skipping because of failed dependencies
2022-05-04 13:02:54 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-my-centos7-katello-
devel.galaxy.local]: Resource is being skipped, unscheduling all events
2022-05-04 13:02:54 [DEBUG ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy_host[foreman-proxy-my-centos7-katello-
devel.galaxy.local]: Evaluated in 0.00 seconds

Says nothing about the problem

4

To answer the headline question: The forklift cutting edge development environment is often broken, because it is the cutting edge development environment of a huge and complex software undergoing active development.

How do full time Katello developers deal with this? Presumably by spending a lot of time working with, and by extension gathering experience, on how to build and maintain their development environments.

If you are new to Katello development, or only very occasionally need a development environment, I think you broadly have two options:

  1. Invest the time into getting the Forklift dev environment working for you. It probably helps to get some interactive help on foreman-dev IRC/matrix rather than always waiting for Forum answers for every little thing while getting started. (People are generally quite willing to help, though it is of course dependent on if you catch them at a good moment).
  2. Maybe one of the stable boxes is sufficient for what you are trying to do? It is possible to “patch” a stable box, thus using it for development.

If you are going to do a lot of development/big features it is probably better to do go with option 1. But if you only need to test one small to medium patch once, it will probably be less overall work to go with option 2. I hope this helps.

5

So I gave the ‘stable’ box a try.
So, my first experience is, that running a simple command in ~vagrant/foreman like
rake help fails. Ok, git pull, bundle install, for $reasons I had to do a bundle update as he complained about rails version.

@quba42 yes, I understand that this is complex software, the more important it is to properly intergration test and fixate all the dependencies. I suggest looking into nix/nixos how such things are handled there. Dependencies need to be properly locked and updated from one working set to another.

6

A note about speeding up Vagrant. The primary reason that the calls are slow is due to the scale of boxes present for Vagrant. We use Forklift for not just development environments, but getting release versions setup for testing. Due to this we introduced the ability to customize a local checkout of Forklift and ignore the things that are not important to you. If you make use of [1], you will significantly improve the speed of Vagrant.

As noted by others, there are times when spinning up a development environment that you may encounter a broken state. Development environment is a point in time and may reflect a state of our nightly repositories that are in transition across Foreman, Katello and plugins or that is broken in some way. While we do try to catch these issues, they do slip through from time to time.

Please reach out if you encounter something so we can investigate and try to resolve quickly for you. I, for one, would appreciate any feedback about how we can make developer setups easier. Forklift has been an evolution through time of developer needs and will continue to be so. Please let us know how we can help.

[1] GitHub - theforeman/forklift: Helpful deployment scripts for Foreman and Katello

7

I don’t think so. It is the way it is implemented that cause each VM, no matter if it is used or not, to get instanciated and subsequent calls to libvirt causes a delay for every box. At least that is what my print debugging shows. Unfortunatelly, the profiler I found for ruby don’t show the culpit.