Foreman nightly pipeline failed:
https://ci.theforeman.org/job/foreman-nightly-rpm-pipeline/713/
The installer failed with:
[DEBUG 2019-03-13T22:10:18 main] Foreman_smartproxy[pipeline-foreman-nightly-centos7.n58.example.com](provider=rest_v3): Making get request to https://pipeline-foreman-nightly-centos7.n58.example.com/api/v2/smart_proxies?search=name=%22pipeline-foreman-nightly-centos7.n58.example.com%22
[DEBUG 2019-03-13T22:10:19 main] Foreman_smartproxy[pipeline-foreman-nightly-centos7.n58.example.com](provider=rest_v3): Received response 500 from request to https://pipeline-foreman-nightly-centos7.n58.example.com/api/v2/smart_proxies?search=name=%22pipeline-foreman-nightly-centos7.n58.example.com%22
[ERROR 2019-03-13T22:10:19 main] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[pipeline-foreman-nightly-centos7.n58.example.com]: Could not evaluate: Proxy pipeline-foreman-nightly-centos7.n58.example.com cannot be retrieved: unknown error (response 500)
[ERROR 2019-03-13T22:10:19 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:7:in `proxy'
[ERROR 2019-03-13T22:10:19 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:13:in `id'
[ERROR 2019-03-13T22:10:19 main] /usr/share/foreman-installer/modules/foreman/lib/puppet/provider/foreman_smartproxy/rest_v3.rb:17:in `exists?'
[ERROR 2019-03-13T22:10:19 main] /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/property/ensure.rb:81:in `retrieve'
but production.log didn’t even register the request so I looked in httpd/error_log:
[ N 2019-03-13 22:07:37.5470 10938/T1 age/Wat/WatchdogMain.cpp:1366 ]: Starting Passenger watchdog...
[ N 2019-03-13 22:07:37.6008 10941/T1 age/Cor/CoreMain.cpp:1339 ]: Starting Passenger core...
[ N 2019-03-13 22:07:37.6011 10941/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode.
[ N 2019-03-13 22:07:37.6167 10941/T1 age/Cor/CoreMain.cpp:1014 ]: Passenger core online, PID 10941
[Wed Mar 13 22:07:37.627953 2019] [mpm_prefork:notice] [pid 10912] AH00163: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips Phusion_Passenger/6.0.2 configured -- resuming normal operations
[Wed Mar 13 22:07:37.628002 2019] [core:notice] [pid 10912] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[ N 2019-03-13 22:07:38.0012 10917/T1 age/Cor/TelemetryCollector.h:531 ]: Message from Phusion: End time can not be before or equal to begin time
[ N 2019-03-13 22:07:38.0347 10917/T1 age/Cor/CoreMain.cpp:1324 ]: Passenger core shutdown finished
[ E 2019-03-13 22:07:39.7792 10941/Tf age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /usr/share/foreman: An operating system error occurred while preparing to start a preloader process: Cannot create FIFO file /tmp/passenger.spawn.XXXXqnEz7T/response/finish: Permission denied (errno=13)
Error ID: 4f900136
Error details saved to: /tmp/passenger-error-LQuIOK.html
[ E 2019-03-13 22:07:39.7893 10941/T9 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is 4f900136. Please see earlier logs for details about the error.
[ N 2019-03-13 22:07:40.2957 10941/T4 age/Cor/SecurityUpdateChecker.h:519 ]: Security update check: no update found (next check in 24 hours)
[ E 2019-03-13 22:07:41.0337 10941/Th age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /usr/share/foreman: An operating system error occurred while preparing to start a preloader process: Cannot create FIFO file /tmp/passenger.spawn.XXXXZZhNcE/response/finish: Permission denied (errno=13)
Error ID: ef45895b
Error details saved to: /tmp/passenger-error-m7gKDx.html
[ E 2019-03-13 22:07:41.0428 10941/Ta age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 2-1] Cannot checkout session because a spawning error occurred. The identifier of the error is ef45895b. Please see earlier logs for details about the error.
[ E 2019-03-13 22:10:19.0534 10941/Tj age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /usr/share/foreman: An operating system error occurred while preparing to start a preloader process: Cannot create FIFO file /tmp/passenger.spawn.XXXX18KrZZ/response/finish: Permission denied (errno=13)
Error ID: 1d19ea17
Error details saved to: /tmp/passenger-error-OScoos.html
[ E 2019-03-13 22:10:19.0634 10941/T9 age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 1-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 1d19ea17. Please see earlier logs for details about the error.
[ E 2019-03-13 22:10:19.2365 10941/Tl age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /usr/share/foreman: An operating system error occurred while preparing to start a preloader process: Cannot create FIFO file /tmp/passenger.spawn.XXXXNJzc0U/response/finish: Permission denied (errno=13)
Error ID: 874e6838
Error details saved to: /tmp/passenger-error-XuyWNn.html
[ E 2019-03-13 22:10:19.2473 10941/Ta age/Cor/Con/CheckoutSession.cpp:276 ]: [Client 2-2] Cannot checkout session because a spawning error occurred. The identifier of the error is 874e6838. Please see earlier logs for details about the error.
Any idea what this could be? since the /tmp/passenger-error- files aren’t saved in the sos report
I’m pretty sure this has to do with the Kafo 3 migration. We changed the hiera format to hiera 5. Pretty sure this included a build without the matching foreman-installer changes which causes it to pull in the passenger upstream repos. That breaks (SELinux?).
Good hunch! snippet from audit.log:
type=AVC msg=audit(1552514857.644:2273): avc: denied { write } for pid=10917 comm="PassengerAgent" name="cert9.db" dev="vda1" ino=100716012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cert_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1552514857.644:2273): arch=c000003e syscall=2 success=no exit=-13 a0=b2dde0 a1=80042 a2=1a4 a3=b2dde0 items=0 ppid=1 pid=10917 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514857.644:2273): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514857.677:2274): avc: denied { write } for pid=10917 comm="PassengerAgent" name="cert9.db" dev="vda1" ino=100716012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cert_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1552514857.677:2274): arch=c000003e syscall=2 success=no exit=-13 a0=b2dde0 a1=80042 a2=1a4 a3=b2dde0 items=0 ppid=1 pid=10917 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514857.677:2274): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514859.757:2275): avc: denied { create } for pid=10941 comm="PassengerAgent" name="finish" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_tmp_t:s0 tclass=fifo_file permissive=0
type=SYSCALL msg=audit(1552514859.757:2275): arch=c000003e syscall=133 success=no exit=-13 a0=7f562c001c68 a1=1180 a2=0 a3=7f565dc6a420 items=0 ppid=10938 pid=10941 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.757:2275): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514859.772:2276): avc: denied { fowner } for pid=11046 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1552514859.772:2276): arch=c000003e syscall=268 success=no exit=-1 a0=4 a1=1f0e598 a2=1c0 a3=2b items=0 ppid=10941 pid=11046 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chmod" exe="/usr/bin/chmod" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.772:2276): proctitle=63686D6F64002D5200752B727778002F746D702F70617373656E6765722E737061776E2E58585858716E457A3754
type=AVC msg=audit(1552514859.772:2277): avc: denied { fowner } for pid=11046 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1552514859.772:2277): arch=c000003e syscall=268 success=no exit=-1 a0=5 a1=1f16848 a2=1c0 a3=2b items=0 ppid=10941 pid=11046 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chmod" exe="/usr/bin/chmod" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.772:2277): proctitle=63686D6F64002D5200752B727778002F746D702F70617373656E6765722E737061776E2E58585858716E457A3754
type=AVC msg=audit(1552514859.772:2278): avc: denied { fowner } for pid=11046 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1552514859.772:2278): arch=c000003e syscall=268 success=no exit=-1 a0=4 a1=1f16728 a2=1c0 a3=2b items=0 ppid=10941 pid=11046 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chmod" exe="/usr/bin/chmod" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.772:2278): proctitle=63686D6F64002D5200752B727778002F746D702F70617373656E6765722E737061776E2E58585858716E457A3754
type=AVC msg=audit(1552514859.885:2279): avc: denied { write } for pid=10941 comm="PassengerAgent" name="cert9.db" dev="vda1" ino=100716012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cert_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1552514859.885:2279): arch=c000003e syscall=2 success=no exit=-13 a0=7f5650065a20 a1=80042 a2=1a4 a3=7f5650065a20 items=0 ppid=10938 pid=10941 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.885:2279): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514859.918:2280): avc: denied { write } for pid=10941 comm="PassengerAgent" name="cert9.db" dev="vda1" ino=100716012 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cert_t:s0 tclass=file permissive=0
type=SYSCALL msg=audit(1552514859.918:2280): arch=c000003e syscall=2 success=no exit=-13 a0=7f56500659c0 a1=80042 a2=1a4 a3=7f56500659c0 items=0 ppid=10938 pid=10941 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514859.918:2280): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514861.011:2281): avc: denied { create } for pid=10941 comm="PassengerAgent" name="finish" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_tmp_t:s0 tclass=fifo_file permissive=0
type=SYSCALL msg=audit(1552514861.011:2281): arch=c000003e syscall=133 success=no exit=-13 a0=7f562c001488 a1=1180 a2=0 a3=5858582e6e776170 items=0 ppid=10938 pid=10941 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="PassengerAgent" exe="/usr/lib64/passenger/support-binaries/PassengerAgent" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514861.011:2281): proctitle=50617373656E67657220636F7265
type=AVC msg=audit(1552514861.026:2282): avc: denied { fowner } for pid=11060 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1552514861.026:2282): arch=c000003e syscall=268 success=no exit=-1 a0=4 a1=1fe5598 a2=1c0 a3=2b items=0 ppid=10941 pid=11060 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chmod" exe="/usr/bin/chmod" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514861.026:2282): proctitle=63686D6F64002D5200752B727778002F746D702F70617373656E6765722E737061776E2E585858585A5A684E6345
type=AVC msg=audit(1552514861.026:2283): avc: denied { fowner } for pid=11060 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0
type=SYSCALL msg=audit(1552514861.026:2283): arch=c000003e syscall=268 success=no exit=-1 a0=5 a1=1fed848 a2=1c0 a3=2b items=0 ppid=10941 pid=11060 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="chmod" exe="/usr/bin/chmod" subj=system_u:system_r:httpd_t:s0 key=(null)
type=PROCTITLE msg=audit(1552514861.026:2283): proctitle=63686D6F64002D5200752B727778002F746D702F70617373656E6765722E737061776E2E585858585A5A684E6345
type=AVC msg=audit(1552514861.026:2284): avc: denied { fowner } for pid=11060 comm="chmod" capability=3 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=capability permissive=0The hunch was based on what I saw while doing the migration locally. I was sure I included this in the PR, but clearly didn’t:
This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.