What is FIPS?
From Wikipedia [1]: The Federal Information Processing Standard (FIPS)
Publication 140-2, (FIPS PUB 140-2), is a U.S. government computer
security standard used to approve cryptographic modules. The title is
Security Requirements for Cryptographic Modules.
What are Implications of FIPS 140-2 Support for Foreman, Katello, and
Smart-Proxy?
Linux system, or rather an SSL library in FIPS-compatible mode will
only have a set of ciphers and hash functions compatible with FIPS.
[2] contains the list of approved cryptographic functions, Oracle
graciously compiled the list of not approved ones, which is more
useful and can be found at [3].
OpenSSL in FIPS mode
My understanding is that only OpenSSL versions 1.0.1 and 1.0.2 have
FIPS 140-2 validated cryptographic modules. OpenSSL raises ABRT signal
when it receives a call to one of the unapproved ciphers/functions.
Foreman in FIPS mode
I haven’t looked at pulp, candlepin, qpid, goferd, etc, and at point
don’t know how and if these can be made to work in FIPS mode. All
tests I’ve done so far were against Rails 5.0, Considering the number
of dependencies, we will need to limit FIPS support to just one
version of Rails.
Rails and other (ruby) dependencies.
MD5 is used (hard-coded) in a few places in Rails, at this point I’m
quite certain that its use is constrained to various built-in caches.
I had to disable *all* Rails caches to be able to run Foreman in FIPS
mode. Additionally, strong ETAG’s cannot be used, I’m not sure if they
are used, or there are plans for them.
Spring uses MD5 to generate application ID, but will use one in
SPRING_APPLICATION_ID environment variable if it’s available.
Gravatar uses MD5 hashes in their urls, doesn’t look like other hashes
are supported.
I think apipie cache uses MD5, but I will need to verify this.
Foreman
app/services/password_crypt uses MD5 for grub2 passwords, which will
need to be switched to SHA512. MD5 will need to be removed from the
list of hash functions
SshKey#generate_fingerprint, call to SSHKey.fingerprint uses MD5
A note: with caching disabled, and issues above fixed, I was able to
get Foreman suite of tests to pass, and get Foreman to start.
Smart-Proxy
Smart-Proxy codebase appears to be compatible with FIPS (ran and
passed tests ok without any changes), but there are issues with
external depdencies.
DHCPD uses MD5-based omapi shared secret. DHCPD shared secret with
bind is also md5-based.
BIND when used with dhcpd uses MD5 hashes stored in TXT as host id.
Puppet needs to be run in FIPS mode (FIPS-compatible hash function
needs to be configured). I assume this covers all of puppet, including
mcollective, puppet run, puppetca.
BMC/IPMI authentication can use MD5 or lower based hashes, older
clients may not have newer hash functions.
Salt appears to use MD5 hashes by default, individual nodes must be
configured to use other hash_type
Any 3rd party SSL certificates that may need to be verified or decoded
by either Foreman or Smart-Proxy must be generated using
FIPS-compatible algorithms/hash functions.
How we can reach FIPS compatibility
The easiest first step would be to replace offending cryptographic and
hash functions in Foreman, and in Smart-Proxy case, 3rd party
configuration files with FIPS-compatible ones. Additionally, any new
code changes that employ MD5 or other non approved functions shouldn’t
be accepted.
The next step would be to create a CI job that will continuously
execute the the full suite of tests on a VM with FIPS mode enabled.
GDB configured with Ruby’s project .gdbinit [4] and a tiny batch [5]
of commands can be used to report on FIPS-related failures.
Considering the amount of dependencies Foreman and Smart-Proxy have, I
think would be useful to have all CI environments switched to run in
FIPS mode: this should increase the probability of discovering of new
FIPS-related issues before our users.
Lastly, a FIPS-compatible caching solution for Rails needs to be
found, if none exist, an existing one needs to be modified to support
FIPS.
Any feedback would be appreciated,
-d
[1] Wikipedia article on FIPS 140-2, https://en.wikipedia.org/wiki/
FIPS_140-2
[2] Approved Security Functions for FIPS 140-2,
https://csrc.nist.gov/csrc/media/publications/fips/140/2/final/documents/
fips1402annexa.pdf
[3] List of algorithms not approved for FIPS 140-2,
https://docs.oracle.com/cd/E36784_01/html/E54953/fips-notok-1.html
[4] Ruby project’s gdb helper functions,
[5] Catching SIGABRTs with gdb and ruby-specific .gdbinit,
Awesome investigation, explanation and ideas on how we can stay compliant
once we reach that milestone. I have a few specific questions in-line:
Sounds like we still need to look at the installer to ensure Kafo, Puppet and the Puppet modules are all FIPS compliant?
--
Eric D. Helms
Red Hat Engineering
What is FIPS?Given we are targeting Rails 5.1 for the SCL we are building and it's the newest, does anything change here with using it? I have heard that the teams that work on those backend projects we use are/have looked into FIPS compliancy so we don't need to worry about that aspect.
From Wikipedia [1]: The Federal Information Processing Standard (FIPS)
Publication 140-2, (FIPS PUB 140-2), is a U.S. government computer
security standard used to approve cryptographic modules. The title is
Security Requirements for Cryptographic Modules.
What are Implications of FIPS 140-2 Support for Foreman, Katello, and
Smart-Proxy?
Linux system, or rather an SSL library in FIPS-compatible mode will
only have a set of ciphers and hash functions compatible with FIPS.
[2] contains the list of approved cryptographic functions, Oracle
graciously compiled the list of not approved ones, which is more
useful and can be found at [3].
OpenSSL in FIPS mode
My understanding is that only OpenSSL versions 1.0.1 and 1.0.2 have
FIPS 140-2 validated cryptographic modules. OpenSSL raises ABRT signal
when it receives a call to one of the unapproved ciphers/functions.
Foreman in FIPS mode
I haven’t looked at pulp, candlepin, qpid, goferd, etc, and at point
don’t know how and if these can be made to work in FIPS mode. All
tests I’ve done so far were against Rails 5.0, Considering the number
of dependencies, we will need to limit FIPS support to just one
version of Rails.
Sounds like we still need to look at the installer to ensure Kafo, Puppet and the Puppet modules are all FIPS compliant?
Rails and other (ruby) dependencies.
MD5 is used (hard-coded) in a few places in Rails, at this point I’m
quite certain that its use is constrained to various built-in caches.
I had to disable *all* Rails caches to be able to run Foreman in FIPS
mode. Additionally, strong ETAG’s cannot be used, I’m not sure if they
are used, or there are plans for them.
Spring uses MD5 to generate application ID, but will use one in
SPRING_APPLICATION_ID environment variable if it’s available.
Gravatar uses MD5 hashes in their urls, doesn’t look like other hashes
are supported.
I think apipie cache uses MD5, but I will need to verify this.
ForemanI suppose we could tests for this? For example, if a user provides custom SSL certificates for the web server we could run a check and warn a user? Is it possible to detect that a system is running in FIPS mode to only warn in that case?
app/services/password_crypt uses MD5 for grub2 passwords, which will
need to be switched to SHA512. MD5 will need to be removed from the
list of hash functions
SshKey#generate_fingerprint, call to SSHKey.fingerprint uses MD5
A note: with caching disabled, and issues above fixed, I was able to
get Foreman suite of tests to pass, and get Foreman to start.
Smart-Proxy
Smart-Proxy codebase appears to be compatible with FIPS (ran and
passed tests ok without any changes), but there are issues with
external depdencies.
DHCPD uses MD5-based omapi shared secret. DHCPD shared secret with
bind is also md5-based.
BIND when used with dhcpd uses MD5 hashes stored in TXT as host id.
Puppet needs to be run in FIPS mode (FIPS-compatible hash function
needs to be configured). I assume this covers all of puppet, including
mcollective, puppet run, puppetca.
BMC/IPMI authentication can use MD5 or lower based hashes, older
clients may not have newer hash functions.
Salt appears to use MD5 hashes by default, individual nodes must be
configured to use other hash_type
Any 3rd party SSL certificates that may need to be verified or decoded
by either Foreman or Smart-Proxy must be generated using
FIPS-compatible algorithms/hash functions.
How often do you see this CI job running? And I assume this would be temporary until we are fully FIPS compliant?
How we can reach FIPS compatibility
The easiest first step would be to replace offending cryptographic and
hash functions in Foreman, and in Smart-Proxy case, 3rd party
configuration files with FIPS-compatible ones. Additionally, any new
code changes that employ MD5 or other non approved functions shouldn’t
be accepted.
The next step would be to create a CI job that will continuously
execute the the full suite of tests on a VM with FIPS mode enabled.
GDB configured with Ruby’s project .gdbinit [4] and a tiny batch [5]
of commands can be used to report on FIPS-related failures.
Considering the amount of dependencies Foreman and Smart-Proxy have, IBig +1 -- if we reach FIPS compliancy, then I am all for turning it on on all the Jenkins slaves as well as ensuring that our Vagrant boxes (for users and CI) have it turned out so we are running in that mode at all times.
think would be useful to have all CI environments switched to run in
FIPS mode: this should increase the probability of discovering of new
FIPS-related issues before our users.
Lastly, a FIPS-compatible caching solution for Rails needs to be
found, if none exist, an existing one needs to be modified to support
FIPS.
···
On Thu, Nov 16, 2017 at 1:35 PM, Dmitri Dolguikh <witlessbird@gmail.com> wrote:
Any feedback would be appreciated,
-d
[1] Wikipedia article on FIPS 140-2, https://en.wikipedia.org/wiki/
FIPS_140-2
[2] Approved Security Functions for FIPS 140-2,
https://csrc.nist.gov/csrc/media/publications/fips/140/2/fin
al/documents/fips1402annexa.pdf
[3] List of algorithms not approved for FIPS 140-2,
https://docs.oracle.com/cd/E36784_01/html/E54953/fips-notok-1.html
[4] Ruby project’s gdb helper functions,
[5] Catching SIGABRTs with gdb and ruby-specific .gdbinit,
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Eric D. Helms
Red Hat Engineering
Given we are targeting Rails 5.1 for the SCL we are building and it's theThe approach stays the same, I think. My bet would be the problems stay the same too (i.e. use of MD5 in caches).
newest, does anything change here with using it?
I have heard that the teams that work on those backend projects we useYes, although I'd imagine puppet itself is going to be ok, as long as FIPS-compatible hash function is configured. Puppet modules have issues, for example dhcpd and bind puppet modules might use MD5s, as those dependencies are configured with MD5-based shared secrets.
are/have looked into FIPS compliancy so we don't need to worry about that
aspect.
Sounds like we still need to look at the installer to ensure Kafo, Puppet
and the Puppet modules are all FIPS compliant?
I suppose we could tests for this? For example, if a user provides custom
SSL certificates for the web server we could run a check and warn a user?Before a certificate is installed, yes. Might get involved though (as most things tend to when it comes to SSL/TLS and certificates).
Is it possible to detect that a system is running in FIPS mode to onlyYes, it is possible to detect that a system is running in FIPS mode.
warn in that case?
A few times a day at first -- to help resolve w/e issues we currently have. After that -- w/e pattern/schedule we currently have or plan to have, in other words, we run *all* CI in FIPS mode.
How often do you see this CI job running? And I assume this would be
temporary until we are fully FIPS compliant?
cheers,
-d
Thanks for proposal. I have no comments on our findings, looks like a
good approach to the problem. Perhaps - can you bit elaborate the GDB
thing? Is this some kind of hook that use used for FIPS stack to
report "mistakes" (e.g. signal or exception when you attempt to use
md5 hash)? I wonder if there is a way to catch these without GDB, for
example with SystemTap which would allow to catch these also in
production. Perhaps I misinterpret how this works, I haven't opened
the PDFs to be honest yet :-)
I understand you only investigated core and smart proxy and I assume plugin authors will need to do the similar audit themselves. Since we are dealing with security, I would like to propose to create kind of FIPS CHECKLIST somewhere (perhaps our wikipage) so we could follow it and also add own findings/places to look for. In security world it's much more useful to have blueprints or checklists, I would not like to reinvent the wheel when doing this for discovery or other plugins.
Once we settle down on your proposal, please do create tracker issue(s) so we can associate our audits or changes to it. If you find anything that needs to be done for all plugins, please create tracker issue as well. Thanks!
LZ
--
Later,
Lukas @lzap Zapletal
I understand you only investigated core and smart proxy and I assume plugin authors will need to do the similar audit themselves. Since we are dealing with security, I would like to propose to create kind of FIPS CHECKLIST somewhere (perhaps our wikipage) so we could follow it and also add own findings/places to look for. In security world it's much more useful to have blueprints or checklists, I would not like to reinvent the wheel when doing this for discovery or other plugins.
Once we settle down on your proposal, please do create tracker issue(s) so we can associate our audits or changes to it. If you find anything that needs to be done for all plugins, please create tracker issue as well. Thanks!
LZ
···
On Thu, Nov 16, 2017 at 7:35 PM, Dmitri Dolguikh <witlessbird@gmail.com> wrote:What is FIPS?
From Wikipedia [1]: The Federal Information Processing Standard (FIPS)
Publication 140-2, (FIPS PUB 140-2), is a U.S. government computer
security standard used to approve cryptographic modules. The title is
Security Requirements for Cryptographic Modules.
What are Implications of FIPS 140-2 Support for Foreman, Katello, and
Smart-Proxy?
Linux system, or rather an SSL library in FIPS-compatible mode will
only have a set of ciphers and hash functions compatible with FIPS.
[2] contains the list of approved cryptographic functions, Oracle
graciously compiled the list of not approved ones, which is more
useful and can be found at [3].
OpenSSL in FIPS mode
My understanding is that only OpenSSL versions 1.0.1 and 1.0.2 have
FIPS 140-2 validated cryptographic modules. OpenSSL raises ABRT signal
when it receives a call to one of the unapproved ciphers/functions.
Foreman in FIPS mode
I haven’t looked at pulp, candlepin, qpid, goferd, etc, and at point
don’t know how and if these can be made to work in FIPS mode. All
tests I’ve done so far were against Rails 5.0, Considering the number
of dependencies, we will need to limit FIPS support to just one
version of Rails.
Rails and other (ruby) dependencies.
MD5 is used (hard-coded) in a few places in Rails, at this point I’m
quite certain that its use is constrained to various built-in caches.
I had to disable *all* Rails caches to be able to run Foreman in FIPS
mode. Additionally, strong ETAG’s cannot be used, I’m not sure if they
are used, or there are plans for them.
Spring uses MD5 to generate application ID, but will use one in
SPRING_APPLICATION_ID environment variable if it’s available.
Gravatar uses MD5 hashes in their urls, doesn’t look like other hashes
are supported.
I think apipie cache uses MD5, but I will need to verify this.
Foreman
app/services/password_crypt uses MD5 for grub2 passwords, which will
need to be switched to SHA512. MD5 will need to be removed from the
list of hash functions
SshKey#generate_fingerprint, call to SSHKey.fingerprint uses MD5
A note: with caching disabled, and issues above fixed, I was able to
get Foreman suite of tests to pass, and get Foreman to start.
Smart-Proxy
Smart-Proxy codebase appears to be compatible with FIPS (ran and
passed tests ok without any changes), but there are issues with
external depdencies.
DHCPD uses MD5-based omapi shared secret. DHCPD shared secret with
bind is also md5-based.
BIND when used with dhcpd uses MD5 hashes stored in TXT as host id.
Puppet needs to be run in FIPS mode (FIPS-compatible hash function
needs to be configured). I assume this covers all of puppet, including
mcollective, puppet run, puppetca.
BMC/IPMI authentication can use MD5 or lower based hashes, older
clients may not have newer hash functions.
Salt appears to use MD5 hashes by default, individual nodes must be
configured to use other hash_type
Any 3rd party SSL certificates that may need to be verified or decoded
by either Foreman or Smart-Proxy must be generated using
FIPS-compatible algorithms/hash functions.
How we can reach FIPS compatibility
The easiest first step would be to replace offending cryptographic and
hash functions in Foreman, and in Smart-Proxy case, 3rd party
configuration files with FIPS-compatible ones. Additionally, any new
code changes that employ MD5 or other non approved functions shouldn’t
be accepted.
The next step would be to create a CI job that will continuously
execute the the full suite of tests on a VM with FIPS mode enabled.
GDB configured with Ruby’s project .gdbinit [4] and a tiny batch [5]
of commands can be used to report on FIPS-related failures.
Considering the amount of dependencies Foreman and Smart-Proxy have, I
think would be useful to have all CI environments switched to run in
FIPS mode: this should increase the probability of discovering of new
FIPS-related issues before our users.
Lastly, a FIPS-compatible caching solution for Rails needs to be
found, if none exist, an existing one needs to be modified to support
FIPS.
Any feedback would be appreciated,
-d
[1] Wikipedia article on FIPS 140-2,
[2] Approved Security Functions for FIPS 140-2,
[3] List of algorithms not approved for FIPS 140-2,
[4] Ruby project’s gdb helper functions,
https://github.com/ruby/ruby/blob/trunk/.gdbinit
[5] Catching SIGABRTs with gdb and ruby-specific .gdbinit,
https://gist.github.com/witlessbird/904fefb0031c2eda96da61bd19424c86
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Later,
Lukas @lzap Zapletal
Perhaps - can you bit elaborate the GDBDGB + ruby-specific .gdbinit are used to report ruby callstack when ABRT is caught. I didn't look for alternatives, seeing that gdb + .gdbinit combination is pretty trivial to use. I don't really care what we use here as long as we can get ruby callstack back.
thing? Is this some kind of hook that use used for FIPS stack to
report "mistakes" (e.g. signal or exception when you attempt to use
md5 hash)? I wonder if there is a way to catch these without GDB, for
example with SystemTap which would allow to catch these also in
production. Perhaps I misinterpret how this works, I haven't opened
the PDFs to be honest yet :-)
Sure, sounds like a good idea. I think it's going to consist of the list of hash-functions and ciphers not to use and a recommendation to test changes on a system with FIPS enabled.
I understand you only investigated core and smart proxy and I assume
plugin authors will need to do the similar audit themselves. Since we
are dealing with security, I would like to propose to create kind of
FIPS CHECKLIST somewhere (perhaps our wikipage) so we could follow it
and also add own findings/places to look for. In security world it's
much more useful to have blueprints or checklists, I would not like to
reinvent the wheel when doing this for discovery or other plugins.
Tracker issue for Foreman already exists:
Once we settle down on your proposal, please do create tracker
issue(s) so we can associate our audits or changes to it. If you find
anything that needs to be done for all plugins, please create tracker
issue as well.
http://projects.theforeman.org/issues/3511.
-d
···
On Mon, Nov 20, 2017 at 2:05 AM, Lukas Zapletal <lzap@redhat.com> wrote:
Thanks, so ABRT is raised when a FIPS-enabled library (e.g. openssl)
catches an attempt to use non-approved cipher.
Do you have some examples how such an abort looks like with and without GDB?
Why I am asking. Once we start supporting FIPS, I wonder how we are gonna provide support for users and customers who are in FIPS mode and "someting is breaking apart". Installing GDB would be really the last thing to do. I wonder if there is any chance of using SystemTap or even ABRT (Automatic Bug Reporting Tool) data (stacktrace with debug info, core dump). Trying to find a workflow which we will use for retrospective analysis to find the Ruby code which is causing this.
Was wondering if we can hook into ABRT and print stacktrace similarly as we do for:
LZ
--
Later,
Lukas @lzap Zapletal
Do you have some examples how such an abort looks like with and without GDB?
Why I am asking. Once we start supporting FIPS, I wonder how we are gonna provide support for users and customers who are in FIPS mode and "someting is breaking apart". Installing GDB would be really the last thing to do. I wonder if there is any chance of using SystemTap or even ABRT (Automatic Bug Reporting Tool) data (stacktrace with debug info, core dump). Trying to find a workflow which we will use for retrospective analysis to find the Ruby code which is causing this.
Was wondering if we can hook into ABRT and print stacktrace similarly as we do for:
LZ
···
On Mon, Nov 20, 2017 at 8:05 PM, Dmitri Dolguikh <witlessbird@gmail.com> wrote:
On Mon, Nov 20, 2017 at 2:05 AM, Lukas Zapletal <lzap@redhat.com> wrote:
Perhaps - can you bit elaborate the GDB
thing? Is this some kind of hook that use used for FIPS stack to
report "mistakes" (e.g. signal or exception when you attempt to use
md5 hash)? I wonder if there is a way to catch these without GDB, for
example with SystemTap which would allow to catch these also in
production. Perhaps I misinterpret how this works, I haven't opened
the PDFs to be honest yet :-)
DGB + ruby-specific .gdbinit are used to report ruby callstack when ABRT is
caught. I didn't look for alternatives, seeing that gdb + .gdbinit
combination is pretty trivial to use. I don't really care what we use here
as long as we can get ruby callstack back.
I understand you only investigated core and smart proxy and I assume
plugin authors will need to do the similar audit themselves. Since we
are dealing with security, I would like to propose to create kind of
FIPS CHECKLIST somewhere (perhaps our wikipage) so we could follow it
and also add own findings/places to look for. In security world it's
much more useful to have blueprints or checklists, I would not like to
reinvent the wheel when doing this for discovery or other plugins.
Sure, sounds like a good idea. I think it's going to consist of the list of
hash-functions and ciphers not to use and a recommendation to test changes
on a system with FIPS enabled.
Once we settle down on your proposal, please do create tracker
issue(s) so we can associate our audits or changes to it. If you find
anything that needs to be done for all plugins, please create tracker
issue as well.
Tracker issue for Foreman already exists:
http://projects.theforeman.org/issues/3511.
-d
--
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
Later,
Lukas @lzap Zapletal
Do you have some examples how such an abort looks like with and without GDB?
with gdb + .gdbinit:
Program received signal SIGABRT, Aborted.
0x00007fb45991e1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); $1 = (rb_vm_t *) 0x117ef20
* #<Thread:0x11b5ef8 rb_thread_t:0x117f5e0 native_thread:0x7fb45ad27740> 0x7fb45259accc <rb_digest_class_s_digest at digest.c:459>:in `digest' (irb):5:in `irb_binding'
0x7fb45a80b318 <rb_f_eval at vm_eval.c:1437>:in `eval'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb/workspace.rb:87:in `evaluate'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb/context.rb:381:in `evaluate'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb.rb:493:in `block (2 levels) in eval_input'
...
with gdb only:
Program received signal SIGABRT, Aborted.
0x00007f080307f1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt
#0 0x00007f080307f1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f08030808e8 in __GI_abort () at abort.c:90
#2 0x00007f07f9f2498f in OpenSSLDie (file=file@entry=0x7f07fa066cd3 "md5_dgst.c", line=line@entry=82,
assertion=assertion@entry=0x7f07fa066cb0 "Digest MD5 forbidden in FIPS mode!") at cryptlib.c:1002
#3 0x00007f07f9f2b209 in MD5_Init (c=0x34af790) at md5_dgst.c:82 #4 0x00007f07fbcfbea0 in algo_init (algo=0x7f07fa518d60 <md5>, pctx=0x34af790) at digest.c:586
#5 0x00007f07fbcfbf4d in rb_digest_base_alloc (klass=54958520) at digest.c:606
#6 0x00007f0803e6a6de in rb_obj_alloc (klass=54958520) at object.c:1863 #7 0x00007f07fbcfbd32 in rb_digest_class_s_digest (argc=0, argv=0x7f0804387668, klass=54958520) at digest.c:472
#8 0x00007f0803f5997b in call_cfunc_m1 (func=0x7f07fbcfbccc <rb_digest_class_s_digest>, recv=54958520, argc=1,
argv=0x7f0804387660) at vm_insnhelper.c:1589
...
I tried installing an abrt signal handler when I was preparing this note, but it never gets called, possibly due to openssl calling exit immediately after raising 'abrt' [1]. Similarly, rubygem-abrt isn't catching 'abrt' signals raised by openssl. The only thing that worked for me was the approach I documented.
-d
[1]
http://docs.huihoo.com/doxygen/openssl/1.0.1c/cryptlib_8c_source.html#l00912
with gdb + .gdbinit:
Program received signal SIGABRT, Aborted.
0x00007fb45991e1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); $1 = (rb_vm_t *) 0x117ef20
* #<Thread:0x11b5ef8 rb_thread_t:0x117f5e0 native_thread:0x7fb45ad27740> 0x7fb45259accc <rb_digest_class_s_digest at digest.c:459>:in `digest' (irb):5:in `irb_binding'
0x7fb45a80b318 <rb_f_eval at vm_eval.c:1437>:in `eval'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb/workspace.rb:87:in `evaluate'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb/context.rb:381:in `evaluate'
/home/vagrant/.rvm/rubies/ruby-2.4.2/lib/ruby/2.4.0/irb.rb:493:in `block (2 levels) in eval_input'
...
with gdb only:
Program received signal SIGABRT, Aborted.
0x00007f080307f1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
56 return INLINE_SYSCALL (tgkill, 3, pid, selftid, sig); (gdb) bt
#0 0x00007f080307f1f7 in __GI_raise (sig=sig@entry=6) at
../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1 0x00007f08030808e8 in __GI_abort () at abort.c:90
#2 0x00007f07f9f2498f in OpenSSLDie (file=file@entry=0x7f07fa066cd3 "md5_dgst.c", line=line@entry=82,
assertion=assertion@entry=0x7f07fa066cb0 "Digest MD5 forbidden in FIPS mode!") at cryptlib.c:1002
#3 0x00007f07f9f2b209 in MD5_Init (c=0x34af790) at md5_dgst.c:82 #4 0x00007f07fbcfbea0 in algo_init (algo=0x7f07fa518d60 <md5>, pctx=0x34af790) at digest.c:586
#5 0x00007f07fbcfbf4d in rb_digest_base_alloc (klass=54958520) at digest.c:606
#6 0x00007f0803e6a6de in rb_obj_alloc (klass=54958520) at object.c:1863 #7 0x00007f07fbcfbd32 in rb_digest_class_s_digest (argc=0, argv=0x7f0804387668, klass=54958520) at digest.c:472
#8 0x00007f0803f5997b in call_cfunc_m1 (func=0x7f07fbcfbccc <rb_digest_class_s_digest>, recv=54958520, argc=1,
argv=0x7f0804387660) at vm_insnhelper.c:1589
...
Why I am asking. Once we start supporting FIPS, I wonder how we are
gonna provide support for users and customers who are in FIPS mode and
"someting is breaking apart". Installing GDB would be really the last
thing to do. I wonder if there is any chance of using SystemTap or
even ABRT (Automatic Bug Reporting Tool) data (stacktrace with debug
info, core dump). Trying to find a workflow which we will use for
retrospective analysis to find the Ruby code which is causing this.
Was wondering if we can hook into ABRT and print stacktrace similarly
as we do for:
https://github.com/theforeman/foreman/blob/698e916ce208b5040b83a908a058c8
3c94d158ee/config/initializers/sig_ttin_trap.rb
I tried installing an abrt signal handler when I was preparing this note, but it never gets called, possibly due to openssl calling exit immediately after raising 'abrt' [1]. Similarly, rubygem-abrt isn't catching 'abrt' signals raised by openssl. The only thing that worked for me was the approach I documented.
-d
[1]
http://docs.huihoo.com/doxygen/openssl/1.0.1c/cryptlib_8c_source.html#l00912
You may not be getting an ABRT because ruby was patched some time ago to
catch this ABRT and a handler was created to make this a non-fatal error
(ruby used to just core dump - see
https://bugzilla.redhat.com/show_bug.cgi?id=717709). I suspect that is why you aren't seeing the expected signal further down the stack.
https://bugzilla.redhat.com/show_bug.cgi?id=717709). I suspect that is why you aren't seeing the expected signal further down the stack.
···
On Tuesday, November 21, 2017 at 1:43:24 PM UTC-7, Dmitri Dolguikh wrote:
I tried installing an abrt signal handler when I was preparing this note,
but it never gets called, possibly due to openssl calling exit immediately
after raising 'abrt' [1]. Similarly, rubygem-abrt isn't catching 'abrt'
signals raised by openssl. The only thing that worked for me was the
approach I documented.
-d
[1]
http://docs.huihoo.com/doxygen/openssl/1.0.1c/cryptlib_8c_source.html#l00912
This doesn't explain the behaviour I'm seeing -- an exception would be
caught and logged, and foreman would continue to run. I'm seeing the app
exit without any stacktraces or core-dumps. I'm not using system ruby
either.
-d
-d
···
On Wed, Nov 22, 2017 at 9:33 AM, James Shewey <jdshewey@gmail.com> wrote:You may not be getting an ABRT because ruby was patched some time ago to
catch this ABRT and a handler was created to make this a non-fatal error
(ruby used to just core dump - see https://bugzilla.redhat.com/
show_bug.cgi?id=717709). I suspect that is why you aren't seeing the
expected signal further down the stack.
I added a bunch of (more) granular-level issues to
http://projects.theforeman.org/issues/3511 tracker.
Cheers,
-d
http://projects.theforeman.org/issues/3511 tracker.
Cheers,
-d
···
On Wed, Nov 22, 2017 at 2:13 PM, Dmitri Dolguikh <witlessbird@gmail.com> wrote:On Wed, Nov 22, 2017 at 9:33 AM, James Shewey <jdshewey@gmail.com> wrote:
You may not be getting an ABRT because ruby was patched some time ago to
catch this ABRT and a handler was created to make this a non-fatal error
(ruby used to just core dump - see https://bugzilla.redhat.com/sh
ow_bug.cgi?id=717709). I suspect that is why you aren't seeing the
expected signal further down the stack.
This doesn't explain the behaviour I'm seeing -- an exception would be
caught and logged, and foreman would continue to run. I'm seeing the app
exit without any stacktraces or core-dumps. I'm not using system ruby
either.
-d