The Foreman project recently discovered that one of our servers, which
hosted websites and package repositories, had been compromised.
We've taken steps to check that the software and packages released
haven't been tampered with, full details below. Our issue tracker
(Redmine) and private GPG key were exposed in this attack, and as a
precaution we've revoked passwords and keys. Affected services have
been moved to replacement servers.
At a minimum, we ask all users to trust our new GPG key only with the
following steps. Redmine users also need to reset passwords.
=== Debian users must:
apt-key del E775FF07
wget -q http://deb.theforeman.org/pubkey.gpg -O- | apt-key add -
or
wget -q
"https://pgp.mit.edu/pks/lookup?op=get&search=0xB3484CB71AA043B8" -O- |
apt-key add -
Verify fingerprint of new key:
apt-key finger
should be 7059 542D 5AEA 367F 7873 2D02 B348 4CB7 1AA0 43B8
=== RPM users must:
rpm -e gpg-pubkey-e775ff07-4cda3cf9
rpm --import http://yum.theforeman.org/RPM-GPG-KEY-foreman
or
rpm --import
"https://pgp.mit.edu/pks/lookup?op=get&search=0xB3484CB71AA043B8"
Verify fingerprint of new key:
$ rpm -qi gpg-pubkey-1aa043b8-53b2e946 | gpg --with-fingerprint -
should be 7059 542D 5AEA 367F 7873 2D02 B348 4CB7 1AA0 43B8
=== Redmine users may reactivate their account via
Foreman while GitHub users
can simply log in again and will need to re-authorise the app.
Full details are included in the following sections if you wish to read on.
Security incident
···
================= In February, our CI/Jenkins infrastructure was compromised through an SSH dictionary attack. The affected CI servers were rebuilt and credentials changed, but the intrusion had spread using SSH keys to the server hosting the website, which wasn't identified at the time.Packages and software integrity
All RPMs back to Foreman 1.2 have been verified against original builds
and to the best of our knowledge, are unaffected. We are unable to
independently verify versions prior to 1.2.
Debian packages for Foreman 1.5.1 have been independently verified and
no signs of unauthorised modification have been found. Versions prior
to 1.5.1 have not been verified, but lack of modifications found thus
far is encouraging. A new build of Foreman 1.4.5 will be made within a
week.
All tarballs on downloads.theforeman.org from Foreman 1.2 onwards have
also been verified and are also unaffected to the best of our knowledge.
No signs of unauthorised modification have been found in our git
repositories, and no credentials for these are believed to have been
compromised in the attack.
Signing keys (GPG)
Effective immediately, a new GPG key has been generated for Debian
package archives. This new key has been used to sign all existing
Debian archives, and new keys will be provided for major releases going
forward.
Foreman 1.4 and 1.5 RPMs have been re-signed and a -2 package released
to update foreman-release.rpm with the new public key. As above, new
keys will be used for major releases going forward.
Key name: Foreman Automatic Signing Key (2014) packages@theforeman.org
Key ID: 0x1AA043B8
Fingerprint: 7059 542D 5AEA 367F 7873 2D02 B348 4CB7 1AA0 43B8
Also available from
https://pgp.mit.edu/pks/lookup?op=get&search=0xB3484CB71AA043B8
Please see the beginning of this e-mail for revocation and trust
instructions.
Redmine issue tracker
The Redmine database was accessible, so all user passwords (and GitHub
OAuth tokens where used) have been reset as a precaution. To access
Redmine again, please use the “Lost password” link to re-gain access:
Foreman. GitHub users can
simply log in again and will be prompted to re-authorise the app.
If you need help with your Redmine account, please do let us know.
Questions
We’ve taken a series of measures in response to this attack intended to
secure our systems and processes. If you have any further questions
about this incident, please don’t hesitate to contact our security team
or myself in confidence at foreman-security@googlegroups.com.
–
Dominic Cleal
Red Hat Engineering