Debian/Ubuntu nightly issues

Hello,

For the past 3 weeks or so, Debian and Ubuntu nightlies have not been published due to the packages failing to pass systests (sanity checks that make sure the packages are installable and can perform basic operations prior to publishing them). This issue is also blocking our ability to release new versions of plugins, as they depend on a working build for their tests.

So far, we have identified 3 different issues that are causing the failures:

  1. (All versions) One of the tests checks that the correct version of foreman has been installed. This test was failing following https://github.com/theforeman/foreman-infra/commit/69679a212ab675b38e9c4b65d7796cc849e1767e, prior to which the version check was being skipped due to the variable not being passed correctly to the test.
    This has now been fixed with https://github.com/theforeman/foreman-infra/commit/c490fadba973e77396e6dc1016454cd26ce83028 that passes the correct value to the tests.
  2. (Ubuntu Bionic only) Some dependent packages were missing on the staging server for Bionic, causing inability to install the installer. This has been fixed now thanks to @mmoll who ensured all the needed packages are present.
  3. (Ubuntu Bionic and Debian Stretch) The images being spun up for the tests fail to correctly set their domain name, leading the installer to fail due to missing $::domain parameter. There appears to be an issue with the Rackspace images that attempt to set the hostname from nova-client using hostnamectl, which fails. This is possibly caused due to dbus not running yet at the time the agent attempts to set the hostname. A support call has been opened to Rackspace to fix this issue.

We will update this thread as we have further information, and thanks @mmoll and @ekohl for your efforts in debugging this.

There has been some progress on #3 with https://github.com/theforeman/forklift/commit/8f5d1ffba5e5203274c13f3613687a9ca4bd9ca2 that ensures the domain is set correctly, this workaround has fixed Bionic but there are still issues with it on Stretch. Thanks @ekohl for working on this!

I only tested on Ubuntu and assumed Debian worked in the same way. Assumptions … Note that they install xe-guest-utilities on Debian Stretch which isn’t in the Debian archives. It is in Ubuntu but provides other files.

https://github.com/theforeman/forklift/pull/818

The final issue has now been fixed and we now have Debian and Ubuntu nightlies being built again after more than a month.
Thanks @ekohl, @mmoll and everyone else who was involved in fixing these issues!

2 Likes