Foreman 1.19 test week

testweek

#1

Foreman 1.19 RC1 has been announced! Foreman 1.19.0-RC1 has been released

It is the time again where we put our effort into making sure that all most important scenarios work. It’s easy to get your hands dirty and help us with testing:

  1. Install Foreman RC version (the most up to date one)
  2. Pick a scenario from this post or add your own scenario
  3. If you find an issue, file it in the RedMine and make a comment in this thread linking the issue
  4. Mark the scenario as checked in this OP (this is an editable wiki post), here is syntax of checked and unchecked lines (you can click on checkboxes directly with mouse):
    • Unchecked
    • Checked
    • Checked (alternative syntax with no special semantics - both are equal)

You can start right away, the ideal timing is from Sunday July 29th until Saturday August 4th but feel free to put your effort anytime before the final release comes out.

Installation

  • Install Foreman using existing script/forklift/beaker
    • RHEL / CentOS latest stable version
    • Debian stable
    • Ubuntu stable LTS
  • Install Foreman manually by following our installation guide
    • RHEL / CentOS latest stable version
    • Debian stable
    • Ubuntu stable LTS
  • Upgrade existing Foreman deployment (advertise in RedMine it was an upgraded instance if you encounter bug)
    • RHEL / CentOS latest stable version
    • Debian stable
    • Ubuntu stable LTS
  • Sanity checks
    • Installation on Red Hat distro with SELinux turned on
    • Packages passenger and tfm-rubygem-passenger are from the same repo (foreman) and in the same version

Provisioning

  • Bare-metal or virtualized PXE provisioning (host exits build mode and reboots)
    • BIOS host with CentOS
    • UEFI host with CentOS
    • BIOS host with Debian or Ubuntu
    • UEFI host with Debian or Ubuntu
    • BIOS host with Atomic OS
  • Compute Resources (VM is successfully created, finish or cloud-init is executed)
    • Create VMware host (Image Based/Network Based)
    • Create OpenStack host (Image Based)
    • Create Ovirt host (Image Based/Network Based)
    • Create Libvirt host (Image Based/Network Based)
    • Creare AWS host (Image Based)
    • Create GCE host
  • Puppet manifest import (classes are imported, parameters recognized)
  • Puppet configuration (class is assigned to a host, agent performs changes, reports and facts appears correctly)
  • Log in using user from LDAP (user account is created from LDAP)
  • Log in using user from FreeIPA (user account is created from FreeIPA)

Foreman Discovery

  • Bare-metal or virtualized provisioning via Provision - Customize Host (host exits build mode and reboots)
    • BIOS with discovery from PXE
    • UEFI with discovery from PXE
    • BIOS with discovery PXE-less
    • UEFI with discovery PXE-less
  • Provision a host via discovery rule
  • Provision a host via Customize UI button
  • Provision a host without hostgroup via Customize UI button
  • Provision a host via hammer via hostgroup
  • Provision a host via hammer via auto provisioning rule

Foreman Bootdisk

  • Bootdisk basic provisioning (host exits build mode and reboots)
    • Full host image
    • Host image
    • Generic image
    • Subnet image

Foreman Ansible

  • Import Roles
    • With/From Smart-Proxy
  • Assign Roles
    • Hostgroup
    • Hosts
  • Play Roles
    • Hostgroup
    • Hosts
  • Run shipped Ansible playbook (job), e.g. to install ansible role from galaxy

works with known issues, atm 2.2.3 is shipped but should be bumped to 2.2.5 - https://github.com/theforeman/foreman-packaging/pull/2804

Foreman Remote Execution

  • Run some job, e.g. ‘ls /etc’ on a system that was provisioned from Foreman, it should work out of the box
  • Run some job against the Foreman host itself, only key configuration should be needed

Foreman Puppet run

  • Trigger Puppet run on host through SSH

Foreman Openscap

  • Create new content file, define a policy, assign it to a host and deploy the foreman_scap_client using puppet
  • Verify ARF report gets uploaded upon foreman_scap_client run and full version of it can be rendered
  • Create tailoring file, assign it to the policy and rerun client with the tailoring file

It seems like port and server puppet class params are not being overrided anymore, I had to manually enable the override, let’s retest with RC3

Foreman Virt Who Configure

  • Create a configuration definition and run it e.g. through REX on some provisioned host. It should succeed as long as it has access to sat tools repo on RHEL, epel (I think) on centos.

Foreman Templates

  • hammer import-templates --lock true # sync newest templates from community-templates repo, see audits
  • mkdir /repo; chown foreman /repo; hammer export-templates --repo /repo # may need setenforce 0

Foreman 1.19 test week event
#2

This affect probably foreman scenario too : Bug #24418: katello scenario with some plugins fails because of missing packages in formean-plugins repository - Katello - Foreman, no ?


pinned #3

#4

GPG check disabled, I upgraded my foreman 1.18 / katello 3.7 test setup.
Package updates finished without problem (though long rake run).
Running foreman-installer --scenario katello --upgrade failed on postgresql step. I m almost sure it s because of it s a postgresql 9.6 (from postgresql.org repo) and the systemd service name is not just “postgresql” but “postgresql-9.6”. There is a postgresql::globals::service_name: postgresql-9.6 in custom-hiera.yaml but it s probably only used by puppet.

[root@foremandev ~]# foreman-installer --scenario katello --upgrade
Resetting puppet server version param...
Upgrading, to monitor the progress on all related services, please do:
  foreman-tail | tee upgrade-$(date +%Y-%m-%d-%H%M).log
Upgrade Step: stop_services...
Redirecting to 'foreman-maintain service'
Running Stop Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Stop applicable services: Stopping the following service(s):

rh-mongodb34-mongod, qdrouterd, qpidd, squid, pulp_resource_manager, pulp_streamer, smart_proxy_dynflow_core, foreman-proxy, pulp_celerybeat, pulp_workers, tomcat, dynflowd, httpd, puppetserver

\ All services stopped                                                [OK]      
--------------------------------------------------------------------------------

katello-service stop finished successfully!
Upgrade Step: start_postgresql...
Redirecting to 'foreman-maintain service'
Running Start Services
================================================================================
Check if command is run as root user:                                 [OK]
--------------------------------------------------------------------------------
Start applicable services: Starting the following service(s):
                                   [FAIL]
No services found matching your parameters
--------------------------------------------------------------------------------
Scenario [Start Services] failed.

The following steps ended up in failing state:

  [service-start]

Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist="service-start"


katello-service start --only postgresql failed! Check the output for error!
Upgrade step start_postgresql failed. Check logs for more information.

[root@foremandev ~]# systemctl list-units "postgres*"
UNIT                   LOAD   ACTIVE SUB     DESCRIPTION
postgresql-9.6.service loaded active running PostgreSQL 9.6 database server

#5

@hartmel, thanks for this report. Could you please file this as an issue against foreman-maintain in our tracker [1]? If not let me know and I can do it.

As a workaround you can try

ln -s /etc/systemd/system/postgresql-9.6.service /etc/systemd/system/postgresql.service

Regards,
Martin

[1] https://projects.theforeman.org/projects/foreman-maintain/issues/new


#6

Created issue Bug #24521: support custom postgresql versions - Foreman Maintain - Foreman
Thanks for the workaround !


#7

Workaround allowed to finish process
[ INFO 2018-08-02T13:46:58 verbose] Installer finished in 1822.137494432 seconds


#8

@hartmel, thanks for your help!


#9

Tried another update run, this time without disabling gpg check on RPMs : Bug #24417: cannot install katello packages without disabling gpg check - Katello - Foreman


#10

Found PXE provisioning blocker issue: Bug #24578: PXE provisioning does not work in 1.19: snippet 'blacklist_kernel_modules' does not exist - Foreman


#11

Among other problems I already created issues for, there are these :



Can someone qualify for which ones I should open issue in redmine ?


#12

With RC2 released - Foreman 1.19.0-RC2 has been released - there is still time to test and make sure we don’t have significant issues in the final 1.19.0 version!
Thanks to everyone who has been testing RC1 so far, thanks to you we have found several blockers that are being addressed with hopes of making 1.19.0 as stable as possible.


#13

Note I messed up the gpgcheck on the plugins repository and that wasn’t fixed in RC2. It was in a previous PR that was closed but didn’t port it to the actual fix PR. I’ll get that into RC3.


#14

Tested foreman_ansible with RC2, the report page with plugin introduced react component is broken again. I thought this was already resolved.

Uncaught Error: Component not found:  ReportJsonViewer among BookmarkContainer, DonutChart, StatisticsChartsList, PowerStatus, NotificationContainer, ToastNotifications, StorageContainer, PasswordStrength, BreadcrumbBar, FactChart
    at Object.markup (bundle-debc7a31d7c1efeae928.js:1)
    at Object.t.mount (bundle-debc7a31d7c1efeae928.js:1)
    at 20:424

#15

It sounds like we need to rebuild all Foreman Plugins after every Foreman build. Also wondering if that the resulting RPM only works with that exact Foreman build.


#16

Isn’t this Bug #24371: In production webpacked_plugins_js_for $PLUGIN doesn’t load the js if the plugin doesn’t provide the sources - Foreman ? if so, it wasn’t merged in time for RC2, and will be in RC3 (but possibly you can apply the patch manually to check if it fixes the issue or not)


#17

I just found that passenger versions are incorrect:

# rpm -qa | grep passenger
tfm-rubygem-passenger-4.0.18-9.12.el7.x86_64
passenger-4.0.53-4.el7.x86_64
tfm-rubygem-passenger-native-libs-4.0.18-9.12.el7.x86_64
mod_passenger-4.0.53-4.el7.x86_64
tfm-rubygem-passenger-native-4.0.18-9.12.el7.x86_64

It’s EPEL vs Foreman yum repo, we should probably update passenger in Foreman to the EPEL version. There are dragons (SELinux paths) so test with SELinux enabled.

https://projects.theforeman.org/issues/24618


#18

Everyone, can you review our checklist? I see that nobody checked “manual installation according our docs”. Looks like everyone is automating this time :slight_smile:

Let’s push for the last couple of days to have a good release!


#19

FYI Bug #24628: foreman installer fails - Foreman


#20

I had a leftover in my install script which turned SELinux to permissive, this was to allow installation when there was a bug. It looks like I forgot to remove this line, so I tested both 1.18 and 1.19 RCs with SELinux turned off.

Can someone confirm 1.19 works fine in enforcing? At least installation.

We have a report on 1.18 which I am currently trying to reproduce:

https://projects.theforeman.org/issues/24616