New test suite: template snapshots

lzap · September 25, 2018, 7:48am

Hello, we’ve merged template snapshots test suite. It basically renders templates in safe and non safe mode mentioned in this list:

github.com

theforeman/foreman/blob/develop/test/unit/foreman/renderer/snapshots.yaml

# Edit this file and then run: rake snapshots:generate RAILS_ENV=test

files:
  - provision/alterator_default.erb
  - cloud_init/cloud_init_default.erb
  - PXELinux/pxelinux_global_default.erb
  - PXEGrub/pxegrub_global_default.erb
  - PXEGrub2/pxegrub2_global_default.erb
  - PXELinux/pxelinux_default_local_boot.erb
  - PXEGrub/pxegrub_default_local_boot.erb
  - PXEGrub2/pxegrub2_default_local_boot.erb
  - PXELinux/pxelinux_default_memdisk.erb
  - PXELinux/pxelinux_chain_ipxe.erb
  - PXELinux/pxelinux_chain_ipxe_undi.erb
  - provision/alterator_default.erb
  - finish/alterator_default_finish.erb
  - provision/atomic_kickstart_default.erb
  - provision/autoyast_default.erb
  - provision/autoyast_sles_default.erb
  - PXELinux/autoyast_default_pxelinux.erb

This file has been truncated. show original

And compare them to expected snapshots, found here:

If there is a difference, it will be shown as diff by minitest and a test will fail. This is part of our normal test suite executed by Jenkins too.

After review of changes you can either manually do the change in snapshot files or have all snapshot files regenerated using the following rake task:

rake snapshots:generate RAILS_ENV=test

This will prevent issues when a template which we don’t test very often cannot be rendered anymore due to regressions in core, templating stack or template itself. Special attention needed when syncing templates from our community-templates repository.

Big up to Kamil and folks from iRonin and Timo and his team at DM for making this happen. This was only possible due to big refactoring of our templating stack.

TimoGoebel · September 25, 2018, 9:38am

Happy to see this merged.

I currently see two follow up tasks:

we should make this available for plugins (As a plugin author I would like to define templates that are seeded and included in snapshot testing) as most plugins, e.g. Katello, uses custom code to seed templates.
Change our template renderer in community-templates with the renderer we use in core.

stbenjam · September 25, 2018, 1:48pm

This is very cool, thanks @lzap!

I agree it’d be good to have a mechanism for plugins, but Katello no longer seeds templates (other than things for REX). We do rely on the redhat_register snippet, so I think it’d be good to have rendering tests on that one. It’s a rather complicated snippet, so supporting multiple snapshots would be helpful.

lzap · September 25, 2018, 1:57pm

Please don’t thank me but @kamils-iRonin and @TimoGoebel I am just a messenger here. Thanks folks again! This was a pain.