Katello seeding failing with foreman tasks

Hello!

The Problem

Currently db:seed is failing in both production and development environments for katello. There is some discussion here: Fixes #32086 - seed triggered by sidekiq processes by m-bucher · Pull Request #8392 · theforeman/foreman · GitHub

This PR introduced a transaction for db:seeds (which is a good thing!), but katello has for a long time triggered a ForemanTask synchronously. The issue is that Foreman tasks executes actions within their own transactions and since the seeding transaction is not closed, the org and users that were created in previous seed files do not exist within the execution of the task. This causes some pretty interesting and spectacular errors!

I’ve filed this: Bug #32277: Seed fails with PG::ForeignKeyViolation: ERROR: insert or update on table "foreman_tasks_tasks" violates foreign key constraint "fk_rails_a56904dd86" - Katello - Foreman

The Fix

A few of us got together and discussed various fixes and settled upon this:

  • The foreman task executed from our seed file will be removed
  • We will replace it with code that creates Active Record objects for the default content view and Library lifecycle environment (which is happening in the task idempotently already)
  • we will add a db column to easily tell us for a given organization if it has been created in candlepin yet
  • We will add a before_filter to certain controllers (products, candlepin_proxies, subscriptions) to run a check before every action
  • in this check, if the new db column indicates an org has not been fully created in candlepin, it will run a synchronous task to create the organization and environment in candlepin
  • we will enhance the organization creation task to be much more tolerant to race conditions, as multiple of these could be run at a time if multiple requests came in immediately

@Jonathon_Turel has volunteered to pick up this task

thoughts?

Thanks,
Justin

4 Likes

Draft pr opened here: Fixes #32277 - make org creation idempotent and async from seeding by jturel · Pull Request #9270 · Katello/katello · GitHub

Thanks for sharing! What happens if the task fails? Will this also improve the situation when I enter a page that uses organization and Candlepin is down? I think it would deserve a nicer error page explaining what happened instead of generic 500.