Rake reports:daily runs reports twice

Support
1

Since our update from 3.15 to 3.16 a few days ago I have noticed that the daily reports at 7am in the morning run twice. /etc/cron.d/foreman contains it only once. I have verified the issue by running /usr/sbin/foreman-rake reports:daily manually. I get reports twice.

For instance, I get the “Configuration Management Summary Report” twice. The first one is a “Summary from 1 day ago to now”, the second one immediately after that is “Summary from less than a minute ago to now”. So it’s no email duplication but actually the report runs twice and generates for the timeperiod from the previous one.

2

Does anyone have an idea?

3

When you run the command manually from the command line, do you also get two reports?

Can you tell from logs something relevant?

4

Yes. When I run

$ RAILS_ENV=production FOREMAN_HOME=/usr/share/foreman /usr/sbin/foreman-rake reports:daily

as user foreman I’ll get all reports twice. Again, the first configuration management summary says:

Summary from about 10 hours ago to now

the second says:

Summary from less than a minute ago to now

I don’t really know what’s relevant in this case. I can see in production.log that it sends 6 emails and that audit_mailer, host_mailer, and errata_mailer are “rendered” twice. But I don’t see anything else suspicious.

5

You likely have defined the same notification twice. Do this:

foreman-rake console
> UserMailNotification.unscoped.all
6

Here is the output.

# foreman-rake console
Loading production environment (Rails 6.0.3.1)
irb(main):001:0> UserMailNotification.unscoped.all
=> #<ActiveRecord::Relation [
#<UserMailNotification id: 2, user_id: 5, mail_notification_id: 9, last_sent: nil, interval: "Subscribe to all hosts", created_at: "2020-05-13 07:08:40", updated_at: "2020-05-13 07:08:40", mail_query: nil>,
#<UserMailNotification id: 4, user_id: 5, mail_notification_id: 7, last_sent: nil, interval: "Subscribe", created_at: "2020-05-13 07:08:40", updated_at: "2020-05-13 07:08:40", mail_query: nil>,
#<UserMailNotification id: 6, user_id: 5, mail_notification_id: 3, last_sent: nil, interval: "Subscribe", created_at: "2020-05-13 07:08:40", updated_at: "2020-05-13 07:08:40", mail_query: nil>,
#<UserMailNotification id: 7, user_id: 5, mail_notification_id: 2, last_sent: nil, interval: "Subscribe", created_at: "2020-05-13 07:08:40", updated_at: "2020-05-13 07:08:40", mail_query: nil>,
#<UserMailNotification id: 1, user_id: 5, mail_notification_id: 6, last_sent: "2020-08-20 05:00:15", interval: "Daily", created_at: "2020-05-13 07:08:40", updated_at: "2020-08-20 05:00:15", mail_query: "">,
#<UserMailNotification id: 3, user_id: 5, mail_notification_id: 4, last_sent: "2020-08-20 05:00:15", interval: "Daily", created_at: "2020-05-13 07:08:40", updated_at: "2020-08-20 05:00:15", mail_query: nil>,
#<UserMailNotification id: 5, user_id: 5, mail_notification_id: 1, last_sent: "2020-08-20 05:00:15", interval: "Daily", created_at: "2020-05-13 07:08:40", updated_at: "2020-08-20 05:00:15", mail_query: nil>]>
7

See, there were likely created somehow by error. Can you remove them in the UI/CLI?

If not, you can easily delete them via:

UserMailNotification.delete(ID)
8

I want those e-mails. Just only once, not twice. For testing, I have removed all notifications from my accounts email preferences pages and the list was empty. Then I have put everything back and it kind of looks the same again. When I run the reports manually with

foreman-rake reports:daily

I’ll get the reports twice again.

If I set all settings in my account to “No emails” and run it manually, I get nothing.

Put it back, and it’s twice again. So basically, foreman-rake reports:daily generates all reports twice.

9

Sounds like a bug, please file a redmine issue. I don’t know what is wrong to be honest.

1 Like
10

Issue #30670 submitted.

11

I’ve tried to replicate your problem but I can’t. Do you have any further information about the bug like version of Foreman? Is it happens at the develop version of Katello?

How is configured your Katello? In other words, could you describe minimal configuration on clean Katello/Foreman instance?

And maybe silly question… Do you have two users with same email?

12

I only install and update to the official release versions. Katello 3.15 was O.K. With 3.16 the issue started. I am on the latest version on CentOS 7, i.e. Foreman 2.1.2 and Katello 3.16.0.

The original installation (3.15) was like this:
foreman-installer --scenario katello --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh --foreman-plugin-tasks-automatic-cleanup=true
plus some certs option to use our own PKI. Later I have added the realm proxy. Beyond that, all real configuration I did was for the content management.

That was my first though, too, but no. I even checked the users table in the postgresql database.

13

It seems the “Remove orphans” task which is running once a week is also duplicated: every week it runs twice. One task succeeds the other one gets paused in error state. They both start at the same time thus I guess the paused one gets locked out from the first one. I suppose “Remove orphans” should also only run once, not twice?

14

Hello,

I’ve tried to reproduce the behavior on my production setup and I see the same. It is really caused by Katello, more specifically this line (thanks @tbrisker for help with debugging). This is a broader issue of using load for rake tasks in Katello, which is why you also see duplicated Remove orphans task.

We should depend on dynflow:client task in Foreman core, every plugin can add a mail notification that may need to start dynflow client in order to schedule some task. We should avoid trying to do this from plugins, this feels as a good core service to offer.

As a workaround until this is properly fixed, you can apply the following patch

diff --git a/lib/katello/tasks/reports.rake b/lib/katello/tasks/reports.rake
index f2e04b8..5ae9dac 100644
--- a/lib/katello/tasks/reports.rake
+++ b/lib/katello/tasks/reports.rake
@@ -1,4 +1,5 @@
 load "#{Rails.root}/lib/tasks/reports.rake"
+Rake::Task['reports:daily'].clear
 
 #Katello reports can generate a foreman task, so mark the task as a dynflow client
 ["reports:daily", "reports:weekly", "reports:monthly"].each { |task| Rake::Task[task].enhance ["dynflow:client"] }

you can apply the same on weekly and monthly if needed. We’ll be happy to hear back if that helped.

15

cc @katello, this deserves more attention, duplicating tasks can cause weird issues