Problem:
We have several Recurring Logics to automatically run some commands on multiple host_collections.
After running this morning on several of the RLs the Next occurrence is older than the Last occurrence
$ hammer recurring-logic info --id 24 --fields 'Id,Last occurrence,Next occurrence'
ID: 24
Last occurrence: 2021-11-24 08:30:12 UTC
Next occurrence: 2021-11-24 08:30:00 UTC
$ hammer recurring-logic info --id 26 --fields 'Id,Last occurrence,Next occurrence'
ID: 26
Last occurrence: 2021-11-24 08:15:11 UTC
Next occurrence: 2021-11-24 08:15:00 UTC
Expected outcome:
Something like this:
$ hammer recurring-logic info --id 24 --fields 'Id,Last occurrence,Next occurrence'
ID: 24
Last occurrence: 2021-11-24 08:30:12 UTC
Next occurrence: 2021-11-25 08:30:00 UTC
$ hammer recurring-logic info --id 26 --fields 'Id,Last occurrence,Next occurrence'
ID: 26
Last occurrence: 2021-11-24 08:15:11 UTC
Next occurrence: 2021-11-25 08:15:00 UTC
$ hammer recurring-logic info --id 27 --fields 'Id,Last occurrence,Next occurrence'
ID: 27
Last occurrence: 2021-11-24 08:30:13 UTC
Next occurrence: 2021-11-24 08:30:00 UTC
Foreman and Proxy versions:
Foreman version: 3.0.1
Foreman and Proxy plugin versions:
Plugins:
- foreman-tasks 5.1.1
- foreman_puppet 1.0.5
- foreman_remote_execution 4.7.0
- katello 4.2.1
Distribution and version:
CentOS Linux release 7.9.2009 (Core)
Other relevant data:
We had this issue about a month ago and we ended disabling the old RLs and created new ones.
Database info:
foreman=# select * from foreman_tasks_recurring_logics where id in (24,26,27);
id | cron_line | end_time | max_iteration | iteration | task_group_id | state | triggering_id | purpose
----+-----------------+----------+---------------+-----------+---------------+--------+---------------+---------
24 | 30 08 * * * | | | 40 | 974 | active | 110 |
27 | 30 08 * * * | | | 40 | 980 | active | 113 |
26 | 15 8 2-31/2 * * | | | 20 | 978 | active | 112 |
(3 rows)
foreman=# select * from foreman_tasks_triggerings where id in (select triggering_id from foreman_tasks_recurring_logics where id in (24,26,27));
id | mode | start_at | start_before
-----+-----------+----------------------------+--------------
110 | recurring | 2021-10-15 15:36:09.369358 |
112 | recurring | 2021-10-15 15:44:36.952605 |
113 | recurring | 2021-10-15 15:46:01.406246 |
(3 rows)
We also double checked and all the task of this morning finished successfully.
$ hammer task info --id c20ed7f6-eed9-4593-9efa-7410d07359c8 # RL 24
ID: c20ed7f6-eed9-4593-9efa-7410d07359c8
Action: Run hosts job: Run yum update -y
State: stopped
Result: success
Started at: 2021/11/24 08:30:12
Ended at: 2021/11/24 08:30:23
Owner: ams
Task errors:
$ hammer task info --id 3d0bf172-b5e7-4e56-a20a-c957ca6dd437 # RL 26
ID: 3d0bf172-b5e7-4e56-a20a-c957ca6dd437
Action: Run hosts job: Run yum update -y; needs-restarting -r; if [ "$(echo $?)" -eq "1" ]; then shutdown -r 1 ; fi
State: stopped
Result: success
Started at: 2021/11/24 08:15:11
Ended at: 2021/11/24 08:15:22
Owner: ams
Task errors:
Although none failed the execution of RL 26 did not for any host - we made no changes to the host_collection
Description Search Query Status Succeeded Failed Pending Total hosts Start
Run yum update -y; needs-restarting -r; if [ "$(echo $?)" -eq "1" ]; then shu... host_collect... succeeded 0 0 0 0 about 10 hours ago
Run yum update -y; needs-restarting -r; if [ "$(echo $?)" -eq "1" ]; then shu... host_collect... failed 6 1 0 7 2 days ago
Run yum update -y; needs-restarting -r; if [ "$(echo $?)" -eq "1" ]; then shu... host_collect... failed 6 1 0 7 4 days ago
Run yum update -y; needs-restarting -r; if [ "$(echo $?)" -eq "1" ]; then shu... host_collect... succeeded 6 0 0 6 6 days ago
hammer host-collection info --id 4
Id: 4
Name: auto-update-and-reboot-even-days
Limit: None
Description:
Total Hosts: 7