User is getting roles from wrong group

Problem:
I have one user (that I know of) which is getting permissions from a usergroup he is not a member of. If I edit the user I see that the group is listed in the Roles tab, under Roles from user groups. But if I look at the actual group the user is not a member (he used to be a member but not any more).

The group membership is collected from external source and fed to Foreman via Apache - this obviously works since the group membership is correct. Looking in the db in usergroup_members everything looks fine too, the user is not a member of the group.

The group is owner of a number of hosts which this user has permissions to view and edit even though he should not.

So how can the user be a member of the group even if he is not listed as a member?

Foreman version:
foreman-3.8.0-1.el8.noarch

Distribution and version:
RHEL8

Other relevant data:
As mentioned, the user was previously a member of the group but not since June. Since then we have restarted the Foreman server several times and also upgraded from 3.7 to 3.8.

I enabled sql debug logs and found the problem, the user is listed as a member of the group in the db table cached_usergroup_members.

How can I purge this cache? And how is it supposed to work, the entry for the user in question has a timestamp from April, why is it still it in the cache?

Managed to find the command foreman-rake fix_db_cache by running foreman-rake -T to list all available rake jobs, which fixed my problem.

The lack of documentation leads me to believe that this should not be needed, but outdated information in the db cache could lead to serious security implications. So I’m still interested in knowing how it could have happened in the first place and how I can make sure it does not happen again.

Can i disable the cache altogether?
Should I run fix_db_cache daily as a cron job?

Grateful for any insights in this!

Regards
Adam

Hello,

this tasks is there exactly for the reason the cache gets corrupted for some reason. This is the first time I hear someone needed to use it since it was created ~10 years ago. It’s definitely not meant to be run regularly, if the cache got corrupted, that means there’s a bug in the permission system. It would be helpful, if you could create reproducer. It will likely be related to the changes of user groups structure and adding/removing users from user groups. Did you do any such changes recently? (you can take a look at audits)

All our groups are linked to an external group and membership is managed elsewhere (fed to Foreman via Apache and mod_lookup_identity).

That said, it is possible there has been some manual intervention sometime but nothing that i can find in our audit logs - but we only keep 6 months of audit logs so maybe it happened before then.

We have been using Foreman for many years now and I’ve never encountered this either so it will be difficult for me to reproduce it. But now i know that this should work without any intervention from our side, we will keep an extra eye on the cache for a while to make sure it’s not recurring.

Thanks!

Curious, what is really the purpose of the cached_usergroup_members as opposed to usergroup_members?