Foreman + Katello and management via ansible (slow)

Problem:

We configure quiet a bit of our foreman instance via ansible in a pipeline. For example: We have a list of 150 Usergroups + 150 Roles which are managed via ansible. One run of this pipeline needs 11 minutes.

It seems that the foreman API is rather slow for something like this. Do some of you have similar problems and maybe found a way to increase the speed?

My next steps:

  • Increase the parallelism of ansible

Foreman and Proxy versions:

Foreman 3.18, Katello 4.20

Distribution and version:

RHEL 9.8

1 Like

Hi @ochnerd ,

Have you gone through our tuning guide? There are quite a few “knobs” that you can turn to help Foreman be more performant depending on your situation. It’s a place to start at least if you haven’t seen it yet.

When you mention having usergroups and roles, it reminds me of this solved issue that I saw somewhat recently: Bug: Orgs and Locs + Job Templates are broken in 3.18 (probably 3.17 as well) - #17 by singularity001

There is a patch in there that could be worth trying depending if the situation is at all similar to yours. The user was reporting that Postgres was doing too much work for some particular endpoints, so it might be helpful if you can narrow down what is taking too long. Is it Postgres? Is it the web server processing things? Or something else? The timing information in the production logs along with postgres logs are your friend here.

2 Likes

Thanks for the fast reply :slight_smile:

I looked at the tuning guide a few time and our Foreman Instance is rather bored right now. The only demanding Tasks are the weekly repository syncs (I’m looking at you elasticsearch repo).

I looked through your linked Bug and have seen that Fixes #39336 - Batch taxonomy subtree lookups to avoid N+1 queries by adamruzicka · Pull Request #10990 · theforeman/foreman · GitHub is merged since last month and the other PR: Fixes #39323 - Optimize grouped authorization filter handling by pablomh · Pull Request #10988 · theforeman/foreman · GitHub is still open.

Our Instance is by far not that big but I hope we can get some improvement. I test the patches as soon as possible.

Is the user you’re using for auth an admin or not? If it’s not an admin, try with an admin for a moment.
Our authorization stack is rather slow and the boolean “is admin” skips quite a lot of checks that cost time.

One example (even if not exactly your case) is Bug #38201: Location creation is very slow unless Administrator checkbox is set - Foreman

1 Like

Oh good idea, the user is not an admin.

I tested it on my own machine with my admin credentials and it goes down from 11 to 2min and 20s. It is nice to know that but I don’t know if I want to give that “Service User” Admin Credentials.

It helped to narrow doen that it is indeed the authorization stack which slows the pipeline/ansible down.

Oh absolutely, this wasn’t meant as “give it admin, it will fix everything”, but if we know skipping permission checks does improve it, we know which direction to look at :slight_smile:

(And by “we” I mean “you”, this has been an issue forever, and nobody dared to tackle it :confused: )

1 Like

I patched our release Foreman Instance with Fixes #39336 - Batch taxonomy subtree lookups to avoid N+1 queries by adamruzicka · Pull Request #10990 · theforeman/foreman · GitHub but I did not get any performance boosts.

I Fixes #39323 - Optimize grouped authorization filter handling by pablomh · Pull Request #10988 · theforeman/foreman · GitHub don’t know how safe it is to patch this because of the db migration. But I will look into it.