Yum Commands are very slow

Support
1

To preface, I am in the testing phase with a new Foreman/Katello installation that is being built to replace our Spacewalk server.

I am running Foreman version 2.5.4 w/Katello version 4.1

Thus far, I have enrolled a few test machines and most things are working as expected. The documentation has been intuitive and the forums here have been very helpful.

However I am having one issue that I am concerned my users will not appreciate

Problem:

Yum commands such as “yum repolist” or “yum install [randompackage]” are very slow

They can take upwards of two minutes, all while hanging on the message “Updating Subscription Management Repositories”

Any yum command I run freezes and must go through the “Updating Subscription Management Repositories” cycle first

Expected outcome:

Quicker performance. I am used to Spacewalk’s response time and may be missing something very basic in the configuration here.

If I run yum repolist on a spacewalk enrolled machine, I quickly get a result after pressing enter

If I do the same with a foreman enrolled machine, it takes two minutes

Foreman and Proxy versions:

Foreman version 2.5.4
Proxy version 2.5.4

Foreman and Proxy plugin versions:

Plugin: Katello
Version: 4.1

Distribution and version:

Almalinux 8.6

Other relevant data:

2

I can think of two reasons that would slow things down (or make it feel like things are slow).

  1. When you run any yum (or dnf) command, the subscription-manager plugin communicates with the subscription service on the Foreman server to ensure you have the latest certificates etc. In most cases this should be a quick “is this data still fresh? yes carry on” communication, but depending on the size of your Foreman instance and the number of clients even this can become slow. You’re saying you only have a few clients subscribed so far, so unless you massively underprovisioned (less than 4 CPUs, less than 20G memory) the system Foreman is running on, I would think this is not the source in your case.
  2. By default, we deploy repository configurations with metadata_expire=1, which effectively means that any metadata obtained from Foreman expires after one second and needs to be re-fetched on every invocation of yum/dnf. That means that if the link between the client and the server is “slow” (metadata can easily be a few ten megabytes), you get this observed slowness because it’s fetching the data over and over again.

For the later, I would ask you to post the output of yum repolist -v from a client.
For the former, the relevant time excerpt of /var/log/rhsm/rhsm.log from a client.

Oh, and a note, your Foreman/Katello versions are old and officially unsupported by now, the latest stable release is Foreman 3.3 with Katello 4.5 and 3.4/4.6 are in RC phase right now.

3

Thank you for the quick reply!

  1. This was something I wanted to double check when initially troubleshooting this issue as well. It’s not overprovisioned, but right at the recommended specs. 4 CPUs, 20G Memory. As of now there are only a few clients registered, so I can confirm this is likely not the case.

  2. I noticed this when looking at the /etc/yum.repos.d/redhat.repo

I assumed this would be refreshing the metadata each time yum is invoked, and tried to manually set this to a much higher value just to test. However, I noticed this seems to revert itself automatically.

Is there any way to change this setting from within Foreman?

Yum repolist -v output:

 yum repolist -v
Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, groups-manager, needs-restarting, playground, product-id, repoclosure, repodiff, repograph, repomanage, reposync, subscription-manager, uploadprofile
Updating Subscription Management repositories.
YUM version: 4.7.0
cachedir: /var/cache/dnf
almalinux_8_baseos                                                                                                                              692  B/s | 3.8 kB     00:05
almalinux_8_extras                                                                                                                              560  B/s | 3.8 kB     00:06
almalinux_8_epel                                                                                                                                333  B/s | 1.9 kB     00:05
almalinux_8_powertools                                                                                                                          264  B/s | 1.9 kB     00:07
almalinux_8_appstream                                                                                                                           537  B/s | 4.2 kB     00:08
almalinux_8_atomic                                                                                                                              422  B/s | 3.0 kB     00:07
almalinux_8_mariadb_10.4                                                                                                                        482  B/s | 3.4 kB     00:07
almalinux_8_remi_modular                                                                                                                        453  B/s | 3.5 kB     00:07
almalinux_8_remi_safe                                                                                                                           303  B/s | 3.0 kB     00:10
Repo-id            : Default_Organization_almalinux_8-atomicorp_almalinux_8_atomic
Repo-name          : almalinux_8_atomic
Repo-revision      : 1658063477
Repo-updated       : Sun 17 Jul 2022 01:11:17 PM UTC
Repo-pkgs          : 331
Repo-available-pkgs: 323
Repo-size          : 831 M
Repo-expire        : 1 second(s) (last: Tue 30 Aug 2022 08:24:51 PM UTC)
Repo-filename      : /etc/yum.repos.d/redhat.repo

tail -f /var/log/rhsm/rhsm.log Output


Deleted (rogue):
  <NONE>
2022-08-31 15:58:57,911 [INFO] yum:1654:MainThread @entcertlib.py:132 - certs updated:
Total updates: 0
Found (local) serial# [6157205966916920032, 6761402302984967236, 2835249350155047206, 3359696163646973350, 6372086559134178504, 2327210808087867592, 1593606716902055384]
Expected (UEP) serial# [1593606716902055384, 3359696163646973350, 6157205966916920032, 6372086559134178504, 2835249350155047206, 6761402302984967236, 2327210808087867592]
Added (new)
  <NONE>
Deleted (rogue):
  <NONE>

And thanks, for this install I stuck with what we had in a previous CentOS build. Once we’re comfortable with using foreman we plan to upgrade to the latest versions of both Foreman and Katello

4

After writing my previous reply, I went looking and could not find a way to influence the metadata_expire value from Foreman/Katello. Maybe someone from @katello knows?

And yes, it is resetting itself, because the file is managed by subscription-manager. You probably can do something like subscription-manager repo-override --repo=<the_name> --add=metadata_expire:1000, but I’ve not tested it.

As to your paste. It reports under 1kB/s speeds fetching things from Foreman, resulting in the metadata being downloaded in 50-60 seconds in total, which matches my second idea above.

Now, there can be a ton of possibilities why stuff is so slow.

One thing that I’d ask you to do is to place a “big” (multiple MB) file into /var/www/html/pub/ on the Foreman system and try to download it from the client using curl or wget (the file will be reachable under http://foreman.example.com/pub/filename). Please try both HTTP and HTTPS and report the speed here.

You said you had a Spacewalk box before, was it (network wise) in a similar spot as the new Foreman box?

5

Seems the answer is no:

6

Ah, so that’s just a part of Katello out of the box. Very unfortunate, but nothing we can do about it.

I’ll try running subscription-manager repo-override --repo=<repo name> on the test client to see if this can be manually overridden

Definitely seems there could be a networking issue here since I’m getting <1kB/s speeds

I’ll give the big file idea a shot as well and post the results here.

To your last question, Spacewalk is much snappier. Both Spacewalk and Foreman are on separate subnets from the test clients I’m working with, but this has never had any ill-effects with Spacewalk

7

I tested downloading a 20 MB text file from the foreman /pub directory to one machine subscribed to foreman and to another subscribed to spacewalk. Both of them are on the same subnet.

They each took 9 seconds to download the file

8

20MB in 9 seconds isn’t light speed, but also not as awful as the metadata download above :confused:

What happens if you install a bigger package, like glibc-all-langpacks (26MB)? That should hit roughly the same code as the metadata (authentication etc in Foreman), so you’d have a rough comparison of network vs network+app?

9

I was able to compare and contrast the install times for a machine registered to Spacewalk vs one registered to Foreman while installing glibc

Both took the same amount of time.

I have noticed that yum commands are infinitely more responsive after adding more time to the metadata_expiration value however. Unlike manually editing redhat.repo, the subscription-manager command actually sticks.

I was able to change this all of the repos my test machine is currently subscribed to, but I fear this will be a recurring maintenance issue for us as we will have to make this change for each additional repo a machine is subscribed to.

I know this appears to be hard coded in Katello as a default, but I wonder if there is any way we could set a new global metadata expiration timeout value

10

Yeah, I think it’s a totally fair RFE for Katello to be able to adjust this value, but at the same time I’m afraid it won’t be a high priority one that will be solved soon :frowning:

I’m glad you have at least some kind of workaround for now, but am still very curious what’s happening in your environment that slows things down so much.

11

Just to compare some random numbers. I have a Katello (well, Satellite, but that doesn’t matter) system sitting on the US East Coast, and a VM on my laptop in Germany subscribed to it.

If the dnf cache is completely empty (after a dnf clean all and friends), it takes a moment to fetch the whole shebang (the network in the direction US→EU is not particularly quick):

Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)        782 kB/s |  51 MB     01:06    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)     743 kB/s |  45 MB     01:02

But once the data is there, the “is my metadata still OK” happens rather quickly:

Red Hat Enterprise Linux 8 for x86_64 - BaseOS (RPMs)       9.7 kB/s | 4.1 kB     00:00    
Red Hat Enterprise Linux 8 for x86_64 - AppStream (RPMs)     11 kB/s | 4.5 kB     00:00

In your paste above, it’s way slower and then multiplied by the fact that you have a lot more repositories enabled.

Am I assuming correctly, that in your setup, it also gets OK-ish speeds (on average) when fetching the full metadata after a dnf clean all?

1 Like
12

Why would it not be a high priority RFE?
This concerns the basic, fundamental performance of Katello’s basic & fundamental functionality.
And the current performance can be aggravatingly slow and poor.

Do the Katello developers not give the slightest damn about the performance of their software, or what the actual user experience is, for people that actually use their product?

13

It’s open source. You are more than welcome to create a pull request to fix whatever problems you might have instead of requesting people to fix your problems and if they don’t complaining that they don’t give a d*** about your problems and don’t make your problems high priority.

Performance is fine for me and considering the complete lack of any necessary information in your post and that you are basically hijacking a thread which is almost a year old, I think the tone in your post is a little bit off and not appropriate.

Otherwise I would suggest you open a new thread and reconsider what to “demand”…

14

It’s a Red Hat product using open source as an excuse to exploit unpaid slave labour for product development whilst ignoring customer needs and insulting them for daring to expect the tiniest sliver of professionalism (as you have just done), in Red Hat’s tried and trusted tradition.

15

Whatever. I get a very useful tool for free. That’s good enough for me. People ranting in community forums like you do will just go into my ignore list. Sorry. Bye.

16

Expecting an RFE for a small, easy, and significant performance improvement, to be taken seriously, isn’t a “demand”. It’s like, normal user feedback which developers should actually appreciate because if a user takes the time to report an issue, this gives the vendor/developer a chance to improve their product.

17

@K2049 would you mind making an RFE here with your ideas? Foreman

Looks like 8 years ago we set this explicitly to always be expired: Bug #10495: Custom repositories do not define 'metadata_expire = 1' - Katello - Foreman

It would be very simple to create a new setting (assuming that’s the wish), in fact even someone without any Ruby experience could probably figure it out quickly by just grepping our code for the use of other settings. Could be a good foot in the door for anyone wanting to contribute!

We’d want the default to stay the same.

Now the thing to consider is, why was it hardcoded to 1? That wasn’t an arbitrary decision. I’m going to guess it was to err on the side of ensuring that everything on clients is up to date in case of something like an incremental update (though I think that was introduced before incremental update was even a feature).

I think it should be safe to introduce a setting, but it’s always worth considering first.

Edit: I didn’t realize this thread was so old, I’ll check if there is already an RFE.

Edit edit: I’m guessing it was set to 1 to match Candlepin’s default for RH repos, from that I think it should be safe to make configurable for custom repos.

18

If anyone wants to take a crack at it, here’s the redmine: Feature #36352: [RFE] Add setting to update metadata_expire for custom repositories - Katello - Foreman

19

Looks like it’s more complicated than just making a global setting because existing repositories will need to be refreshed. So, seems like a rake script to set the manifest_expire for all existing custom repos will have to do the trick.

21

Fixes #36352 - Add metadata_expire attribute for custom yum repos by parthaa · Pull Request #10544 · Katello/katello · GitHub if some one wants to labor their way testing it :slight_smile:

1 Like