Dnf (yum) suddenly stopped working on RockyLinux 8 machines

Support
1

I recently published a new content view for our RockyLinux 8 machines, after that, dnf updates suddenly started failing on all RL8 machines with this error:

Module yaml error: Document type encountered twice. [line 84 col 1]
terminate called after throwing an instance of ‘libdnf::ModulePackageContainer::ResolveException’
what(): Failed to update from string: Unexpected YAML event in document stream [line 84 col 11]
Aborted

I checked system logs and dnf logs but couldn’t find any clue about what is causing this, the only thing I can see is that this happens right after reading the repomd.xml files from the Foreman server. To make it even more strange, this only happens on the main server, RL8 machines connected to other smart proxies work fine. Any help/suggestions would greatly be appreciated …

1 Like
2

Found it using ‘strace dnf update’

…
openat(AT_FDCWD, “/var/cache/dnf/-28c2b6e31df4a949/repodata/97b1fd5831c3126f365967e4f7e45cb2e1464f984e4902a8a5f17c5553501cab-modules.yaml”, O_RDONLY) = 9
fstat(9, {st_mode=S_IFREG|0644, st_size=2691, …}) = 0
fstat(9, {st_mode=S_IFREG|0644, st_size=2691, …}) = 0
lseek(9, 0, SEEK_SET) = 0
read(9, “—\ndata:\n api:\n rpms:\n -”…, 2691) = 2691
lseek(9, 2691, SEEK_SET) = 2691
close(9) = 0
write(2, "Module yaml error: ", 19Module yaml error: ) = 19
write(2, “Document type encountered twice.”…, 48Document type encountered twice. [line 84 col 1]) = 48
write(2, “\n”, 1
) = 1

So something wrong with the repo / repo metadata - going to remove the repo from the content view and try again. After that maybe delete and re-create the repo. Hope that fixes things …

1 Like
3

The error follows reading the dnf cache. Maybe the cache is corrupt? Did you try a dnf clean all?

1 Like
4

Hi @Polle ,

What Katello and Foreman versions are you running? Also mind showing pulp status or rpm -qa | grep pulp ?

5

Well, I think this is a bug - just to answer your questions first, I'm using 'the latest and the greatest' version of Foreman / Katello 3.6.1 - 4.8.0 and yes I did clean the dnf cache. I found this issue occurred for one specific repo - Mercurial (https://www.mercurial-scm.org/release/centos8/ ) and the problem occurs with the modules.yaml file. I'm not that familiar with creating repos / module streams but that file is a part of it: https://www.mercurial-scm.org/release/centos8/modules.yaml Now, if you look at that file it starts with: 

data:
  module: mercurialstandalone
  profiles:
    default:
    - everything
  stream: default
document: modulemd-defaults
version: 1
---
data:
  api:
    rpms:
...

and if you run it through a parser, syntax is just fine. 2 documents separated by a triple dash. Now what I see in the file from the dnf cache is: 

---
data:
  api:
    rpms:
    - mercurial
    - tortoisehg
....
document: modulemd
version: 2
data:
  module: mercurialstandalone
  profiles:
    default:
    - everything
  stream: default
document: modulemd-defaults
version: 1

So the second document declaration has been moved to the end but not the separator, so now there are 2 document declarations in the same block which results in an error thrown by dnf. Since the source is correct, I assume that the yaml is modified during the repo import (pulp maybe?), no idea why/how but the result is clearly wrong ...

1 Like
6

And since the modules is something that was introduced in EL8, that explains why I don’t see this problem in my CentOS 7 clients …

1 Like
7

Are you able to confirm that the module yaml in your dnf cache matches what is being hosted by Katello?

Also @dralley do you know of any fixed module metadata issue like this? I recall talking to you a while ago about a modular metadata issue but I’m not sure if it relates to this…

Katello 4.8 is on python39-pulp-rpm-3.19.2

8

Well, afaik it is pretty hard to find back that file in /var/lib/pulp
I did a quick check and took a look at the cached modules.yaml for the PowerTools repo and I see that it starts with — (so not only between the docs, also at the start) and ends with …
Looks to me the pulp is parsing and changing it - maybe requires a very strict syntax (from what I read, only a separator — between the docs is valid syntax though)

9

… and looking at products-repos, I see that Foreman reports 10 module streams for PowerTools and only 1 for the Mercurial repo, that should be 2 …

10

So it looks like the RPM module parsing code assumes that each module starts with —, because using the official tooling it does. But it would seem that it’s not universally the case.

So what happens is when it gets pieced back together you sometimes get a module that doesn’t start with — dropped underneath another one (as opposed to the very beginning of the document) which no longer gets parsed as 2 different documents.

11
1 Like
12

Wow - that was a quick response …
I guess that this fix will need to be back-ported to Foreman/Katello before we can ‘enjoy’ it?

1 Like
13

Hmm - I applied that fix to /usr//lib/python3.9/site-packages/pulp_rpm/app/modulemd.py
Ran a full content sync and published a new version but the problem remains

1 Like
14

For the record (after discussing with the other of that repo), it turned out that pyyaml has 2 parameters: explicit_start=True and explicit_end=True
So he’s now re-generating and uploading with those set - that might (most likely will) make the difference

1 Like
15

Those 2 flags made the difference …
… which doesn’t mean that pulp should be made a little more robust to handle the different flavors of syntactically correct module.yaml files …

1 Like