Add Amazon Linux 2 repository into Forman katello

Hello team,

I’m trying to create “Amazon Linux 2” product into my Forman katello serve and add “Amazon Linux 2” repositories. I create a yum repository as below :

[root@forman~]# hammer repository info --id 23
Id:                 23
Name:               Amazon Linux 2 core
Label:              name_Amazon_Linux_2_core_repository
Description:
Organization:       My_Organization
Red Hat Repository: no
Content Type:       yum
Mirror on Sync:     yes
Url:                http://amazonlinux.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
Publish Via HTTP:   yes
Published At:       https://localhost/pulp/content/My_Organization/Library/custom/AMAZON-LINUX-2/name_Amazon_Linux_2_core_repository/
Relative Path:      My_Organization/Library/custom/AMAZON-LINUX-2/name_Amazon_Linux_2_core_repository
Download Policy:    immediate
HTTP Proxy:
    Id:                1
    Name:              myproxy.mydomain.net
    HTTP Proxy Policy: use_selected_http_proxy
Product:
    Id:   5
    Name: AMAZON-LINUX-2
GPG Key:
    Id:   6
    Name: GPG-KEY-AMAZON-LINUX-2
Sync:
    Status:         Warning
    Last Sync Date: about 17 hours
Created:            2021/05/08 18:01:01
Updated:            2021/05/08 18:10:42
Content Counts:
    Packages:       0
    Source RPMS:    0
    Package Groups: 0
    Errata:         0
    Module Streams: 0

But I’m not able to sync the repository and I get error below :
403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/24c8c7b64056e3963a158cefd9b301c94443194a8569684056656260cd49c8f4/java-11-amazon-corretto-headless-11.0.7+10-1.amzn2.x86_64.rpm')

Did you guys tried to add “Amazon Linux 2” repositories into Forman katello install?

Info about Forman/Katello version :

Regards,

If I try to access that URL I also get a 403. Maybe you have to set up authentication or use a certificate to access those RPMs?

I am not sure about authentication, but one customer explained to me it requires an Amazon Linux to get access. So what we did was using reposync to get a mirror and as reposync does not sync all metadata use modifyrepo_c to add the additional metadata from the systems yum cache. This local repository was then synced to Katello.

I have saved me a copy of the scripts a everything was straight forward except from the authentication which the customer already prepared.

Hello @Dirk,

Thanks for your answer, can you share steps that you used to make a repo mirror from AWS repository and local repository synced to Katello.

Can you also clarify how did you managed to install subscription-manager on “Amazon Linux 2” and register it into your Forman server

Regards,

This was one technical step in a pre-sales date, so I am not sure if I remember everything.

The system was already configured to get updates from Amazon so yum configuration already existed and was working fine. reposync allows to repository locally based on the yum configuration, we added the options --downloadcomps --download-metadata --newest-only to reduce size but get all metadata. We then recognized that errata metadata are missing as reposync did not seem to handle them, but they were stored to /var/cache/yum and with modifyrepo_c allowing to add metadata we took them from the cache and added them to the local repository. Not sure if a createrepo_c was required first, but I think the initial metadata were provided by reposync.

With this we got the repository to Katello and then the customer wanted to test deployment and subscription-manager, as he reported no error I think it worked without any hassle, but I do not know.

Hope this helps at least a bit. If you get it to work, feel free to write a short tutorial. If not, I will try to help but I have no access to Amazon Linux at the moment (and no time) to work through it completely.

Hello @Dirk,

Hope you are doing well,

I’m still not able to add Amazon Linux 2 repositories into Katello even with local mirroir using reposync

My Forman/Katello server is an AWS EC2 server runing RHEL7.

I’m wondering if you have some documentation that I can follow to put this in place.

Thanks for your help.

Regards,

Do you have a server which is running Amazon Linux 2 and has access to the repositories? The repo file should contain all information necessary to access the repository, unless they use a special plugin or limit access by IP address…

Hello @gvde,

On my test server running Amazon Linux 2 I have repositories below :

[root@amazon-linux ~]# yum repolist all
Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
repo id                                                                                                 repo name                                                                                                              status
amzn2-core/2/x86_64                                                                                     Amazon Linux 2 core repository                                                                                         enabled: 24,416
amzn2-core-debuginfo/2/x86_64                                                                           Amazon Linux 2 core repository - debuginfo packages                                                                    disabled
amzn2-core-source/2                                                                                     Amazon Linux 2 core repository - source packages                                                                       disabled
amzn2extra-docker/2/x86_64                                                                              Amazon Extras repo for docker                                                                                          enabled:     41
amzn2extra-docker-debuginfo/2/x86_64                                                                    Amazon Extras debuginfo repo for docker                                                                                disabled
amzn2extra-docker-source/2                                                                              Amazon Extras source repo for docker                                                                                   disabled
repolist: 24,457
[root@amazon-linux ~]# 
 

[root@amazon-linux ~]# cat /etc/yum.repos.d/amzn2-core.repo 
[amzn2-core]
name=Amazon Linux 2 core repository
mirrorlist=$awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/$product/$target/$basearch/mirror.list
priority=10
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
enabled=1
metadata_expire=300
mirrorlist_expire=300
report_instanceid=yes

[amzn2-core-source]
name=Amazon Linux 2 core repository - source packages
mirrorlist=$awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/$product/$target/SRPMS/mirror.list
priority=10
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
enabled=0
metadata_expire=300
mirrorlist_expire=300
report_instanceid=yes

[amzn2-core-debuginfo]
name=Amazon Linux 2 core repository - debuginfo packages
mirrorlist=$awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/$product/$target/debuginfo/$basearch/mirror.list
priority=10
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
enabled=0
metadata_expire=300
mirrorlist_expire=300
report_instanceid=yes

[root@amazon-linux ~]# cat /etc/yum.repos.d/amzn2-extras.repo 

[amzn2extra-docker-source]
enabled = 0
name = Amazon Extras source repo for docker
mirrorlist = $awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/extras/docker/latest/SRPMS/mirror.list
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
priority = 10
skip_if_unavailable = 1
report_instanceid = yes

[amzn2extra-docker-debuginfo]
enabled = 0
name = Amazon Extras debuginfo repo for docker
mirrorlist = $awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/extras/docker/latest/debuginfo/$basearch/mirror.list
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
priority = 10
skip_if_unavailable = 1
report_instanceid = yes

[amzn2extra-docker]
enabled = 1
name = Amazon Extras repo for docker
mirrorlist = $awsproto://$amazonlinux.$awsregion.$awsdomain/$releasever/extras/docker/latest/$basearch/mirror.list
gpgcheck = 1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2
priority = 10
skip_if_unavailable = 1
report_instanceid = yes

[root@amazon-linux ~]# 

So I tryed to setup a yum repository on my Foreman/Katello server as below :

But I’m not able to sync the repository and I get error below :
403, message='Forbidden', url=URL('http://amazonlinux.us-east-1.amazonaws.com/blobstore/24c8c7b64056e3963a158cefd9b301c94443194a8569684056656260cd49c8f4/java-11-amazon-corretto-headless-11.0.7+10-1.amzn2.x86_64.rpm')


My Forman/Katello server is an AWS EC2 server runing RHEL7, it’s not a proxy issue (wide open rules for testing)

Regards,

You can run “yum repolist -v amzn2-core” to see the actual URL used.

The report_instanceid in the repo configuration seems to be an Amazon extension. I looks to me as if they want to make sure that it’s only accessible on servers in their cloud running AL2.

Can you access the URL on the AL2 server?

Hello @gvde ,

Below output of yum repolist -v amzn2-core

[root@amazon-linux ~]# yum repolist -v amzn2-core
Loading "extras_suggestions" plugin
Loading "langpacks" plugin
Loading "priorities" plugin
Loading "update-motd" plugin
Adding en_US to language list
Config time: 0.308
Yum version: 3.4.3
Setting up Package Sacks
pkgsack time: 0.006
Repo-id      : amzn2-core/2/x86_64
Repo-name    : Amazon Linux 2 core repository
Repo-status  : enabled
Repo-revision: 1620071695
Repo-updated : Mon May  3 19:54:55 2021
Repo-pkgs    : 24,416
Repo-size    : 29 G
Repo-mirrors : https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
Repo-baseurl : https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/2/core/2.0/x86_64/6fca8b8886cbbd8e64d2e7174a8a9d18b949eb2253cc99bc585916a38ca66127/
Repo-expire  : 300 second(s) (last: Thu May 13 14:29:32 2021)
  Filter     : read-only:present
Repo-filename: /etc/yum.repos.d/amzn2-core.repo

I’m not able to get the RPM with wget from the Amazon Linux box:

[root@amazon-linux ~]# wget http://amazonlinux.us-east-1.amazonaws.com/blobstore/24c8c7b64056e3963a158cefd9b301c94443194a8569684056656260cd49c8f4/java-11-amazon-corretto-headless-11.0.7+10-1.amzn2.x86_64.rpm
--2021-05-13 16:42:00--  http://amazonlinux.us-east-1.amazonaws.com/blobstore/24c8c7b64056e3963a158cefd9b301c94443194a8569684056656260cd49c8f4/java-11-amazon-corretto-headless-11.0.7+10-1.amzn2.x86_64.rpm
Resolving amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)... 52.217.64.102, 2600:1fa0:80b4:da88:34d9:6cb6::
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.217.64.102|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2021-05-13 16:42:00 ERROR 403: Forbidden.

But I can get mirror.list file from the same box

[root@amazon-linux ~]# wget https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
--2021-05-13 16:42:15--  https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/2/core/latest/x86_64/mirror.list
Resolving amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)... 52.217.99.88
Connecting to amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com (amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com)|52.217.99.88|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 148 [binary/octet-stream]
Saving to: ‘mirror.list’

100%[========================================================>] 148         --.-K/s   in 0s      
2021-05-13 16:42:15 (7.46 MB/s) - ‘mirror.list’ saved [148/148]

Here the content of mirror.list file :

[root@amazon-linux ~]# cat mirror.list 
https://amazonlinux-2-repos-us-east-1.s3.us-east-1.amazonaws.com/2/core/2.0/x86_64/6fca8b8886cbbd8e64d2e7174a8a9d18b949eb2253cc99bc585916a38ca66127

Hi @laimsi

The url works if I use the encoded “+” character, check this:

wget http://amazonlinux.us-east-1.amazonaws.com/blobstore/24c8c7b64056e3963a158cefd9b301c94443194a8569684056656260cd49c8f4/java-11-amazon-corretto-headless-11.0.7%2B10-1.amzn2.x86_64.rpm
Connecting to amazonlinux.us-east-1.amazonaws.com (amazonlinux.us-east-1.amazonaws.com)|52.216.111.85|:80… connected.
HTTP request sent, awaiting response… 200 OK <<<<<<<<<<<<

It seems to be related in how the URL is being requested. it’s also mentioned in the AWS S3 docs under " Characters that might require special handling" -> Creating object key names - Amazon Simple Storage Service

I hope that helps, have a good one!

1 Like

Hello @gmc,

indeed replacing “+” with “%2b” using wget download the rpm.

So how can katello manage that or how can we manage that from forman/katello server? is that somthing that should be improved from dev side?

Regards,

I guess you should open an issue at Issues - Katello - Foreman

Hello!, is there any update on this?

Yes. This was fixed in 3.14.1 but then regressed again in (I think) 3.14.3. I’m working on fixing it for the next release.

https://pulp.plan.io/issues/9329