Unable to create content - suspect cert related

Support
1

Problem:
using the webGUI or hammerCLI i am unable to create rocky 8 content,

Expected outcome:
creation of new product and content
Foreman and Proxy versions:
foreman-3.11.0-1.el8.noarch
foreman-proxy-3.11.0-1.el8.noarch
katello-4.13.0-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:
NAME=“Red Hat Enterprise Linux”
VERSION=“8.7 (Ootpa)”

Other relevant data:

I suspect my issue is some how tied to certificates as i’m running my foreman katello instances with FreeIPA, its a clean installation with the following install command.

sudo foreman-installer \
--scenario katello \
--foreman-initial-organization "Demo Networks" \
--foreman-initial-location "Demoverse" \
--foreman-server-ssl-cert "/etc/pki/tls/certs/foreman.crt" \
--foreman-server-ssl-key "/etc/pki/tls/private/foreman.key" \
--foreman-server-ssl-ca "/etc/pki/tls/certs/idx.bundle.pem" \
--foreman-server-ssl-chain "/etc/pki/tls/certs/idx.bundle.pem" \
--foreman-ipa-authentication=true \
--enable-foreman-plugin-ansible \
--enable-foreman-proxy-plugin-ansible --enable-foreman-cli-ansible

when running the following hammer command with debug turned on

--organization-id 1 \ 
--product "Rocky 8 Linux for x86_64" \
--name "Rocky 8 Base RPMS" \
--label "Rocky_8_Base_RPMS" \
--content-type "yum" \
--download-policy "on_demand" \
--gpg-key-id 3 \
--url "https://mirror.aarnet.edu.au/pub/rocky/8/BaseOS/x86_64/os/" \
--mirroring-policy 'mirror_complete'````
/var/log/httpd/foreman-ssl_access_ssl.log show the following entry that corresponds to the attempt

I get the output

2024-07-16T16:06:42 [I|app|e20a9926] Completed 500 Internal Server Error in 345ms (Views: 0.1ms | ActiveRecord: 36.4ms | Allocations: 98403)
2024-07-16T16:06:47 [I|app|beba3bb8] Started GET "/notification_recipients" for 192.168.10.5 at 2024-07-16 16:06:47 +1000
2024-07-16T16:06:47 [I|app|beba3bb8] Processing by NotificationRecipientsController#index as JSON
2024-07-16T16:06:47 [I|app|beba3bb8] Completed 200 OK in 3ms (Views: 0.1ms | ActiveRecord: 0.6ms | Allocations: 1604)
...skipping...
               "label" => "Rocky_8_Base_RPMS",
          "product_id" => 3,
        "content_type" => "yum",
                 "url" => "https://mirror.aarnet.edu.au/pub/rocky/8/BaseOS/x86_64/os/",
          "gpg_key_id" => 3,
     "download_policy" => "on_demand",
    "mirroring_policy" => "mirror_complete"
}
[DEBUG 2024-07-16T16:06:42 API] Headers: {}
[DEBUG 2024-07-16T16:06:42 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[ERROR 2024-07-16T16:06:42 API] 500 Internal Server Error
[DEBUG 2024-07-16T16:06:42 API] {
    "displayMessage" => "Connection reset by peer",
            "errors" => [
        [0] "Connection reset by peer"
    ]
}
[DEBUG 2024-07-16T16:06:42 Exception] Using exception handler HammerCLIKatello::ExceptionHandler#handle_internal_error
[ERROR 2024-07-16T16:06:42 Exception] Connection reset by peer
Could not create the repository:
  Connection reset by peer
[ERROR 2024-07-16T16:06:42 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:353:in `block in rest_client_call_block'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:834:in `process_result'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
    /usr/share/ruby/net/http.rb:933:in `start'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:69:in `post'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:185:in `call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/resource.rb:21:in `call'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/share/gems/gems/hammer_cli_foreman-3.11.0/lib/hammer_cli_foreman/commands.rb:180:in `send_request'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:140:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/bin/hammer:142:in `<top (required)>'
    /usr/bin/hammer:23:in `load'

192.168.x.x - - [16/Jul/2024:14:14:46 +1000] “POST /katello/api/repositories HTTP/1.1” 500 83 “-” “rest-client/2.1.0 (linux x86_64) ruby/2.7.8p225”


I'm unsure where to go from here to determine the cause of the internal server error and would appreciate any assistance on how to look into the issue further.
2

Hi,

Is it only this repo or any repos you try to create?

3

Thanks for your reply,

Yes it does appear to be any yum/rpm repo,
I have avoided doing deb repos to keep things simple.

I have since ran

katello-certs-check

and

foreman-installer --scenario katello \
                      --certs-server-cert "/etc/pki/tls/certs/foreman.crt" \
                      --certs-server-key "/etc/pki/tls/private/foreman.key" \
                      --certs-server-ca-cert "/etc/pki/tls/certs/idx.bundle.pem" \
                      --certs-update-server --certs-update-server-ca

and at least running the hammer command is now returning a 500 internal server message error that is indicative of a certificate error, where as before I couldn’t figure out what the cause of the 500 error message was.

[ INFO 2024-07-16T17:28:05 API] POST /katello/api/repositories
[DEBUG 2024-07-16T17:28:05 API] Params: {
                "name" => "Rocky 8 Base RPMS",
               "label" => "Rocky_8_Base_RPMS",
          "product_id" => 3,
        "content_type" => "yum",
                 "url" => "https://mirror.aarnet.edu.au/pub/rocky/8/BaseOS/x86_64/os/",
          "gpg_key_id" => 3,
     "download_policy" => "on_demand",
    "mirroring_policy" => "mirror_complete"
}
[DEBUG 2024-07-16T17:28:05 API] Headers: {}
[DEBUG 2024-07-16T17:28:05 API] Using authenticator: HammerCLIForeman::Api::InteractiveBasicAuth
[ERROR 2024-07-16T17:28:06 API] 500 Internal Server Error
[DEBUG 2024-07-16T17:28:06 API] {
    "displayMessage" => "SSL_read: tlsv1 alert unknown ca",
            "errors" => [
        [0] "SSL_read: tlsv1 alert unknown ca"
    ]
}
[DEBUG 2024-07-16T17:28:06 Exception] Using exception handler HammerCLIKatello::ExceptionHandler#handle_internal_error
[ERROR 2024-07-16T17:28:06 Exception] SSL_read: tlsv1 alert unknown ca
Could not create the repository:
  SSL_read: tlsv1 alert unknown ca
[ERROR 2024-07-16T17:28:06 Exception]

RestClient::InternalServerError (500 Internal Server Error):
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:249:in `exception_with_response'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/abstract_response.rb:129:in `return!'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:353:in `block in rest_client_call_block'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:834:in `process_result'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:743:in `block in transmit'
    /usr/share/ruby/net/http.rb:933:in `start'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:727:in `transmit'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:163:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/request.rb:63:in `execute'
    /usr/share/gems/gems/rest-client-2.1.0/lib/restclient/resource.rb:69:in `post'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/api.rb:185:in `call'
    /usr/share/gems/gems/apipie-bindings-0.6.0/lib/apipie_bindings/resource.rb:21:in `call'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/share/gems/gems/hammer_cli_foreman-3.11.0/lib/hammer_cli_foreman/commands.rb:180:in `send_request'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/subcommand/execution.rb:18:in `execute'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:66:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/lib/hammer_cli/abstract.rb:103:in `run'
    /usr/share/gems/gems/clamp-1.3.2/lib/clamp/command.rb:140:in `run'
    /usr/share/gems/gems/hammer_cli-3.11.0/bin/hammer:142:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'
4

Built a second host without the IPA integration,
so using the katello certificates.

I’m going to go over the documentation on custom ssl certificates and try and figure out what i could be missing.

5

I have mostly managed to solve this issue by regenerating the SSL certificates using the following

sudo ipa-getcert request \
-f /etc/pki/tls/certs/foreman.crt \
-k /etc/pki/tls/private/foreman.key \
-F /etc/pki/tls/certs/idx.bundle.pem \
-K HTTP/dem0.example.com \
-D dem0.example.com \
-u digitalSignature \
-u nonRepudiation \
-u keyEncipherment \
-u dataEncipherment \
-U id-kp-serverAuth \
-U id-kp-clientAuth \
-U id-kp-codeSigning \
-U id-kp-emailProtection

I am still unable to create content due to the error message

Could not create the repository:
  SSL_read: tlsv1 alert unknown ca

I am able to work around this on one of my dev builds by modifying /etc/httpd/conf.d/05-foreman-ssl.conf so that SSLVerifyClient optional is SSLVerifyClient optional_no_ca.

I’d like to resolve this without having to resort to using optional_no_ca in the httpd config,
any advice on what i may be missing in my custom certificates would be greatly appreciated.