What setting are required within foreman to use ansible callback

[jon-nfc]

Problem: When using the ansible callback to report to foreman I’m receiving HTTP/403 for the client and the server reports No SSL cert with CN supplied or malformed packet The following issues are also identified

• Turning off setting Restrict registered smart proxies causes there to be no requirement for authentication to upload to the reports endpoint. Even when headers SSL_CLIENT_VERIFY are set including SSL_CLIENT_VERIFY=FAILURE:reason

• removing header SSL_CLIENT_CERT and only having SSL_CLIENT_S_DN and SSL_CLIENT_VERIFY has no effect, even when SSL_CLIENT_VERIFY=SUCCESS

Scouring DR Google and the forums have yielded nothing of value. Any assistance would be greatly appreciated. Bottom Line trying in integrate AWX job runs to be reported back to foreman.

Expected outcome: can use ansible callback (authenticated) to upload reports to foreman

Foreman and Proxy versions: 3.9.1

Foreman and Proxy plugin versions:

Distribution and version: Custom. Alpine linux container running on kubernetes with nginx ingress controller.

Other relevant data:

ansible.cfg

[defaults]

callback_whitelist = foreman

callbacks_enabled = foreman

[callback_foreman]

client_cert = /home/sysadmin/git/scratchpad/Certificate-Authority-Testing/certs/client2.crt

client_key = /home/sysadmin/git/scratchpad/Certificate-Authority-Testing/certs/client2.pem

url = https://< truncated >.com

report_type = foreman

verify_certs = 0

Foreman log

2024-03-09T07:44:50 [I|app|b41717a8] Started GET "/notification_recipients" for < truncated > at 2024-03-09 07:44:50 +0000

2024-03-09T07:44:50 [I|app|b41717a8] Processing by NotificationRecipientsController#index as JSON

2024-03-09T07:44:50 [D|tax|b41717a8] Current location set to none

2024-03-09T07:44:50 [D|tax|b41717a8] Current organization set to none

2024-03-09T07:44:51 [D|not|b41717a8] Cache Hit: notification, reading cache for notification-4

2024-03-09T07:44:51 [D|app|b41717a8] Body: {"notifications":[]}

2024-03-09T07:44:51 [I|app|b41717a8] Completed 200 OK in 516ms (Views: 0.3ms | ActiveRecord: 1.5ms | Allocations: 1445)

2024-03-09T07:44:53 [I|app|002eaf1b] Started POST "/api/v2/hosts/facts" for < truncated > at 2024-03-09 07:44:53 +0000

2024-03-09T07:44:53 [I|app|002eaf1b] Processing by Api::V2::HostsController#facts as JSON

2024-03-09T07:44:53 [I|app|002eaf1b] Parameters: {"name"=>"laptop2", "facts"=>"[FILTERED]", "apiv"=>"v2", "host"=>{"name"=>"laptop2"}}

2024-03-09T07:44:53 [D|app|002eaf1b] Importer DiscoveryFactImporter does not implement authorized_smart_proxy_features.

2024-03-09T07:44:53 [D|app|002eaf1b] Importer Katello::RhsmFactImporter does not implement authorized_smart_proxy_features.

2024-03-09T07:44:53 [W|app|002eaf1b] No SSL cert with CN supplied - request from < truncated >

2024-03-09T07:44:53 [D|app|002eaf1b] Rendering layout api/v2/layouts/error_layout.json.erb

2024-03-09T07:44:53 [D|app|002eaf1b] Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout

2024-03-09T07:44:53 [I|app|002eaf1b] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 2.2ms | Allocations: 638)

2024-03-09T07:44:53 [I|app|002eaf1b] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 3.3ms | Allocations: 897)

2024-03-09T07:44:53 [I|app|002eaf1b] Filter chain halted as #<Proc:0x00007fae171ba108 /home/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected

2024-03-09T07:44:53 [I|app|002eaf1b] Completed 403 Forbidden in 37ms (Views: 8.7ms | ActiveRecord: 13.6ms | Allocations: 3586)

2024-03-09T07:44:56 [I|app|a0101f62] Started POST "/api/v2/config_reports" for < truncated > at 2024-03-09 07:44:56 +0000

2024-03-09T07:44:56 [I|app|a0101f62] Processing by Api::V2::ConfigReportsController#create as JSON

2024-03-09T07:44:56 [I|app|a0101f62] Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}

2024-03-09T07:44:56 [W|app|a0101f62] No SSL cert with CN supplied - request from < truncated >

2024-03-09T07:44:56 [D|app|a0101f62] Rendering layout api/v2/layouts/error_layout.json.erb

2024-03-09T07:44:56 [D|app|a0101f62] Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout

2024-03-09T07:44:56 [I|app|a0101f62] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.7ms | Allocations: 278)

2024-03-09T07:44:56 [I|app|a0101f62] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.1ms | Allocations: 396)

2024-03-09T07:44:56 [I|app|a0101f62] Filter chain halted as #<Proc:0x00007fae1057b410 /home/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected

2024-03-09T07:44:56 [I|app|a0101f62] Completed 403 Forbidden in 8ms (Views: 2.5ms | ActiveRecord: 0.5ms | Allocations: 3375)

Headers from behind the NginX Ingress

when setting proxy_set_header SSL_CLIENT_CERT $ssl_client_raw_cert the following headers are set

ssl-client-verify: SUCCESS

ssl-client-subject-dn: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

ssl-client-issuer-dn: emailAddress=no@reply.com,CN=Testing ONLY CA,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

X-Request-ID: 92bf4c390c528d4a1c478316c714d778

X-Real-IP: < truncated >

X-Forwarded-For: < truncated >

X-Forwarded-Host: debug.local

X-Forwarded-Port: 443

X-Forwarded-Proto: https

X-Forwarded-Scheme: https

X-Scheme: https

SSL_CLIENT_S_DN: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

SSL_CLIENT_VERIFY: SUCCESS

SSL_CLIENT_CERT: -----BEGIN CERTIFICATE-----

MIIGBTCCA+2gAwIBAgIUHctJCt3jhSeOtfoOniqVOSDxeUQwDQYJKoZIhvcNAQEL

BQAwfDELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM

GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZ

IENBMRswGQYJKoZIhvcNAQkBFgxub0ByZXBseS5jb20wHhcNMjQwMzA3MTU1NDIx

WhcNMjUwMzE3MTU1NDIxWjBXMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1T

dGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRAwDgYDVQQD

DAdjbGllbnQyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7RIC+Sjw

O4Y95uEYcAEHdxbue6mer/KC6+PcA8Sr7zI3BnrxihAH9XDf2lX1+BLVSpeyalJp

uSPfm3w9U20fbgj6obFZD56xhT7MC915Bx6xqC4ba81ZltcXcRuaf1JCOj9r4KUa

boz6flv5UC9mkr2xnUtTOUGXy/qFopSsUA5YTltoQDcfqF7bF1qUi8tpO8yqsXLP

xcT1kMBswED+SIY1r1B4yqeigaZxav2qvzj2M+EZT/fIE6Ui0WCMGWJUgier4XGe

FWm6dyKeKk0Dp0jbJ53+6PS5ejHOJG0oqZ+89EqMYmEmFXuy/YzbYJ9BOjdQpon6

giuAM58fI9ImkQIDAQABo4IBojCCAZ4wDwYDVR0TAQH/BAUwAwIBADARBglghkgB

hvhCAQEEBAMCB4AwHQYDVR0OBBYEFDncLvrw33MLZp8PF0ZAOSfv1F0uMIG6BgNV

HSMEgbIwga+AFJkP9SMkLIe91IE6UWAt3G/2xNFCoYGApH4wfDELMAkGA1UEBhMC

QVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdp

dHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZIENBMRswGQYJKoZIhvcN

AQkBFgxub0ByZXBseS5jb22CFDX9/iZqCgLU2XtnjSZn1+f4TU5jMHQGCCsGAQUF

BwEBBGgwZjA8BggrBgEFBQcwAoYwaHR0cHM6Ly9leGFtcGxlLm15dXJsL2NhLmNy

dC9jZXJ0aWZpY2F0ZXMvY2EuY3J0MCYGCCsGAQUFBzABhhpodHRwczovL2V4YW1w

bGUubXl1cmwvb2NzcDAOBgNVHQ8BAf8EBAMCA6gwFgYDVR0lAQH/BAwwCgYIKwYB

BQUHAwIwDQYJKoZIhvcNAQELBQADggIBALhNoR9xpmMfEepzmQdYdiMiRbxEHEZw

PU5k/Pir1pMnVtino0TfCzX/+k7B/F5IYHy+rwRV23reePMLx+gJmcTc1TpmgeNN

qVBEwXT6NLo8e/J3OKbghuLt6L4adgdkDSsEyCutuxDCzLovm+5SVnEd4qLQcSDR

BbxPU+1OTx9wCHBYCWMGIsJqVWNqILu3bVZa6vJgrypV579Q05pF2vYvnXdvCMsy

FVJ2DNCkCxhhIPaikUcWcbIXYZ5cFiQW4XA+XJGItq85p6yKRdPaLUHvMRvCQhZo

ErdnxMdu2HnsEFWgnx/zjilBlOmZOg8V0DCL3P9Dztn4yHxDk2DhPbuJO0dpHVnC

hYpvy+JhtHecJ0IDLUu4EQkpv9l32Tg2a84UuVgxO3l+DIkwdoLHt9B/59000ea0

MgAMzx6ZmGcKWJMn4Qjuz0rveyg/8B5IFIOzhOmawi9tx6oqCvMd02nr5bwFJHDj

H5vLnE5ewnR6txAUWpGvENFjiXb5OLU4r/Dyqt9RJcS5C6nqtNvMjFtOcJcs1sbj

AKDVeufJm9QP2z5NhSdvtCEszwF6YCILot18E3XvTa6U35BpQ0QoIq2/UUc/453U

XLSQOjRg73o4nzzGFKOc4IA4By+SZURHSJF0W2jptgUTx+fNPbl0JOIM692zanfn

E3KNAAiW87El

ansible.cfg

``` ini

[defaults]

callback_whitelist = foreman

callbacks_enabled = foreman

[callback_foreman]

client_cert = /home/sysadmin/git/scratchpad/Certificate-Authority-Testing/certs/client2.crt

client_key = /home/sysadmin/git/scratchpad/Certificate-Authority-Testing/certs/client2.pem

url = https://< truncated >.com

report_type = foreman

verify_certs = 0

Foreman log

2024-03-09T07:44:50 [I|app|b41717a8] Started GET "/notification_recipients" for < truncated > at 2024-03-09 07:44:50 +0000

2024-03-09T07:44:50 [I|app|b41717a8] Processing by NotificationRecipientsController#index as JSON

2024-03-09T07:44:50 [D|tax|b41717a8] Current location set to none

2024-03-09T07:44:50 [D|tax|b41717a8] Current organization set to none

2024-03-09T07:44:51 [D|not|b41717a8] Cache Hit: notification, reading cache for notification-4

2024-03-09T07:44:51 [D|app|b41717a8] Body: {"notifications":[]}

2024-03-09T07:44:51 [I|app|b41717a8] Completed 200 OK in 516ms (Views: 0.3ms | ActiveRecord: 1.5ms | Allocations: 1445)

2024-03-09T07:44:53 [I|app|002eaf1b] Started POST "/api/v2/hosts/facts" for < truncated > at 2024-03-09 07:44:53 +0000

2024-03-09T07:44:53 [I|app|002eaf1b] Processing by Api::V2::HostsController#facts as JSON

2024-03-09T07:44:53 [I|app|002eaf1b] Parameters: {"name"=>"laptop2", "facts"=>"[FILTERED]", "apiv"=>"v2", "host"=>{"name"=>"laptop2"}}

2024-03-09T07:44:53 [D|app|002eaf1b] Importer DiscoveryFactImporter does not implement authorized_smart_proxy_features.

2024-03-09T07:44:53 [D|app|002eaf1b] Importer Katello::RhsmFactImporter does not implement authorized_smart_proxy_features.

2024-03-09T07:44:53 [W|app|002eaf1b] No SSL cert with CN supplied - request from < truncated >

2024-03-09T07:44:53 [D|app|002eaf1b] Rendering layout api/v2/layouts/error_layout.json.erb

2024-03-09T07:44:53 [D|app|002eaf1b] Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout

2024-03-09T07:44:53 [I|app|002eaf1b] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 2.2ms | Allocations: 638)

2024-03-09T07:44:53 [I|app|002eaf1b] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 3.3ms | Allocations: 897)

2024-03-09T07:44:53 [I|app|002eaf1b] Filter chain halted as #<Proc:0x00007fae171ba108 /home/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected

2024-03-09T07:44:53 [I|app|002eaf1b] Completed 403 Forbidden in 37ms (Views: 8.7ms | ActiveRecord: 13.6ms | Allocations: 3586)

2024-03-09T07:44:56 [I|app|a0101f62] Started POST "/api/v2/config_reports" for < truncated > at 2024-03-09 07:44:56 +0000

2024-03-09T07:44:56 [I|app|a0101f62] Processing by Api::V2::ConfigReportsController#create as JSON

2024-03-09T07:44:56 [I|app|a0101f62] Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}

2024-03-09T07:44:56 [W|app|a0101f62] No SSL cert with CN supplied - request from < truncated >

2024-03-09T07:44:56 [D|app|a0101f62] Rendering layout api/v2/layouts/error_layout.json.erb

2024-03-09T07:44:56 [D|app|a0101f62] Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout

2024-03-09T07:44:56 [I|app|a0101f62] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.7ms | Allocations: 278)

2024-03-09T07:44:56 [I|app|a0101f62] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.1ms | Allocations: 396)

2024-03-09T07:44:56 [I|app|a0101f62] Filter chain halted as #<Proc:0x00007fae1057b410 /home/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected

2024-03-09T07:44:56 [I|app|a0101f62] Completed 403 Forbidden in 8ms (Views: 2.5ms | ActiveRecord: 0.5ms | Allocations: 3375)

Headers from behind the NginX Ingress

when setting proxy_set_header SSL_CLIENT_CERT $ssl_client_raw_cert the following headers are set

ssl-client-verify: SUCCESS

ssl-client-subject-dn: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

ssl-client-issuer-dn: emailAddress=no@reply.com,CN=Testing ONLY CA,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

X-Request-ID: 92bf4c390c528d4a1c478316c714d778

X-Real-IP: < truncated >

X-Forwarded-For: < truncated >

X-Forwarded-Host: debug.local

X-Forwarded-Port: 443

X-Forwarded-Proto: https

X-Forwarded-Scheme: https

X-Scheme: https

SSL_CLIENT_S_DN: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

SSL_CLIENT_VERIFY: SUCCESS

SSL_CLIENT_CERT: -----BEGIN CERTIFICATE-----

MIIGBTCCA+2gAwIBAgIUHctJCt3jhSeOtfoOniqVOSDxeUQwDQYJKoZIhvcNAQEL

BQAwfDELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM

GEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZ

IENBMRswGQYJKoZIhvcNAQkBFgxub0ByZXBseS5jb20wHhcNMjQwMzA3MTU1NDIx

WhcNMjUwMzE3MTU1NDIxWjBXMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1T

dGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRAwDgYDVQQD

DAdjbGllbnQyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7RIC+Sjw

O4Y95uEYcAEHdxbue6mer/KC6+PcA8Sr7zI3BnrxihAH9XDf2lX1+BLVSpeyalJp

uSPfm3w9U20fbgj6obFZD56xhT7MC915Bx6xqC4ba81ZltcXcRuaf1JCOj9r4KUa

boz6flv5UC9mkr2xnUtTOUGXy/qFopSsUA5YTltoQDcfqF7bF1qUi8tpO8yqsXLP

xcT1kMBswED+SIY1r1B4yqeigaZxav2qvzj2M+EZT/fIE6Ui0WCMGWJUgier4XGe

FWm6dyKeKk0Dp0jbJ53+6PS5ejHOJG0oqZ+89EqMYmEmFXuy/YzbYJ9BOjdQpon6

giuAM58fI9ImkQIDAQABo4IBojCCAZ4wDwYDVR0TAQH/BAUwAwIBADARBglghkgB

hvhCAQEEBAMCB4AwHQYDVR0OBBYEFDncLvrw33MLZp8PF0ZAOSfv1F0uMIG6BgNV

HSMEgbIwga+AFJkP9SMkLIe91IE6UWAt3G/2xNFCoYGApH4wfDELMAkGA1UEBhMC

QVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdp

dHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZIENBMRswGQYJKoZIhvcN

AQkBFgxub0ByZXBseS5jb22CFDX9/iZqCgLU2XtnjSZn1+f4TU5jMHQGCCsGAQUF

BwEBBGgwZjA8BggrBgEFBQcwAoYwaHR0cHM6Ly9leGFtcGxlLm15dXJsL2NhLmNy

dC9jZXJ0aWZpY2F0ZXMvY2EuY3J0MCYGCCsGAQUFBzABhhpodHRwczovL2V4YW1w

bGUubXl1cmwvb2NzcDAOBgNVHQ8BAf8EBAMCA6gwFgYDVR0lAQH/BAwwCgYIKwYB

BQUHAwIwDQYJKoZIhvcNAQELBQADggIBALhNoR9xpmMfEepzmQdYdiMiRbxEHEZw

PU5k/Pir1pMnVtino0TfCzX/+k7B/F5IYHy+rwRV23reePMLx+gJmcTc1TpmgeNN

qVBEwXT6NLo8e/J3OKbghuLt6L4adgdkDSsEyCutuxDCzLovm+5SVnEd4qLQcSDR

BbxPU+1OTx9wCHBYCWMGIsJqVWNqILu3bVZa6vJgrypV579Q05pF2vYvnXdvCMsy

FVJ2DNCkCxhhIPaikUcWcbIXYZ5cFiQW4XA+XJGItq85p6yKRdPaLUHvMRvCQhZo

ErdnxMdu2HnsEFWgnx/zjilBlOmZOg8V0DCL3P9Dztn4yHxDk2DhPbuJO0dpHVnC

hYpvy+JhtHecJ0IDLUu4EQkpv9l32Tg2a84UuVgxO3l+DIkwdoLHt9B/59000ea0

MgAMzx6ZmGcKWJMn4Qjuz0rveyg/8B5IFIOzhOmawi9tx6oqCvMd02nr5bwFJHDj

H5vLnE5ewnR6txAUWpGvENFjiXb5OLU4r/Dyqt9RJcS5C6nqtNvMjFtOcJcs1sbj

AKDVeufJm9QP2z5NhSdvtCEszwF6YCILot18E3XvTa6U35BpQ0QoIq2/UUc/453U

XLSQOjRg73o4nzzGFKOc4IA4By+SZURHSJF0W2jptgUTx+fNPbl0JOIM692zanfn

E3KNAAiW87El

-----END CERTIFICATE-----

when setting proxy_set_header SSL_CLIENT_CERT $ssl_client_escaped_cert the following headers are set

ssl-client-verify: SUCCESS

ssl-client-subject-dn: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

ssl-client-issuer-dn: emailAddress=no@reply.com,CN=Testing ONLY CA,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

X-Request-ID: added607a8103e957745479fed0b3e4d

X-Real-IP: < truncated >

X-Forwarded-For: < truncated >

X-Forwarded-Host: debug.local

X-Forwarded-Port: 443

X-Forwarded-Proto: https

X-Forwarded-Scheme: https

X-Scheme: https

SSL_CLIENT_S_DN: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

SSL_CLIENT_VERIFY: SUCCESS

SSL_CLIENT_CERT: -----BEGIN%20CERTIFICATE-----%0AMIIGBTCCA%2B2gAwIBAgIUHctJCt3jhSeOtfoOniqVOSDxeUQwDQYJKoZIhvcNAQEL%0ABQAwfDELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM%0AGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZ%0AIENBMRswGQYJKoZIhvcNAQkBFgxub0ByZXBseS5jb20wHhcNMjQwMzA3MTU1NDIx%0AWhcNMjUwMzE3MTU1NDIxWjBXMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1T%0AdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRAwDgYDVQQD%0ADAdjbGllbnQyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7RIC%2BSjw%0AO4Y95uEYcAEHdxbue6mer%2FKC6%2BPcA8Sr7zI3BnrxihAH9XDf2lX1%2BBLVSpeyalJp%0AuSPfm3w9U20fbgj6obFZD56xhT7MC915Bx6xqC4ba81ZltcXcRuaf1JCOj9r4KUa%0Aboz6flv5UC9mkr2xnUtTOUGXy%2FqFopSsUA5YTltoQDcfqF7bF1qUi8tpO8yqsXLP%0AxcT1kMBswED%2BSIY1r1B4yqeigaZxav2qvzj2M%2BEZT%2FfIE6Ui0WCMGWJUgier4XGe%0AFWm6dyKeKk0Dp0jbJ53%2B6PS5ejHOJG0oqZ%2B89EqMYmEmFXuy%2FYzbYJ9BOjdQpon6%0AgiuAM58fI9ImkQIDAQABo4IBojCCAZ4wDwYDVR0TAQH%2FBAUwAwIBADARBglghkgB%0AhvhCAQEEBAMCB4AwHQYDVR0OBBYEFDncLvrw33MLZp8PF0ZAOSfv1F0uMIG6BgNV%0AHSMEgbIwga%2BAFJkP9SMkLIe91IE6UWAt3G%2F2xNFCoYGApH4wfDELMAkGA1UEBhMC%0AQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdp%0AdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZIENBMRswGQYJKoZIhvcN%0AAQkBFgxub0ByZXBseS5jb22CFDX9%2FiZqCgLU2XtnjSZn1%2Bf4TU5jMHQGCCsGAQUF%0ABwEBBGgwZjA8BggrBgEFBQcwAoYwaHR0cHM6Ly9leGFtcGxlLm15dXJsL2NhLmNy%0AdC9jZXJ0aWZpY2F0ZXMvY2EuY3J0MCYGCCsGAQUFBzABhhpodHRwczovL2V4YW1w%0AbGUubXl1cmwvb2NzcDAOBgNVHQ8BAf8EBAMCA6gwFgYDVR0lAQH%2FBAwwCgYIKwYB%0ABQUHAwIwDQYJKoZIhvcNAQELBQADggIBALhNoR9xpmMfEepzmQdYdiMiRbxEHEZw%0APU5k%2FPir1pMnVtino0TfCzX%2F%2Bk7B%2FF5IYHy%2BrwRV23reePMLx%2BgJmcTc1TpmgeNN%0AqVBEwXT6NLo8e%2FJ3OKbghuLt6L4adgdkDSsEyCutuxDCzLovm%2B5SVnEd4qLQcSDR%0ABbxPU%2B1OTx9wCHBYCWMGIsJqVWNqILu3bVZa6vJgrypV579Q05pF2vYvnXdvCMsy%0AFVJ2DNCkCxhhIPaikUcWcbIXYZ5cFiQW4XA%2BXJGItq85p6yKRdPaLUHvMRvCQhZo%0AErdnxMdu2HnsEFWgnx%2FzjilBlOmZOg8V0DCL3P9Dztn4yHxDk2DhPbuJO0dpHVnC%0AhYpvy%2BJhtHecJ0IDLUu4EQkpv9l32Tg2a84UuVgxO3l%2BDIkwdoLHt9B%2F59000ea0%0AMgAMzx6ZmGcKWJMn4Qjuz0rveyg%2F8B5IFIOzhOmawi9tx6oqCvMd02nr5bwFJHDj%0AH5vLnE5ewnR6txAUWpGvENFjiXb5OLU4r%2FDyqt9RJcS5C6nqtNvMjFtOcJcs1sbj%0AAKDVeufJm9QP2z5NhSdvtCEszwF6YCILot18E3XvTa6U35BpQ0QoIq2%2FUUc%2F453U%0AXLSQOjRg73o4nzzGFKOc4IA4By%2BSZURHSJF0W2jptgUTx%2BfNPbl0JOIM692zanfn%0AE3KNAAiW87El%0A-----END%20CERTIFICATE-----%0A

Content-Length: 864

User-Agent: python-requests/2.31.0

Accept-Encoding: gzip, deflate

Accept: */*

Content-Type: application/json

when setting proxy_set_header SSL_CLIENT_CERT $ssl_client_escaped_cert the following headers are set

ssl-client-verify: SUCCESS

ssl-client-subject-dn: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

ssl-client-issuer-dn: emailAddress=no@reply.com,CN=Testing ONLY CA,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

X-Request-ID: added607a8103e957745479fed0b3e4d

X-Real-IP: < truncated >

X-Forwarded-For: < truncated >

X-Forwarded-Host: debug.local

X-Forwarded-Port: 443

X-Forwarded-Proto: https

X-Forwarded-Scheme: https

X-Scheme: https

SSL_CLIENT_S_DN: CN=client2,O=Internet Widgits Pty Ltd,ST=Some-State,C=AU

SSL_CLIENT_VERIFY: SUCCESS

SSL_CLIENT_CERT: -----BEGIN%20CERTIFICATE-----%0AMIIGBTCCA%2B2gAwIBAgIUHctJCt3jhSeOtfoOniqVOSDxeUQwDQYJKoZIhvcNAQEL%0ABQAwfDELMAkGA1UEBhMCQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoM%0AGEludGVybmV0IFdpZGdpdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZ%0AIENBMRswGQYJKoZIhvcNAQkBFgxub0ByZXBseS5jb20wHhcNMjQwMzA3MTU1NDIx%0AWhcNMjUwMzE3MTU1NDIxWjBXMQswCQYDVQQGEwJBVTETMBEGA1UECAwKU29tZS1T%0AdGF0ZTEhMB8GA1UECgwYSW50ZXJuZXQgV2lkZ2l0cyBQdHkgTHRkMRAwDgYDVQQD%0ADAdjbGllbnQyMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA7RIC%2BSjw%0AO4Y95uEYcAEHdxbue6mer%2FKC6%2BPcA8Sr7zI3BnrxihAH9XDf2lX1%2BBLVSpeyalJp%0AuSPfm3w9U20fbgj6obFZD56xhT7MC915Bx6xqC4ba81ZltcXcRuaf1JCOj9r4KUa%0Aboz6flv5UC9mkr2xnUtTOUGXy%2FqFopSsUA5YTltoQDcfqF7bF1qUi8tpO8yqsXLP%0AxcT1kMBswED%2BSIY1r1B4yqeigaZxav2qvzj2M%2BEZT%2FfIE6Ui0WCMGWJUgier4XGe%0AFWm6dyKeKk0Dp0jbJ53%2B6PS5ejHOJG0oqZ%2B89EqMYmEmFXuy%2FYzbYJ9BOjdQpon6%0AgiuAM58fI9ImkQIDAQABo4IBojCCAZ4wDwYDVR0TAQH%2FBAUwAwIBADARBglghkgB%0AhvhCAQEEBAMCB4AwHQYDVR0OBBYEFDncLvrw33MLZp8PF0ZAOSfv1F0uMIG6BgNV%0AHSMEgbIwga%2BAFJkP9SMkLIe91IE6UWAt3G%2F2xNFCoYGApH4wfDELMAkGA1UEBhMC%0AQVUxEzARBgNVBAgMClNvbWUtU3RhdGUxITAfBgNVBAoMGEludGVybmV0IFdpZGdp%0AdHMgUHR5IEx0ZDEYMBYGA1UEAwwPVGVzdGluZyBPTkxZIENBMRswGQYJKoZIhvcN%0AAQkBFgxub0ByZXBseS5jb22CFDX9%2FiZqCgLU2XtnjSZn1%2Bf4TU5jMHQGCCsGAQUF%0ABwEBBGgwZjA8BggrBgEFBQcwAoYwaHR0cHM6Ly9leGFtcGxlLm15dXJsL2NhLmNy%0AdC9jZXJ0aWZpY2F0ZXMvY2EuY3J0MCYGCCsGAQUFBzABhhpodHRwczovL2V4YW1w%0AbGUubXl1cmwvb2NzcDAOBgNVHQ8BAf8EBAMCA6gwFgYDVR0lAQH%2FBAwwCgYIKwYB%0ABQUHAwIwDQYJKoZIhvcNAQELBQADggIBALhNoR9xpmMfEepzmQdYdiMiRbxEHEZw%0APU5k%2FPir1pMnVtino0TfCzX%2F%2Bk7B%2FF5IYHy%2BrwRV23reePMLx%2BgJmcTc1TpmgeNN%0AqVBEwXT6NLo8e%2FJ3OKbghuLt6L4adgdkDSsEyCutuxDCzLovm%2B5SVnEd4qLQcSDR%0ABbxPU%2B1OTx9wCHBYCWMGIsJqVWNqILu3bVZa6vJgrypV579Q05pF2vYvnXdvCMsy%0AFVJ2DNCkCxhhIPaikUcWcbIXYZ5cFiQW4XA%2BXJGItq85p6yKRdPaLUHvMRvCQhZo%0AErdnxMdu2HnsEFWgnx%2FzjilBlOmZOg8V0DCL3P9Dztn4yHxDk2DhPbuJO0dpHVnC%0AhYpvy%2BJhtHecJ0IDLUu4EQkpv9l32Tg2a84UuVgxO3l%2BDIkwdoLHt9B%2F59000ea0%0AMgAMzx6ZmGcKWJMn4Qjuz0rveyg%2F8B5IFIOzhOmawi9tx6oqCvMd02nr5bwFJHDj%0AH5vLnE5ewnR6txAUWpGvENFjiXb5OLU4r%2FDyqt9RJcS5C6nqtNvMjFtOcJcs1sbj%0AAKDVeufJm9QP2z5NhSdvtCEszwF6YCILot18E3XvTa6U35BpQ0QoIq2%2FUUc%2F453U%0AXLSQOjRg73o4nzzGFKOc4IA4By%2BSZURHSJF0W2jptgUTx%2BfNPbl0JOIM692zanfn%0AE3KNAAiW87El%0A-----END%20CERTIFICATE-----%0A

Content-Length: 864

User-Agent: python-requests/2.31.0

Accept-Encoding: gzip, deflate

Accept: */*

Content-Type: application/json

edit:

• fix formatting and add images

[jon-nfc]

anyone?

is this a bug or have I misconfigured something?

[Marek_Hulan]

The certificate, that the callback uses needs to be trusted by the CA, that is specified in the Foreman config file. You may mix multiple CA certificates in such pem file. You could probably reuse the Foreman’s own certificate, though that feels more like a hack. If you have default Foreman without Katello, your setup is using Puppet CA for generating certificates, so you may just generate a new one for your AWX callback. If you have katello, there’s katello-ssl-tool IIRC.

[jon-nfc]

G’day Marek, thanks for taking the time to reply.

Of note from your response. All certificates are trusted and have been signed by the same CA. (OpenSSL used to create CA and subsequent certs, all with correct extensions.) The mTLS from the client works to the https reverse proxy that sits in front of foreman. The same SSL certs are used within foreman and on the reverse proxy.

I have an inkling you may have missed an important part of the OP. Being Distribution and version: Custom. Alpine linux container running on kubernetes with nginx ingress controller.

I have completely dockerized foreman. I’m using Alpine linux as the base image with foreman installed from ruby gems with node running the interface. Currently I’m testing the dockerized foreman on kubernetes. Whilst the container appears to be working well, I’m having difficulty in being able to authenticate (mTLS) the foreman proxies and the ansible foreman callback. I’ve tried using both nginx and apache as the reverse proxy for foreman. however with no luck when it comes to foreman proxy/ansible callback auth. The proxy has been setup to do the TLS termination and also pass the vars back to foreman. mTLS is successful from the reverse proxy, both apache and nginx.

If I set Restrict registered smart proxies=no this setting appears to allow unauthenticated access to the proxy API, including the ansible foreman callback.

However if I set Restrict registered smart proxies=yes authentication fails as foreman seems to be of the opinion of No SSL cert with CN supplied. this leads me to believe one of two things is occurring: I have misconfigured something or foreman does not respect/use the env vars passed from the proxy (SSL_CLIENT_VERIFY, SSL_CLIENT_CERT and SSL_CLIENT_S_DN)

I’m not a ruby or JS developer so am sort of at a loss. I did source code dive and arrived at smart_proxy_auth.rb Lines 46-63, which has lead me to believe that foreman may not be respecting/using the headers passed by the proxy. I do have the proxy sending headers X-Forwarded-Proto: https, X-Forwarded-Scheme: https and X-Scheme: https. Although I have arrived at this opinion I’m fully aware that I have no idea re the code. honestly, ruby to me looks alien.

Any assistance you may be able to offer would be appreciated.