Upgrade to RHEL 8.6 and Foreman 3.3 - /pub directory returning 404 and not listing anything

Problem: /pub directory not listing under the foreman server, same goes for the smart proxies sites.

Expected outcome: expecting /pub to be able to list content.

Foreman and Proxy versions: Foreman 3.3 with RHEL 8.6 (Latest)

Foreman and Proxy plugin versions: Foreman 3.3 with RHEL 8.6 (latest)

Distribution and version: RHEL 8.6

Other relevant data:

Any assitance on this, or this is the new behavior after the upgrade?

Did you stop all services and ran foreman-installer after the upgrade?

What I’m seeing is that directory browsing doesn’t seem to be working, though defined properly in Apache configs, on my particular RHEL 8.6 based instance, though I’ve ensured +Indexes was set on the right Location tag for /pub, which is why /pub isn’t showing a directory listing for me. I can access files within /pub no problem if I specify the full path to the file, such as /pub/katello-ca-consumer-latest.noarch.rpm. If I figure mine out, I’ll share.

1 Like

gvde, yes all services were stop prior to the upgrade.

I am still getting the folloing under /pub

Not Found
The requested URL was not found on this server.

Thank you Wes for the reply, perhaps this is a small bug that was left out of 3.3,

Maybe someone can help us with the apache config parameters that need to be adjusted for the listing to start working again. Thank you.

So you did not run foreman-installer after the upgrade? That would set up the apache configuration…

Hi gvde, do you happen to have a workaround for those of us experiencing the issue? Thank you in advance.

You have to run foreman-installer after the upgrade…

Hi gvde,
I try running the foreman-installer last night and the issue with /pub remains. Please provide guidance on a workaround if avaiable. Thank you.

Did foreman-installer complete without any errors?

What do you have in /etc/http/conf.d/05-foreman.conf and 05-foreman-ssl.conf?

There is little point manually adding the it to those files. It would be much more important to understand why it’s missing and solve that…

Hi gvde,
Thank you for your input…below find the output of my two files:

Code for 05-foreman.conf


cat /etc/httpd/conf.d/05-foreman.conf 
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
# 
<VirtualHost *:80>
  ServerName foreman.localdomain

  ## Vhost docroot
  DocumentRoot "/usr/share/foreman/public"

  ## Directories, there should at least be a declaration for /usr/share/foreman/public

  <Directory "/usr/share/foreman/public">
    Options SymLinksIfOwnerMatch
    AllowOverride None
    Require all granted
  </Directory>

  ## Load additional static includes
  IncludeOptional "/etc/httpd/conf.d/05-foreman.d/*.conf"

  ## Logging
  ErrorLog "/var/log/httpd/foreman_error.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/foreman_access.log" combined 

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X_FORWARDED_PROTO "http"
  RequestHeader set SSL_CLIENT_S_DN ""
  RequestHeader set SSL_CLIENT_CERT ""
  RequestHeader set SSL_CLIENT_VERIFY ""
  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

  <Location "/pulp/deb">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/isos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/repos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>
Alias /pub /var/www/html/pub

<Location /pub>
  <IfModule mod_passenger.c>
    PassengerEnabled off
  </IfModule>
  Options +FollowSymLinks +Indexes
  Require all granted
</Location>

  <Location "/pulp/content">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost On
  ProxyAddHeaders On
  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / unix:///run/foreman.sock|http://foreman/ retry=0 timeout=900
  ProxyPassReverse / unix:///run/foreman.sock|http://foreman/
  ## Rewrite rules
  RewriteEngine On

  #Upgrade Websocket connections
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*) unix:///run/foreman.sock|ws://foreman/$1 [P,L]


  ## Server aliases
  ServerAlias foreman

  ## Custom fragment
  # Return compressed assets if they are precompiled
<IfModule mod_rewrite.c>
  RewriteEngine on
  # Make sure the browser supports gzip encoding and file with .gz added
  # does exist on disc before we rewrite with the extension
  RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b
  RewriteCond %{REQUEST_FILENAME} \.(css|js|svg)$
  RewriteCond %{REQUEST_FILENAME}.gz -s
  RewriteRule ^(.+) $1.gz [L]

  # Set headers for all possible assets which are compressed
  <FilesMatch \.css\.gz$>
    ForceType text/css
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
  <FilesMatch \.js\.gz$>
    ForceType text/javascript
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
  <FilesMatch \.svg\.gz$>
    ForceType image/svg+xml
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
</IfModule>

<LocationMatch "^/(assets|webpack)">
  Options SymLinksIfOwnerMatch
  AllowOverride None
  Require all granted

  # Use standard http expire header for assets instead of ETag
  <IfModule mod_expires.c>
    Header unset ETag
    FileETag None
    ExpiresActive On
    ExpiresDefault "access plus 1 year"
  </IfModule>
</LocationMatch>


  AddDefaultCharset UTF-8
</VirtualHost>

Code for 05-foreman-ssl.conf

cat /etc/httpd/conf.d/05-foreman-ssl.conf 
# ************************************
# Vhost template in module puppetlabs-apache
# Managed by Puppet
# ************************************
# 
<VirtualHost *:443>
  ServerName foreman.localdomain

  ## Vhost docroot
  DocumentRoot "/usr/share/foreman/public"

  ## Directories, there should at least be a declaration for /usr/share/foreman/public

  <Directory "/usr/share/foreman/public">
    Options SymLinksIfOwnerMatch
    AllowOverride None
    Require all granted
  </Directory>

  ## Load additional static includes
  IncludeOptional "/etc/httpd/conf.d/05-foreman-ssl.d/*.conf"

  ## Logging
  ErrorLog "/var/log/httpd/foreman-ssl_error_ssl.log"
  ServerSignature Off
  CustomLog "/var/log/httpd/foreman-ssl_access_ssl.log" combined 

  ## Request header rules
  ## as per http://httpd.apache.org/docs/2.2/mod/mod_headers.html#requestheader
  RequestHeader set X_FORWARDED_PROTO "https"
  RequestHeader set SSL_CLIENT_S_DN "%{SSL_CLIENT_S_DN}s"
  RequestHeader set SSL_CLIENT_CERT "%{SSL_CLIENT_CERT}s"
  RequestHeader set SSL_CLIENT_VERIFY "%{SSL_CLIENT_VERIFY}s"
  RequestHeader unset REMOTE_USER
  RequestHeader unset REMOTE_USER_EMAIL
  RequestHeader unset REMOTE_USER_FIRSTNAME
  RequestHeader unset REMOTE_USER_LASTNAME
  RequestHeader unset REMOTE_USER_GROUPS

  # SSL Proxy directives
  SSLProxyEngine On

  ProxyPass /pulp_ansible/galaxy/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp_ansible/galaxy/
  ProxyPassReverse /pulp_ansible/galaxy/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp_ansible/galaxy/

  <Location "/pulpcore_registry/v2/">
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE_USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    ProxyPass unix:///run/pulpcore-api.sock|http://pulpcore-api/v2/
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pulpcore-api/v2/
  </Location>

  ProxyPass /pulp/container/ unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/container/
  ProxyPassReverse /pulp/container/ unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/container/

  <Location "/pulp/deb">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/isos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/repos">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>
Alias /pub /var/www/html/pub

<Location /pub>
  <IfModule mod_passenger.c>
    PassengerEnabled off
  </IfModule>
  Options +FollowSymLinks +Indexes
  Require all granted
</Location>

  <Location "/pulp/content">
    RequestHeader unset X-CLIENT-CERT
    RequestHeader set X-CLIENT-CERT "%{SSL_CLIENT_CERT}s" env=SSL_CLIENT_CERT
    ProxyPass unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content disablereuse=on timeout=600
    ProxyPassReverse unix:///run/pulpcore-content.sock|http://pulpcore-content/pulp/content
  </Location>

  <Location "/pulp/api/v3">
    RequestHeader unset REMOTE_USER
    RequestHeader set REMOTE_USER "%{SSL_CLIENT_S_DN_CN}s" env=SSL_CLIENT_S_DN_CN
    RequestHeader set REMOTE_USER "admin" "expr=%{SSL_CLIENT_S_DN_CN} == 'sjglcsappprdg15.na.gilead.com'"
    ProxyPass unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/api/v3 timeout=600
    ProxyPassReverse unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/api/v3
  </Location>

  ProxyPass /pulp/assets/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/assets/
  ProxyPassReverse /pulp/assets/ unix:///run/pulpcore-api.sock|http://pulpcore-api/pulp/assets/

  ## Proxy rules
  ProxyRequests Off
  ProxyPreserveHost On
  ProxyAddHeaders On
  ProxyPass /pulp !
  ProxyPass /pulp2 !
  ProxyPass /streamer !
  ProxyPass /pub !
  ProxyPass /icons !
  ProxyPass / unix:///run/foreman.sock|http://foreman/ retry=0 timeout=900
  ProxyPassReverse / unix:///run/foreman.sock|http://foreman/
  ## Rewrite rules
  RewriteEngine On

  #Upgrade Websocket connections
  RewriteCond %{HTTP:Upgrade} =websocket [NC]
  RewriteRule /(.*) unix:///run/foreman.sock|ws://foreman/$1 [P,L]


  ## Server aliases
  ServerAlias foreman

  ## SSL directives
  SSLEngine on
  SSLCertificateFile      "/etc/pki/katello/certs/katello-apache.crt"
  SSLCertificateKeyFile   "/etc/pki/katello/private/katello-apache.key"
  SSLCertificateChainFile "/etc/pki/katello/certs/katello-server-ca.crt"
  SSLVerifyClient         optional
  SSLVerifyDepth          3
  SSLCACertificateFile    "/etc/pki/katello/certs/katello-default-ca.crt"
  SSLOptions +StdEnvVars +ExportCertData

  ## Custom fragment
  # Return compressed assets if they are precompiled
<IfModule mod_rewrite.c>
  RewriteEngine on
  # Make sure the browser supports gzip encoding and file with .gz added
  # does exist on disc before we rewrite with the extension
  RewriteCond %{HTTP:Accept-Encoding} \b(x-)?gzip\b
  RewriteCond %{REQUEST_FILENAME} \.(css|js|svg)$
  RewriteCond %{REQUEST_FILENAME}.gz -s
  RewriteRule ^(.+) $1.gz [L]

  # Set headers for all possible assets which are compressed
  <FilesMatch \.css\.gz$>
    ForceType text/css
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
  <FilesMatch \.js\.gz$>
    ForceType text/javascript
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
  <FilesMatch \.svg\.gz$>
    ForceType image/svg+xml
    Header set Content-Encoding gzip
    SetEnv no-gzip
  </FilesMatch>
</IfModule>

<LocationMatch "^/(assets|webpack)">
  Options SymLinksIfOwnerMatch
  AllowOverride None
  Require all granted

  # Use standard http expire header for assets instead of ETag
  <IfModule mod_expires.c>
    Header unset ETag
    FileETag None
    ExpiresActive On
    ExpiresDefault "access plus 1 year"
  </IfModule>
</LocationMatch>


  AddDefaultCharset UTF-8
</VirtualHost>

I am sorry, but it‘s extremely hard if you don‘t answer the questions you have been asked…

Again: Did foreman-installer run without errors after the upgrade?

What do you have in /var/www/html/pub?

Hi gdve, The formain-installer run fine after the upgrade no issues.

under /var/www/html/pub I have the following:

[root@foreman pub]# ls -la
total 112
drwxr-xr-x 2 apache apache   274 Jul  1 01:30 .
drwxr-xr-x 3 root   root      17 Jul  1 01:29 ..
-rw-r--r-- 1 apache apache 74211 Apr 20 06:00 bootstrap.py
lrwxrwxrwx 1 root   root      84 Jul  1 01:30 katello-ca-consumer-latest.noarch.rpm -> /var/www/html/pub/katello-ca-consumer-foreman.localdomain-1.0-1.noarch.rpm
-rw-r--r-- 1 root   root   10168 Jul  1 01:30 katello-ca-consumer-foreman.localdomain-1.0-1.noarch.rpm
-rw-r--r-- 1 root   root    9445 Jul  1 01:30 katello-ca-consumer-foreman.localdomain-1.0-1.src.rpm
-rwxr-xr-x 1 root   root    7837 Jul  1 01:30 katello-rhsm-consumer
-rw-r--r-- 1 root   root    2516 Jul  1 01:30 katello-server-ca.crt

/pub is there and it’s configured in httpd.

Verify foreman.localdomain really resolves to your foreman server and that your client/browser isn’t using a proxy interfering.

$ host foreman.localdomain

$ curl -v http://foreman.localdomain/pub

$ curl -v http://foreman.localdomain/pub/

Hi gvde, Below if the output I mam getting:

host foreman.localdomain
foreman.localdomain has address 10.22.9.140


curl -v http://foreman.localdomain/pub
*   Trying 10.22.9.140...
* TCP_NODELAY set
* Connected to foreman.localdomain (10.22.9.140) port 80 (#0)
> GET /pub HTTP/1.1
> Host: foreman.localdomain
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Sun, 03 Jul 2022 16:30:56 GMT
< Server: Apache
< Content-Length: 196
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
* Connection #0 to host foreman.localdomain left intact



[root@foreman conf.d]# curl -v http://foreman.localdomain/pub/
*   Trying 10.22.9.140...
* TCP_NODELAY set
* Connected to foreman.localdomain (10.22.9.140) port 80 (#0)
> GET /pub/ HTTP/1.1
> Host: foreman.localdomain
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Sun, 03 Jul 2022 16:31:24 GMT
< Server: Apache
< Content-Length: 196
< Content-Type: text/html; charset=iso-8859-1
< 
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
</body></html>
* Connection #0 to host foreman.localdomain left intact

And 10.22.9.140 is the up address of that foreman server you are on?

What files do you have in /etc/httpd/conf and conf.d?

$ ls -laR /etc/httpd/conf{,.d}

Hi gvde,
Below is the output of the conf directory as requested.

FYI, everything is working fine with my forman except the publish of the /pub directory.

ls -laR /etc/httpd/conf{,.d}
/etc/httpd/conf:
total 24
drwxr-xr-x 2 root root    55 Jul  1 01:31 .
drwxr-xr-x 5 root root   105 Jul  1 01:27 ..
-rw-r--r-- 1 root root  1029 Jul  1 01:31 httpd.conf
-rw-r--r-- 1 root root 13064 Jun 15 05:27 magic
-rw-r--r-- 1 root root   185 Jul  1 01:31 ports.conf

/etc/httpd/conf.d:
total 24
drwxr-xr-x 4 root root  141 Jul  3 09:24 .
drwxr-xr-x 5 root root  105 Jul  1 01:27 ..
-rw-r--r-- 1 root root 4643 Jul  3 09:28 05-foreman.conf
-rw-r----- 1 root root 4643 Jul  3 09:24 05-foreman.conf.original.07032022
drwxr-xr-x 2 root root    6 Jul  1 01:33 05-foreman.d
-rw-r--r-- 1 root root 6569 Jul  1 01:33 05-foreman-ssl.conf
drwxr-xr-x 2 root root   26 Jul  1 01:33 05-foreman-ssl.d

/etc/httpd/conf.d/05-foreman.d:
total 0
drwxr-xr-x 2 root root   6 Jul  1 01:33 .
drwxr-xr-x 4 root root 141 Jul  3 09:24 ..

/etc/httpd/conf.d/05-foreman-ssl.d:
total 4
drwxr-xr-x 2 root root  26 Jul  1 01:33 .
drwxr-xr-x 4 root root 141 Jul  3 09:24 ..
-rw-r--r-- 1 root root 791 Jul  1 01:33 katello.conf

gvde, any new update son this pending item? Please advise.

Configuration looks good. /pub should be served directly out of httpd. You should check httpd logs in /var/log/httpd if there is anything indicating why it’s not serving /pub. Also check httpd logs right after a restart if it shows any errors or warnings.