RFC: Putting smart-proxy behind Apache

This is more of a conversational RFC to collect up feelings on the idea, edge cases, caveats, and generally spark some discussion.

Proposal:

Deploy smart-proxy running on localhost behind Apache using reverse proxying to send requests for the smart proxy to the smart proxy running on ports 9090 and 8000. Apache would be:

  • Exposing 443 and 80 for communication with the smart-proxy
  • Terminate SSL and free up the smart-proxy from having to handle it

Benefits

  • Reduce use of ports that are used to common 443 and 80 on the main server (where a smart-proxy is deployed by default)
  • Allows for common pattern in deployments such as Katello that deploys a reverse proxy on external proxies already

Open Questions

  • Are there cases where Apache hurts the deployment of a smart-proxy?
  • Is this an advanced use case rather than the default?
  • Should this be the default just for Katello use cases?
  • What workflows would break with this change?
  • The smart-proxy API is not currently rooted, changing the smart-proxy API spec to more of a standard with a namespace (this may be its own RFC)
1 Like

To answer some of your questions:

  • Are there cases where Apache hurts the deployment of a smart-proxy?
    Yes, Windows installations of the Smart Proxy, and also when it is already hard to convince a costumer to get a Smart Proxy installed on an already running system (like a DNS or DHCP server in a separate subnet).
  • Is this an advanced use case rather than the default?
    I would say so, many people have at least some knowledge about Apache, but redirect and proxy is still hard for them.
  • Should this be the default just for Katello use cases?
    I would also be fine with it being default on Foreman without Katello to have both setup option be more in line with each other.
  • What workflows would break with this change?
    I see no breaks with it except from the already mentioned problems. I never installed a Smart Proxy on a system already running Apache.

This also allows us to leverage httpd as (reverse) proxy. There are at least two smart-proxy modules which are simply dump http(s) proxies: templates and discovery. If this proposal is successfully implemented, we could actually use Apache httpd to do proxying for us. More than that, reverse proxy could be set up so Foreman could also talk to managed systems in managed subnets via an unified way. This could be used for accessing hypervisors as well.

I like the idea. Getting rid of X.509 in proxy would be great, if we are able to figure out Windows deployments. This is an awful territory tho, we are lacking documentation, automation (no unified installer) and know-how.

Maybe it’s time to say goodbye to WinAPI and start deploying only on WSL? Like having a WSL smart-proxy Ubuntu repository that would pull all the dependencies like httpd with it. We could even use puppet installer without changes.

Not sure if this tech is in older Windows Servers tho, probably not.

I personally don’t see this as a benefit. Keeping it on its own port(s) keeps things much more straight forward.

Note that this wasn’t mentioned here, but the expectation is that there’s now a single (ip/name, port) tuple for Foreman and Foreman Proxy. So this wouldn’t be a separate vhost (which would mitigate a lot of my concerns).

Another (smaller) one is the increased attack vector. Currently you can have a local connection directly to the port Apache is talking to, it is possible to pretend a valid certificate is presented with invalid data. This can be mitigated by letting the Smart Proxy listen on a unix socket with strict permissions.

It increases the system requirements and makes debugging more complex. Currently a single system needs to be verified (Foreman Proxy) to work. With a reverse proxy, it’s another place that can fail.

I think this only makes life easier for the maintainers of Foreman Proxy but moves that overhead into an operational overhead for users. I fail to see how this makes it easier for users.

IMHO we should aim to reduce the differences between Foreman and Katello rather than to divert.

This makes the certificate situation more complex. Currently Foreman uses the following settings when connecting to a Smart Proxy:

  • ssl_certificate
  • ssl_priv_key
  • ssl_ca_file

The last one is important. It means that all the CAs for every Smart Proxy must match which implies that if you’re using a separate CA (like Let’s Encrypt or similar) on one, all of them must use that. If you’re mixing Foreman and Foreman Proxy on the same vhost and want to commercial certificates for https://foreman.example.com, it will mean that you now need to buy a certificate for every Smart Proxy.

I’m not sure what you exactly mean by this. Do you mean it’s currently hard bound to / rather than being able to live in /foreman-proxy?

Overall this feels like a step backwards to me.

A follow up to try to help me compartmentalize. I could extrapolate this argument out to Foreman itself. What’s the difference? Or would you make the same argument for Foreman?

My example would be Katello where a reverse proxy is deployed by default in all cases.

In the Katello case, this is already true. I think you are hitting on some of why I suggested maybe this fits into Katello’s current deployment model which adds some expectation layers and opinions on deployment.

Not just being able to, but done so by default. As in the changing the spec itself to have a common root, e.g. /smart-proxy/api/v1/features

I consider ports basically free. Why would we want to reduce the number of used ports?

Why is this used on Katello deployments?

In my opinion, this reduced the complexity and it makes automated testing harder. Any application logic should be handled inside the application with proper unit tests and not being handled in the infrastructure.

Apologies for being so blunt: In my opinion, this is at best a workaround.

I consider this a good thing to not solve this in the infrastructure but rather in the application.

Why is that?

Technically yes, but Foreman is already more complex since it has static files. Something the proxy doesn’t.

One benefit I do see is that there’s now a single way to set ciphers and protocols (Apache), rather than in an application specific format. However, that can also be achieved by a reverse proxy in Apache on its own port.

To be clear, I am strongly against putting the Foreman Proxy on the same port as Foreman. Regarding Apache as a reverse proxy in front of Foreman Proxy I’m more nuanced and can see some benefits.

That surprises me because I always assumed Katello worked in the same way. I consider this a bug that it doesn’t.

IMHO it’s an illusion that you get rid of it, unless you fully change the authentication mechanism to something else. Even if you offload the actual SSL to a reverse proxy, you still need to be aware of it when authenticating API endpoints.

There are users in sectors that have more stringent security rules and prefer to have as few ports as possible exposed.

When Katello deploys an external smart-proxy, a reverse proxy is deployed to allow those hosts managed by that proxy to have an isolated connection. When registering or receiving updates, communication flows through the reverse proxy back to the server. The Apache present also serves up content coming from Pulp.

What is the application logic in this case that Apache would be handling rather than the smart-proxy?

A work around for what exactly?

You could also argument that they prefer less running services. A reverse proxy in general does not add any security benefit. I’d prefer dedicated ports per services.

If you start to introduce url rewriting and similar things, I consider this part of the application logic.

I was actually hoping for an answer to this question. Can we solve this without the need to rely on infrastructure components for our app(s) to run?
From my experience devs try to make their lives easier by putting parts of the application into infrastructure. This usually makes the lives of the folks who have to run the apps harder.

+1 - I believe this adds unnecessary complexity and coupling for little to no gain

For the security conscious, it’s about more than just limiting the number of ports. For any security sensitive infrastructure (a service that controls DNS, DHCP, etc, is definitely security sensitive), it will always be a good idea to protect that service with a battle hardened, well understood infrastructure component. I see several reasons for this - none of which are unique to Foreman:

  • most developers, are not security experts, so I would rather place trust in something like an nginx or HAproxy reverse proxy. This is similar to the adage of friends don’t let friends implement crypto

  • using a reverse proxy can allow more flexibility in TLS termination, mutual authentication, WAF rules, etc. This also removes the burden of implementing these features in the application.

  • Limiting ports is of some benefit, especially if it means that the underlying security infrastructure does not need to be customized to handle http(s) traffic on non-standard ports. This is a bigger issue when you get into dynamic port allocation and the use of high numbered ephemeral ports.

Granted, most Foreman traffic should be traveling over well protected management networks, but I don’t think you can assume your users will always know or choose to do the right thing. So, if you can build some defensive muscle in to your offering, without having to code it yourselves, that seems to me like something worth considering.

It’s not just about exposed ports. These users are running security scanners which were causing issues to smart proxy in the past, for example making the proxy unavailable (port scan DoS essentially). Not a huge gain tho.

This ain’t illusion, SSL termination will ultimately lead to most of the X.509 code being dropped from smart-proxy core. I want code to be dropped, heck a code which handles security I want to be dropped so hard.

These are all good points and why I’m not against having a reverse proxy. It can be much easier on an admin if they only need to know how to configure Apache to use the secure ciphers. However, Apache can serve both applications on separate ports. This gives the benefit that you can actually firewall (using iptables) the smart-proxy to only allow Foreman rather than all clients. If you combine both services into a single port, you lose this advantage and need to do it at the Apache level which is more complex and less safe.

Smart-proxy does not do any crypto itself. It uses trusted libraries (in the end the same OpenSSL that nginx, httpd or HAproxy use). No crypto code is written by us.

More flexibility means there is the possibility for unsecure configuration. We should ship a secure configuration by default. We can do this better if we control the software. From the top of my head, the lines of code necessary to activate TLS are very limited. We currently use client-based certificate auth, which is hard to do with reverse proxies as the application has to trust the reverse proxy but cannot verify that the connection originated from that proxy. Offloading SSL is something that was done in the 90s for performance reasons. Today it’s more secure to encrypt the whole connection. WAFs are snakeoil implemented to gain points in a security audit that you would otherwise fail to pass.

Is this still an issue in 2020?

That is something we should address and I’ve seen similar issues as “negative” results in a security audit. I believe switching to puma should help here. A reverse proxy won’t change the underlying cause. This brings me to yet another item: When we introduce a reverse proxy to the stack, we’d have to carefully synchronize timeouts and connection limits.

Is there a lot of code in place today?

I’m not referring just to crypto. I’m also not looking to criticize Foreman developers for any specific choices they may or may not have made. All I’m saying is, generally speaking, most developers are not security experts (nor should they be expected to be) and so robust security is not a core part of the design of their software. For this reason, I think that security sensitive code should be well protected. Software like Nginx, being always exposed to the Internet, will have, by necessity, developed a hardened code base and a responsive security team. Most backend systems code will never see that kind of rigorous testing, so it is unlikely to be as robust.

Yes, a secure by default configuration is always desirable. So, if a reverse proxy is part of the shipped application, you could provide a secure default reverse proxy configuration.

If the reverse proxy is part of the deployed application, then there is no need for the Foreman code to participate in the TLS connection at all. Again this might be a win as it gets security sensitive code out of Foreman, which simplifies the code base and places security responsibility in the hands of security hardened code.

TLS offload is still used today and not just for performance reasons. See my next comment below.

Defense-in-depth is a core security practice. It acknowledges that no single line of defense is ever perfect.

If you need to communicate between a less secure environment and a more secure environment, it is advisable to put a firewall between the two. A packet filter can help to limit access to only the service you wish to expose, but packet filters tend to only protect up to layer 4. A WAF and a reverse proxy allow you to provide protection at layer 7 - especially if they can see the HTTP traffic. I wouldn’t characterize mod_security as snake oil, it and other WAFs play an important function in protecting applications that speak HTTP(S).

If you put the reverse proxy on the same host as the Foreman code, then there is no risk of a man-in-the-middle attack and you gain much greater visibility and control over your application’s traffic. If the reverse proxy is not on the same host, you can still do TLS between your application and the inside of the reverse proxy.

Absolutely! If anything it is a way bigger problem now than it used to be.

I think the core of the issue is conceptual complexity for defenders. It is often said that complexity is the enemy of security. Nowhere is this more true than in the ability of defenders to understand the systems they are defending.

It used to be that we built networks with hard crunchy exteriors and soft chewy centers. In this design, complex or non-standard application behavior on the LAN was not an issue, because no one looked at it. Now the evolving best practice is a so called “zero trust” design, where we assume that networks are compromised and yet we have to find trust anchors within these compromised networks. In this kind of environment, defenders now need to be able to conceptualize all layers of their infrastructure and they need control/visibility points distributed throughout the infrastructure.

When you’re dealing with bare metal (compute, networking, storage), you’re still dealing with the same basic building blocks that have been in place since the 90s. This is the space in which Foreman is used. We don’t have the benefits of the infrastructure abstraction that is provided by overlay networks, service registration in etcd, etc.

I know I’m taking a long hard look at the tools I use, in light of this zero trust paradigm and the realities of the cyber security environment that is driving these changes, and I’m trying to decide if their complexity is justified or whether something more simple or more easily controlled should be used.

This is a really long winded way of saying that I think simplicity and robust security is worth striving for down at the layer in which Foreman functions. The people and systems we use to deploy and manage infrastructure are an obvious target for attackers as they provide privilege escalation, amplification, extreme ease of lateral movement, persistence and largely invisible access to the applications in the layers above. We cannot assume that the hard crunchy exterior will protect the management network, because really the perimeter has fallen and the attackers are inside the fortress.

1 Like

@mason: Thanks for the long reply. I think we can both agree that security is very important for server management tools like Foreman and your efforts to strengthen the security are very much appreciated. While I personally enjoy this conversation, I would suggest that we continue it over some beer if we ever get the chance to meet in person so we can get back to the initial proposal here: Putting smart-proxy behind a reverse proxy.

Let me try to summarise what I understand:

  • Putting smart-proxy behind a reverse proxy has no clear measurable benefits.
  • This kind of deployment increases the complexity of the stack and involves a considerable amount of work. Securing the connection between the reverse proxy and the smart-proxy needs development effort and can only be achieved in a secure matter via unix sockets. We don’t have a solution for Windows deployments.
  • Smart Proxy is meant to be a lightweight agent. Putting it behind a reverse proxy contradicts this idea.
  • This change will most likely cause instability as the stack needs to be tuned (timeouts, number of connection, …)
  • The debugging effort and effort to run the application will increase.
  • We might fix some issues that motivate this change with the switch to puma.

So, my vote is to reconsider if this causes any benefits. If this task were to be paid with my development budget, I wouldn’t spend any more effort on this.

It would simplify hardening in terms of SSL protocols and ciphers. Now there’s a single language (Apache config) to set it. We also recently learned that the Ruby version is relevant and was limiting us (Ciphers Inconsistent with Documentation).

In my opinion, the price for this is too high. Have we any experience if this is still relevant with a puma only setup?

Thanks for the great conversations so far and thanks Timo for summarizing for me. I’ve learned quite a bit through this conversation.

I don’t believe have as of yet. I think it’s of value for folks to have a gander at just the changes required to implement Puma [1] acting in this way to present an HTTPS and HTTP server running from the same service. I for one find the code complex.

On the plus side, this change is now passing tests and ready to be pulled in for a full nightly cycle of testing before Foreman 2.1 release. This should help to gather some of these answers.

[1] https://github.com/theforeman/smart-proxy/pull/623/files

While this is true there is some code around OpenSSL. For example, list of enabled/disabled ciphers/sets is something we hardcode in our codebase, then somebody asked us to do blacklist, then whitelist and we ended up with an awful pair of YAML options. On the other hand, most admins do know how to setup Apache cipher list.

Well, OpenStack, OpenShift, containers in general - isn’t SSL termination the key concept? How you want to scale up without SSL termination? I am not so sure this argument is fair.

Granted it’s not so much

True. But I guess we’d ship a default and secure cipher list for apache and most users would not change these settings. Can’t we do the same for smart-proxy?
I guess users with very strict security requirements have the expertise to configure the cipher list for both httpd and smart-proxy.

Lucky for me, we don’t want to scale in this case. So I do believe this argument is still fair in this case. But I agree, there are use cases where it’s still valid in 2020.