But lorax creates bootable ISO images, it’s a tool used to generate Fedora installation or LiveCD ISO files. This does not make sense to me how this fits Katello/Foreman workflows.
Also, why to use ssh/cli when there is lorax-composer daemon API? That could be used directly by Katello.
It fits when you want to build your own images that should be then distributed by Katello and used in Foreman provisioning.
Sorry for the confusion: l had different usage of Rex in mind, than the original though in this post, which probably caused the misunderstanding: this is not about using ssh for talking to the service, but just getting the service up and running. The integration in the first phase will be probably more on documentation level, perhaps evolving into some UI integration later.
@aruzicka @iNecas is it possible currently to get an interactive SSH with REX these days? Basically what we would want is to send STDIN to a job execution that last as long as the session lasts. Or is this more of a DIY use case?
It’s not possible today, and seems like something that goes against the rex use-case itself, as here seems more like keeping a tunnel opened. OTOH I can imagine we could expose this functionality, but not for rex purposes itself, but for the cockpit integration case, so that we don’t have to keep the ssh-part on various places + leverage the proxy infrastructure.
help me understand how this would work with REX. My understanding is this:
Is this a correct assumption? If so, the browser connects to the Foreman Server, and the capsule connects to the host.
I assumed with Cockput that the browser needs toconnect to the host, or, we set up some bridging through the smart proxy. Having a connector run on the host is good, but it does not get the bridge we need via the capsule.
Daniel, can you elaborate more on what kind of integration is this all about? I still don’t understand the whole context. Is this integration of remote shell Cockpit feature (the nifty xterm.js they have for remote access via HTTPS)?
Here is a demo of the first baby steps: https://youtu.be/mGJB64jxsL0
Next I want to dig out the SSH parameters from Foreman that a remote execution would use for the given host, and pretend they will work from the Foreman master. Dealing with smart proxies and segmented networks comes later.
Marius, it looks great. Thank you for sharing with us the progress.
What is your plan in regard to authentication on the cockpit side? Using UNIX PAM account is probably not the best way, stealing the credentials should not give the attacker full root access. Perhaps some separate PAM account could be used and stored in Foreman inventory as parameters.
What is your plan in regard to authentication on the cockpit side?
The cockpit web server (cockpit-ws) can run as a unprivileged user or even in a container. It will use SSH to get access to the target host and run cockpit-bridge there.
The credentials to use with SSH will come from the Foreman API: user, password, private key, passphrase for the private key, etc. Foreman will only hand out those credentials to clients that are authorized appropriately.
In the demo, cockpit-ws uses the _session_id cookie, which fortunately also allows access to the API.
My idea right now is that the actual user, password, private key that cockpit-ws will get from Foreman are the ones that remote execution would use by default. People will need to setup their hosts for remote execution in order to get a seamless Cockpit.
My idea right now is that the actual user, password, private key that cockpit-ws will get from Foreman are the ones that remote execution would use by default. People will need to setup their hosts for remote execution in order to get a seamless Cockpit.
Here is a demo that shows this: https://www.youtube.com/watch?v=IM374KBtV04
I like seeing this. a couple of questions:
–bk
The code I have right now doesn’t work through a proxy. But we definitely need to involve the smart proxy somehow because of two reasons (afaics): The ssh private key might only be accessible to the smart proxy and not the Foreman master; and the target host might only be reachable by the smart proxy and not by the Foreman master.
Can you elaborate what you mean with “per user”?
My idea is that there is one new API entry point (in the foreman_remote_execution plugin) which gives out the ssh credentials. We can add any kind of authorization rules to that API so the credentials are only given out to certain users, for example.
We can also completely separate the Cockpit ssh credentials from the Remote Execution Credentials, but that would have the drawback that people would not be able to reuse their existing Remote Execution setup. They would need to distribute another set of public keys, for example.
The “Foreman Way” is probably to make this all possible and configurable, right?
We can also completely separate the Cockpit ssh credentials from the Remote Execution Credentials
I like reusing REX credentials. Besides it works out of the box, it’s single key pair which is easier to maintain.
Note that each REX proxy has private+public key, public keys are available via API, so that Foreman can deploy them during provisioning. But Foreman doesn’t have access to private key, that’s stored on smart proxy. So perhaps the “through smart-proxy connection” should be figured out first.
Yes, I start to think that, too. I ran into limitations of our cockpit-ssh already, and before starting to fix those, it would be good to have figured out the whole picture. A couple of options off the top of my head:
the smart proxy can reflect the sshd port of the target host to the smart proxy host, and cockpit-ws would connect to that.
the smart proxy remote execution plugin can get a new “interactive job” API call that would upgrade the GET or POST request to a WebSocket and cockpit-ws would talk to that. (Foreman could reuse that to run an interactive shell on the target host. I think there were some ideas in that direction.)
cockpit-ws could be co-located with the smart proxy and the reverse proxy in front of Foreman could redirect to that.
The second option is interesting. What would end/close such job? The third seems most natural but requires extra infra to setup and maintain. The first is probably most straightforward, but wouldn’t it have to ignore target host key verification? Ssh known hosts check would fail I guess after first connection.
Thanks for the feedback, that helps a lot. Please keep it coming. 
The client closing the TCP connection would end the job, the same as pressing C-d closes a regular SSH session.
I start to like this option most since it fits best into the existing Foreman architecture, no? No new communication patterns between master and proxies, no new ports need to be opened, etc.
But this option also needs the biggest code changes to both Foreman and Cockpit. I’ll play around with this a bit to get a feel for how big.
Okay, this starts to look promising.
Here is a demo, with terrible sound (apologies):
The central piece for the Cockpit session is cockpit-auth-foreman: https://github.com/mvollmer/foreman-cockpit/blob/master/cockpit-auth-foreman
It uses the new /api/v2/remote_execution/ssh_params API of Foreman to get parameters like proxy, user, keyfile, password, etc, that are also used by REX.
It then uses the new /remote_execution/ssh_session API in the Smart Proxy to start a interactive SSH session on the target host that runs cockpit-bridge. Once running, it will pump the Cockpit protocol messages in both directions until the session ends. That API does a protocol upgrade on its connection, which is something new for Smart Proxy, I think.
Code for the /api/v2/remote_execution/ssh_params API: https://github.com/mvollmer/foreman_remote_execution/commit/d35647905b19b23b76bbdd2f25c96fbbdff5a62e
Code for the /remote_execution/ssh_session API: https://github.com/mvollmer/smart-proxy/commit/5f0ded7db4049a7c08877f3c86f9137b8c7ad21d
Super dirty code for the button in Foreman and the OAuth bits: https://github.com/mvollmer/foreman/commits/seamless-cockpit
Open issues:
I should rewrite cockpit-auth-foreman in Ruby.
This doesn’t work without a proxy, but I think it should. A Ruby version of cockpit-auth-foreman can probably quite easily use Net::SSH directly on that case, and we should factor the code to make that elegant.
The ssh_session API needs to be implemented by smart_proxy_remote_execution_ssh, not by smart-proxy.
The code that uses Net::SSH should share as much as possible with script_runner.rb in foreman_remote_execution
But overall, I am pretty happy with this. Next steps for me would be
Rewrite cockpit-auth-foreman in Ruby.
Figure out how Foreman runs behind Apache in production and put Cockpit behind it as well.
Make a PR for the ssh_params API, with tests and everything.
Nice to see the progress here! So if I understand the workflow properly, when I click the Web Console,
It that right?
If I understand it correctly, in point 4, Cockpit uses the same session id to authenticate to the Foreman, as user’s browser, right?
I’m just trying to rephrase this here to make sure I understand how the whole thing works, as there are multiple pieces in play.
Yes, except for the “trust” part.
Cockpit itself is unprivileged and doesn’t make decisions related to authentication. The credentials given out by Foreman in step 4 just need to be accepted by sshd in step 5. Foreman might decline access in step 4 based on which user the session is for, etc.
Yes, the Foreman API fortunately allows access with a valid _session_id cookie, and the token exchanged in steps 2 and 3 is the value of that cookie. cockpit-auth-foreman turns the token back into a cookie.
A customer asked for better integration in regard to PCP performance data from hosts. While we will unlikely be writing a Foreman PCP plugins to present data from PCP, we should be able to have a button for hosts that opens up Cockpit interface on Performance page. I see there are plans on elaborating Cockcpit-PCP integration: https://github.com/cockpit-project/cockpit/wiki/Feature:-PCP-Integration