This is a fun one from a while ago. Let me warn you up front though: it goes against the principle to not login on your servers via SSH and use deployment tools like Ansible, but it has its use cases. So, what’s this all about?
Well, at a job I had some time ago, it was requested that developers should be able to login, upon request, on the production servers via SSH. This is popularly called JIT (Just in Time) access.
Now regarding the SSH access, before you go all berserk, this was at a small environment, let’s say 50 employees tops, and these developers were very tech savvy. Also, we’re talking non-admin access here on 20 servers max. So put away your pitchforks and read on.
Normally access to production systems should be limited to (in order of preference):
- Production CI/CD pipelines
- Configuration management tools like Ansible
- Server administrators
The solution
After some research and deliberation, I found this challenge can be solved with standard SSH tools available by default in every Linux distro. For this to work you should have SSH key based access configured. You should always have this configured (contrary to username/password access) but if you haven’t please read this first. It’s a good article, although maybe a bit long, explaining all ins and outs how to set this up.
Server setup
OK so we were already setup or just got setup with SSH key based access. Next step is something called CA signed keys. With SSH CA signed keys we can give temporary access to these live production systems. At the production server we’ve got the SSH keys stashed of every developer in the /root/.keys directory. We’re deploying this and keeping this up to date with Ansible. You can put these public keys anywhere on the server, just take care its secure.
# chown -R root:root /root/.keys
# chmod -R 0400 /root/.keys
# cd /etc/ssh
# ssh-keygen -f ssh_ca -t rsa -b 8192 -C "ssh_ca@servername.yourdomain.com"
# chmod 400 ssh_ca*
To be sure, I’m explaining these flags:
-f is what we’re calling our new keypair
-t is the type of the keypair
-b is the amount of the bytes for our new keys
-C is the comment the public key shows up with (arbitrary)
And just to be clear: per server, we’re creating one of these CA key pairs, on the server itself. As you can see, we’re putting this CA key in /etc/ssh. The above command creates the ssh_ca file (private key) and the ssh_ca.pub file (public key).
In every /etc/ssh/sshd_config file on every server we must add the below line to recognize and allow the CA key which we’ve configured above:
TrustedUserCAKeys /etc/ssh/ssh_ca.pub
# /usr/sbin/sshd -t -f /etc/ssh/sshd_config
# systemctl restart sshd
At the clients, we’ve setup normal SSH key pairs as already said above and as described elaborately in the linked article if you hadn’t done so already. The public key of the developer is also in the servers /root/.keys directory, renamed to match the developers name.
Now for the access. When access is requested, a server administrator has to perform the below on the production server in the keys directory. We’re using user ‘Henk Batelaan’ as an example.
# ssh-keygen -s /etc/ssh/ssh_ca -I hbatelaan -n devhba -V +1d hbatelaan.pub
You can probably understand where the flags are for, but just to be sure:
-s is for the private key to sign with
-I is for the name of the key identity (an arbitrary name)
-n is for the user we want access to (the actual user you use to login on the server)
-V is for the time of validity
Here we’re granting access for a day (+1d). We can also grant it for a couple of hours (+4h) or weeks (+2w), etc. The above command will generate the hbatelaan-cert.pub file. Provide this key to the developer and remove it from the server once provided.
~/.ssh/config
file:
AddKeysToAgent yes
ForwardAgent yes
IdentityFile ~/.ssh/id_rsa
hbatelaan-cert.pub key to connect to the production system. In our example the key is in the home directory.
$ cd
$ chmod 400 hbatelaan-cert.pub
$ ssh -i hbatelaan-cert.pub devhba@servername.yourdomain.com
The developer should be connected. If you’d like, you can check expiration of the key on your local machine with:
$ ssh-keygen -L -f hbatelaan-cert.pub
This is for sure not one of the most elegant solutions I setup back in the day, but for this use case it was a perfect fit. It also might seem a bit tedious to set it up, but when you have the basis (up till ‘Client setup’) in place, and keep your config up to date with Ansible, you can have a developer up and running in 5 minutes.
As a final note, you might have discovered a security loophole here. Can you spot it on your own? Spoiler ahead!
This setup does not prevent that a developer puts his or her own, original public key in the /home/devhba/.ssh/authorized_keys file on the server, thereby granting his or herself unlimited access.
This can however be monitored and prevented in multiple ways, depending on available tooling. I decided to make the file and directory immutable with Ansible.
You can do this manually with:
# chmod 400 /home/devhba/.ssh/authorized_keys
# chattr +i /home/devhba/.ssh/authorized_keys
# chattr +i /home/devhba/.ssh/
I’m a passionate, communicative go-getter and highly motivated to build, maintain and improve a stable and effective IT infrastructure at different sized companies. My hearth is with open source, Linux, DevOps, Kubernetes and everything that is cloud native.
