SSH host key CA with Cloud-Init (and Terraform)

When dealing with volatile infrastructure, we treat servers as cattle not pets. In a VPC, jumphost servers might even be launched on demand only. To ensure that we only connect to trusted servers launched by us, we can establish a Certificate Authority for our SSH host keys.

SSH host keys

Each SSH server instance identifies itself with a unique SSH host key. Upon every login, this host key is presented to the user:

$ ssh in-awe.fra1.o11ystack.org
The authenticity of host 'in-awe.fra1.o11ystack.org (157.245.24.111)' can't be established.
ECDSA key fingerprint is SHA256:EinMrd6XfGaRBgP2hnu...
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

In reality, nobody really checks the SSH host key upon login. The message is either always discarded with yes or configured away in your ~/.ssh/config via

StrictHostKeyChecking no

The following message should lead to a heads up, as the host key presented does not match the host key in you /.ssh/known_hosts file.

$ ssh in-ewe.fra1.o11ystack.org
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The ECDSA host key for in-ewe.fra1.o11ystack.org has changed,
and the key for the corresponding IP address 157.245.24.111
has a different value. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
Offending key for IP in /Users/torsten/.ssh/known_hosts:114
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:EinMrd6XfGaRBgP2hnu0z...
Please contact your system administrator.
[...]

Again, reality strikes again: The mismatching lines gets deleted from the /.ssh/known_hosts file and voila: everything is fine. Or is it?

SSH host key CA

The basic idea is to have a Certificate Authority (CA) signing each SSH host key. The CA’s public key is known to the client and can be used to verify the authenticity of the server.

Creating CA keys

$ ssh-keygen -f /etc/ssh/ca -t ecdsa
$ cat /etc/ssh/ca
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktd...
-----END OPENSSH PRIVATE KEY-----
$ cat /etc/ssh/ca.pub
ecdsa-sha2-nistp256 AAAAE2VjZHNh...

Signing host keys

As we are dealing with volatile infrastructure, we want to sign every SSH host key of every server we create. I use cloud-init to set up servers as it integrates seamlessly with Terraform and ~~most~~ all cloud providers.

#cloud-config
merge_how:
 - name: list
   settings: [append]
 - name: dict
   settings: [recurse_array]

write_files:
  - path: /etc/ssh/ca
    owner: root:root
    permissions: '0600'
    content: |
      -----BEGIN OPENSSH PRIVATE KEY-----
      b3BlbnNzaC1rZXktd...
      -----END OPENSSH PRIVATE KEY-----
  - path: /etc/ssh/ca.pub
    owner: root:root
    permissions: '0644'
    content: ecdsa-sha2-nistp256 AAAAE2VjZHNh...

runcmd:
  # sign and configure ssh host keys
  - ssh-keygen -s /etc/ssh/ca -I "$(hostname)" -n "$(hostname),$(hostname -I|tr ' ' ',')$(hostname).<FQDN>" -V -5m:+520w -h /etc/ssh/ssh_host_rsa_key.pub /etc/ssh/ssh_host_ecdsa_key.pub /etc/ssh/ssh_host_ed25519_key.pub

  # configure ca usage on the server
  - echo "@cert-authority * $(cat /etc/ssh/ca.pub)" >> /etc/ssh/ssh_known_hosts

  # configure new signed host certificates
  - for i in /etc/ssh/ssh_host*_key-cert.pub; do echo "HostCertificate $i" >>/etc/ssh/sshd_config; done

  # remove ssh key signing private key
  - rm -f /etc/ssh/ca

  # restart sshd to make changes have effect
  - systemctl restart sshd

Hostname principals

When signing the key you need to configure the principals (-n) (FQDNs and server names) the host can be reached with. The example above registers the hostname and the IPs configured.

$ hostname
in-awe
$ hostname -I|tr ' ' ','
157.230.28.200,10.17.0.4,10.135.0.1,2a03:b0c0:...

You might want to add the host’s FQDNs (hostname -f) to the list.

Certifcate validity

The validity is given as a validity interval (for server time skew). In the example above the certificate is valid for 5 years.

-V -5m:+520w

Caveats

Well, we are copying the CA’s private key to the server. I’d love to avoid that and precompute the host’s SSH keys, but unfortunately this is currently not possible in plain Terraform (only using external data providers).