VPS SSH Hardening Without the Drama: FIDO2 Hardware Keys, SSH CA, and Safe Key Rotation Step-by-Step

So there I was late one evening, watching a stream of failed SSH logins scroll by like a slot machine that never pays out, and it hit me: most of us treat SSH like a trusty old door lock. It works, we’re used to it, and we only think about it when something rattles. Then a client called about a contractor who’d left the company, their key was still trusted on dozens of servers, and the one person who knew how to clean it up was on vacation. You can imagine the stress. That’s when I doubled down on a calmer way to run SSH: hardware-backed keys, short-lived certificates, and a rotation plan that doesn’t make your heart race.

In this guide, I’ll walk you through the playbook I use today: FIDO2 hardware keys for real “something-you-have” authentication, an SSH certificate authority (CA) to stop sprinkling raw keys across every server, and a safe, two-phase key rotation process. Think of it as moving from a pocketful of mismatched keys to a single, sensible access badge that can be updated or revoked from the couch. We’ll keep it human, avoid the stiff jargon, and make sure each step builds on the last. And yes, we’ll talk about the little hiccups I’ve run into and how to dodge them.

Why Hardening SSH Feels Scary (And Why It Doesn’t Have to Be)

SSH is the front door to your VPS world. If you’ve ever watched bots hammer your server or worried about a lost laptop with private keys on it, you already know the stakes. The good news is that modern SSH, especially with FIDO2 hardware keys and certificates, lets you lock things down without turning everyone’s day into a scavenger hunt. The trick is setting the foundation so rotation and revocation are, frankly, boring. Boring is good.

Here’s the thing most people miss: raw public keys in authorized_keys work fine until you need to quickly revoke, expire, or audit at scale. Add a few dozen servers and a couple of human mistakes, and that simplicity turns into weekend work. FIDO2 keys help because they live on a hardware token, and SSH certificates help because you stop trusting raw keys and start trusting signed identities. It’s like stamping a temporary pass on top of a key; the pass expires, the key without a stamp means nothing, and life moves on.

FIDO2 Hardware Keys for SSH: The Calm, Physical Factor

I remember the first time I set up a FIDO2 key for SSH and realized I needed to touch it to log in. It felt oddly reassuring—like turning a physical deadbolt. If someone copied my private key file, it wouldn’t help them, because the signing happens on the hardware. Even better, the token can require a PIN and a touch, so you get that second factor vibe with zero SMS drama.

Let’s keep the overview friendly. FIDO2 is an open standard for secure authentication—hardware keys that prove you are you without handing out secrets like candy. If you want to read more background later, the FIDO Alliance FIDO2 page is a good starting point. OpenSSH has supported FIDO/U2F keys since 8.2, and it calls them “security key” types. On most systems you’ll use ssh-keygen with -t ed25519-sk or -t ecdsa-sk. The “-sk” stands for security key, the hardware thing you plug in or tap. The docs for this are surprisingly readable—see the OpenSSH ssh-keygen manual if you want every detail and option.

What You’ll Need

You’ll want the following in place: a recent OpenSSH client (8.2 or later has FIDO support), the system libraries for FIDO (often libfido2 and friends), and a hardware key that supports FIDO2 (YubiKey, SoloKey, Nitrokey, etc.). On macOS and modern Linux, it’s usually plug-and-go after you install the libs. Windows works too (recent builds of OpenSSH and vendor tools are fine), though the setup flow can feel a bit different.

Generating a Hardware-Backed SSH Key

Warm up those fingers—you’re going to touch the key a few times. Here’s a practical, safe default I use for most people:

ssh-keygen -t ed25519-sk -O resident -O verify-required -C "alice@laptop (FIDO2)"

What’s going on here? The -t ed25519-sk chooses a hardware-backed Ed25519 key. The -O resident option stores the credential on the key so you can use it across machines (handy if you enroll the same token on multiple laptops later). The -O verify-required tells OpenSSH to require that physical touch during authentication. The comment is just to make your life easier when you look at it later.

When prompted, set a PIN on the hardware key if you haven’t already; it’s your friend. You’ll end up with a public key (something like id_ed25519_sk.pub) and the private side bound to the hardware. If you’re curious about options, take a stroll through the ssh-keygen docs later. If you prefer ECDSA over Ed25519 for compatibility reasons, just swap ed25519-sk for ecdsa-sk.

Adding Your Public Key to a Server (The Old Way, Briefly)

Let’s do a quick sanity test before we get fancy with certificates. Copy your public key to a server’s authorized_keys:

ssh-copy-id -i ~/.ssh/id_ed25519_sk.pub user@server

Or, if you prefer to do it by hand, append id_ed25519_sk.pub into ~/.ssh/authorized_keys on the server. Try logging in. You should be asked to touch your hardware key. It feels a bit magical the first time.

While we won’t live in authorized_keys land for long, this step proves the hardware works. If you run into trouble, check logs on the server with journalctl -u ssh or sudo tail -f /var/log/auth.log. Many times it’s a permissions thing—SSH is picky about ownership and chmod 600 on authorized_keys.

Locking the Front Door: Tighter sshd_config Defaults

Since you’re here for hardening, let’s set a good baseline on the server. I like to make changes while keeping an existing session open, just in case. Consider these from experience:

# /etc/ssh/sshd_config (snippets)
Protocol 2
PubkeyAuthentication yes
PasswordAuthentication no
KbdInteractiveAuthentication no
ChallengeResponseAuthentication no
PermitRootLogin prohibit-password
MaxAuthTries 3
LoginGraceTime 20
PubkeyAcceptedKeyTypes ssh-ed25519,[email protected],[email protected]

Reload with sudo systemctl reload ssh (or sshd depending on your OS). Test before you close your other session. I can’t tell you how many times that last sentence saved me a trip to a data center.

SSH Certificates (CA): The Grown-Up Way to Trust Keys

Here’s the big mindset shift: stop trusting raw keys and start trusting certificates. Instead of sprinkling public keys across every server, we create a simple SSH CA—just a keypair that signs user keys—and we tell servers to trust the CA. Users keep their hardware-backed keys, but those keys need a time-limited certificate stamp to log in. When the certificate expires, access quietly fades. If someone leaves the team, you don’t have to hunt down which servers had their raw key; you just stop signing certificates for them or revoke their certs.

Think of it like a short-lived gym pass. Your identity is your hardware key, but the certificate is the sticker that says you’re allowed in this month. No sticker, no treadmill.

Step 1: Create the User CA

On a secure machine (ideally offline or a well-guarded management host), create a user CA:

ssh-keygen -t ed25519 -f ~/ssh-user-ca -C "SSH User CA"
# Creates ssh-user-ca (private) and ssh-user-ca.pub (public)

Backup the private CA key carefully. It’s the golden stamp. If it walks away, you’ll be rotating everything a lot sooner than you planned. Some teams store it on an encrypted USB in a safe, with checks to sign certs via a controlled process.

Step 2: Tell Servers to Trust the CA

Copy the public part (ssh-user-ca.pub) to each server and configure sshd to trust it:

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ca/trusted_user_ca.pub

Put the public CA key at /etc/ssh/ca/trusted_user_ca.pub with proper permissions. Reload sshd. Nothing changes yet for users, but now the server is ready to accept user certificates signed by your CA.

Step 3: Sign User Keys With Principals and Short TTLs

Instead of adding raw keys to authorized_keys, we sign a user’s public key with the CA and include one or more “principals.” Principals are like roles or identities. You might use someone’s username plus a group, like alice,ops. You can also set source-address limits and command restrictions on the certificate, which means your policies travel with the cert, not the server config.

# Sign Alice's hardware-backed key for 1 week, with principals "alice,ops"
ssh-keygen -s ~/ssh-user-ca -I alice-weekly -n alice,ops -V +1w -z 1001 
  -O source-address=203.0.113.0/24 ~/alice/id_ed25519_sk.pub

# Result: ~/alice/id_ed25519_sk-cert.pub (the certificate)

The -I is a cert identity string; I use something human like alice-weekly. The -n sets principals. The -V sets validity—one week is a nice place to start. The -z is a serial number; unique values make revocation and auditing friendlier. The -O source-address pins where the user can come from. Feel free to skip that option if your IPs are more fluid.

On the client side, Alice uses id_ed25519_sk plus the new certificate id_ed25519_sk-cert.pub when connecting. OpenSSH glues them together automatically if they share the same base name and directory.

Step 4: Prune authorized_keys and Enjoy the Quiet

Once your CA trust is live on all servers and users have certificates, you can remove raw keys from authorized_keys. I usually leave a break-glass key in there with a forced command and tight IP restriction, then store that key offline. It’s an insurance policy for bad days. In normal life, you’ll forget authorized_keys exists, because the trust moved to your CA.

Bonus: Host Certificates to Kill TOFU Anxiety

We’ve talked about user certificates, but you can also sign host keys with a host CA. This replaces the awkward “Are you sure you want to continue connecting (yes/no/[fingerprint])?” moment with a clean assertion: your client trusts any host that presents a certificate signed by your host CA.

# Create a host CA
ssh-keygen -t ed25519 -f ~/ssh-host-ca -C "SSH Host CA"

# On each server, sign the server's host key (usually /etc/ssh/ssh_host_ed25519_key.pub)
ssh-keygen -s ~/ssh-host-ca -I host-example -h -n server.example.com -V +52w 
  /etc/ssh/ssh_host_ed25519_key.pub

# Add the cert to sshd_config on that server
HostKey /etc/ssh/ssh_host_ed25519_key
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

On your clients, add a line to ~/.ssh/known_hosts that says “this CA is trusted for these domains.” It looks like this:

@cert-authority *.example.com ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...host-ca-pub...

Now every server cert signed by your host CA within *.example.com Just Works. No TOFU (trust on first use) finger-crossing. It’s amazing how much calmer your SSH sessions feel with this in place.

Safe Key Rotation: A No-Drama Two-Phase Plan

Rotation is where people usually get nervous. I don’t blame them. Years ago, I watched a team flip to new keys in one sweep and lock themselves out for half a day. The fix? Stagger the change. Always add before you remove. And give yourself overlap so you can roll back without calling someone at 3 a.m.

Rotate the User CA (or Introduce a New One) in Two Phases

Sometimes you need to rotate the CA itself—say you’re improving key hygiene, or you’re retiring an old algorithm. The two-phase dance makes it painless:

Phase A: Add. Deploy the new CA public key to every server alongside the old one. Your sshd_config ends up with two lines, or one file containing both keys:

# /etc/ssh/sshd_config
TrustedUserCAKeys /etc/ssh/ca/trusted_user_ca.pub
TrustedUserCAKeys /etc/ssh/ca/trusted_user_ca_new.pub

Start issuing new user certificates from the new CA while still honoring the old. Keep cert validity short. People authenticate as usual, and most won’t even notice the swap.

Phase B: Remove. After everyone’s certs are refreshed and you’ve run for a comfort period, remove the old CA from the servers and stop issuing certs from it. If you need to cut earlier access, you can also publish a revocation list for cert serials you want to deny before expiry.

Rotate User Keys with Overlap

User key rotation is similar. The user generates a new hardware-backed key, you sign both the old and new keys with short-lived certs, and you give them an overlap period to migrate their workflows. If they forget to update a jump host or a CI runner, the old cert covers the gap. After the overlap window, stop issuing certs for the old key, and optionally revoke the old cert serials if you need an immediate cutoff.

Rotate Host Keys Without Surprises

Host keys are trickier because clients cache host identities. The nicest pattern is to sign both the old and new host keys with your host CA and deploy both certs on the server during the overlap window. Because clients trust the CA, they accept either cert and don’t panic. When you’re confident, remove the old host key and cert and reload sshd. Most people won’t notice anything happened, which is exactly the outcome you want.

Break-Glass Access and Rollback

Have a clearly labeled, heavily restricted break-glass key. Force a specific command (maybe a one-time rotation script), limit source addresses, and keep the private key offline. Note down the rollback steps: re-add the old CA public key, restore the old host cert, and reload sshd. It sounds obvious now, but written checklists save hours when adrenaline kicks in.

Day-to-Day: Short-Lived Certs, Logs, and Health Checks

In daily life, I like 24-hour or 1-week user certs. Short-lived certs are the quiet superpower. You don’t need a mass revoke if a laptop is lost—just stop issuing certs to that key. It feels almost too simple, and that’s the beauty of it.

Logging is your early warning system. Watch for certificate expiry denials, failed principal matches, or unexpected source addresses. If you want a calm way to centralize and visualize auth logs, I’ve written about a setup that’s lightweight and friendly. Take a look at Centralized Logging on a VPS with Loki, Promtail, and Grafana—it’s a great companion for SSH hardening because you can alert on certificate errors and weird logins right away.

On the client side, I’ve found that using the URI-friendly name or a good comment helps people keep track of which key is which. Some folks prefer resident credentials on the token for portability; others keep per-machine keys without resident storage. Either path is fine—just be consistent so rotation doesn’t become a scavenger hunt.

Step-by-Step: A Practical Rollout You Can Actually Do

Let’s put it all together in a simple, repeatable plan. This is the playbook I use when a team asks for “more secure SSH” but doesn’t want downtime.

1) Prep and Baseline

– Inventory your servers and who needs access. Keep it light—usernames and the machines they touch are enough.

– Update OpenSSH on clients and servers to support FIDO2. Install libfido2 if needed. Make sure you can tap a key and see it light up.

– Tighten sshd_config defaults as shown earlier. Test by keeping one existing session open while you reload.

2) Enroll FIDO2 Keys for Your Team

– Have each person generate a hardware-backed key with ssh-keygen -t ed25519-sk -O resident -O verify-required -C "name (FIDO2)".

– Sanity test by adding the public key to a staging server’s authorized_keys and logging in. Get that “touch to authenticate” flow working comfortably.

3) Introduce the User CA

– Create a user CA and store it safely.

– Deploy the CA’s public key to all servers and set TrustedUserCAKeys.

– Sign user keys with principals and a short validity, then remove their raw pubkeys from authorized_keys after a calm overlap period.

4) Add Host Certificates

– Create a host CA, sign host keys on each server, and configure clients with a @cert-authority line in known_hosts for your domain. Hostname warnings melt away, and your team stops second-guessing TOFU prompts.

5) Build a Rotation Habit

– Put a calendar reminder to rotate user certs weekly or daily, depending on risk appetite. Short-lived certs are the best revoke you’ll never do.

– Document a two-phase CA rotation and a two-phase host key rotation. Practice in staging first. The muscle memory pays off.

6) Monitor, Alert, and Practice the Break-Glass

– Centralize logs and alert on certificate expiry denials and unusual source addresses.

– Test that your break-glass key truly works with forced commands and tight IP fences. Then put it back where it belongs—offline.

Little Gotchas and Friendly Fixes

Hardware-backed SSH keys can surprise you the first week, mostly in delightful ways. Still, let me share a few speed bumps I’ve hit and how to glide over them.

– Agents and SK keys: Traditional SSH agents don’t “hold” hardware-backed keys the way they do file-backed keys. The signing happens on the token, so you’ll touch the key when needed. This is good—don’t fight it. If you use an agent, it typically just helps with file-backed keys or cert selection.

– Resident vs. non-resident credentials: Resident storage lets the key store credentials so you can discover them on a new machine. It’s handy if you roam, but keep your token PIN safe. Non-resident is more classic—you pair a local private key wrapper to the hardware token. Both are fine; pick one and be consistent.

– Multiple tokens: I like issuing two tokens per person: a primary and a backup locked away. Sign both keys with the same principals, keep cert lifetimes short, and rotate calmly if one goes missing.

– Windows and WSL: It works, but test your flow. Sometimes it’s a matter of the right driver or ensuring your OpenSSH build has FIDO support. Yubico’s developer docs are a helpful reference if you go that route; start at Yubico’s SSH guides.

– Server-side policy: Don’t cram every rule into sshd_config. Push source restrictions and command limits into the certificate options when possible. It’s elegant because the policy travels with the identity, not the host.

What Good Looks Like (A Day in the Life)

Imagine a typical Monday. You plug in your FIDO2 key, tap to unlock the PIN, and SSH into a server. Your certificate expires in 24 hours, so on Tuesday morning your automation refreshes it with a small command or a friendly web portal that signs via the CA. Any laptop left in a taxi won’t do much damage—no cert, no access. If someone leaves the team, you stop issuing certificates for their principal. No frantic key hunts.

Your servers trust the user CA and the host CA. Your team never sees TOFU warnings. You add a new server; it starts serving a host cert signed by the CA, and everyone sails in without touching their known_hosts file. Logs are clean and centralized. If a certificate fails due to expiry, you see the alert and fix it before someone gets stuck in a deploy.

When it’s time to rotate the CA, you don’t sweat. You add the new CA to TrustedUserCAKeys, run both for a while, then remove the old one. You rotate host keys with an overlap, clients keep connecting, and your Tuesday stays boring. That’s the goal: security that blends into the furniture.

Common Questions I Hear Over Coffee

I’ve been asked a dozen variations of these. If they’re on your mind too, you’re in good company.

What if I lose the hardware key? You’ll be glad you kept cert lifetimes short. If you’ve issued certs for one day, by tomorrow that key isn’t getting in. Have a backup token enrolled, keep it safe, and you’ll be back online without a scramble.

Is Ed25519-sk better than ECDSA-sk? Don’t overthink it. I lean Ed25519 for modern defaults, but if compatibility nudges you toward ECDSA, that’s fine. Your real win is “-sk” plus certificates and a rotation habit.

Can I still use jump hosts and tunnels? Absolutely. It all works the same. You’ll just touch the hardware key when the cryptographic magic needs your sign-off. If you cache short-lived certs locally, the experience feels smooth.

Is all this extra work worth it? I’ve found it saves time within two months. The first setup takes a day or two if you’re methodical, and then rotations, revokes, and host additions become the kind of administrative chores you can do with coffee in hand.

Extra Reading and Handy References

If you want to dig deeper into the knobs and switches, the primary sources are excellent. The ssh-keygen manual covers certificates, principals, validity windows, and every obscure option you might want someday. For FIDO2 context, the FIDO2 overview helps explain why hardware-backed keys are so resilient. And if you’re using a YubiKey, the Yubico SSH developer pages have practical advice that maps neatly onto OpenSSH.

Wrap-Up: Calm, Strong, and Repeatable

If you take one thing from this, let it be this: SSH can be both simple and strong when you anchor it with hardware keys, short-lived certificates, and a rotation plan that errs on the side of boring. Start by getting a FIDO2 login working. Introduce a user CA and push trust to the servers. Add host certificates to kill the TOFU anxiety once and for all. Then make rotation a habit with a gentle two-phase rollout whenever you change something important.

Your future self will thank you the first time you need to revoke access fast or add a new server at 5 p.m. on a Friday. You’ll reach for a few well-documented steps instead of crossing your fingers. Hope this was helpful! If you want me to dive deeper into automating the CA workflow or building a self-serve certificate portal for your team, let me know—I’m happy to share what’s worked for me.