The Failover HSM That Cannot Sign: Mechanism Drift in Post-Quantum Key Clusters
Most distributed HSM outages that are hard to diagnose start with a cluster where every member holds the key and not every member can use it. The replication layer copies the wrapped private key object faithfully, the health check reports green, and the standby partition runs firmware that does not implement the ML-DSA or hybrid mechanism the key was created for. Nobody notices until the primary fails, and then failover moves traffic to a node that answers every signing request with a mechanism-invalid error.
Why review never catches it
The change that introduces the fault looks harmless in a pull request: a key policy file adds a post-quantum signing profile, names the cluster as the target, and sets the replication factor to cover every member. The reviewer checks what the diff shows, which is intent, and the intent is correct. Nothing in the diff encodes which firmware each member runs, because that state lives in the appliance fleet, outside the repository, and drifts on its own schedule as vendors ship PQC support in staggered releases. Replication succeeds because copying an encrypted key blob does not require the receiving module to understand the algorithm inside it. Module validation schemes such as FIPS 140 do not help either, since a validated module that lacks a mechanism is still validated, just for less.
What the failure looks like at 3 a.m.
Your cluster health probe almost certainly does a login, a slot enumeration and maybe a classical ECDSA test signature, so every member passes while the post-quantum keys sit unusable on half of them.
A detection step you can run today
Point pkcs11-tool at each member individually, using its list mechanisms option against the production slot, sort the output, and diff the results across members; any mechanism present on the primary and absent elsewhere is a failover that will break. Then go one level deeper, because a matching mechanism list does not prove the key is usable: pin the client library to one member at a time and sign a fixed canary message with each production key ID, checking that the signature verifies against the published public key. Run this canary on a schedule, every 15 minutes is a reasonable cadence for signing keys, and alert on the first member that fails rather than on cluster aggregate status. The cost is real, since each canary consumes HSM operations and generates audit log entries that your compliance team will ask about, so tag canary keys or canary messages clearly and budget the load. My estimate is that the mechanism diff alone catches most of these incidents, while the per-member canary catches the rest, such as attribute mismatches where a replica imported the key without the signing permission set.
The rule that ends the repeat incident
Treat a key as highly available only when every member of its failover set has passed a signed canary with that exact key and mechanism, and make your key provisioning pipeline refuse to mark a new profile as production until that proof exists. This moves the check from human review, which sees policy, to automation, which sees the fleet. It also forces an uncomfortable decision during PQC migration: either you hold new algorithm profiles back until the slowest member is upgraded, or you shrink the failover set for those keys and accept that a post-quantum key has less redundancy than its classical twin. Neither is free, but a smaller failover set you know about is safer than a large one that exists only in the dashboard.
Technical FAQ: Can't the HSM vendor's own cluster health check detect this?
Usually not, because vendor health checks confirm membership, replication state and sync status, not whether each member can run every mechanism referenced by every key it holds, so you still need a per-member canary that uses the real key and the real mechanism.