Ten million patient records is the point where architectural shortcuts stop being technical debt and start being regulatory exposure. This is the blueprint we landed on after eighteen months running a HIPAA-regulated microservice estate on Kubernetes, and the three decisions we would make again without hesitation.

Encryption-at-rest is a storage-class decision, not an application one

Application-layer encryption is where most teams start, and it is where most teams get stuck: key rotation becomes a deploy, every new service re-implements the envelope, and audit evidence lives in code review comments. We pushed encryption down to the CSI driver instead, backing every PersistentVolumeClaim with a KMS-managed key scoped per namespace.

The practical consequence is that a namespace is the compliance boundary. A service that cannot mount a volume outside its namespace cannot leak PHI across a tenant boundary, and the evidence an auditor wants is a Kubernetes RBAC export rather than a code walkthrough.

Audit logging belongs in the data path

Sidecar log shippers drop records under load, and "best effort" is not a phrase that survives a 45 CFR 164.312 review. We moved audit emission into the request path: every handler writes its audit event inside the same transaction that mutates patient state, and a change-data-capture stream forwards those rows to immutable storage.

  • Audit writes share the transaction boundary with the mutation they describe.
  • CDC replication is idempotent and replayable from any offset.
  • Retention is enforced by the storage tier, not by an application cron.

Zero-trust mesh without the sidecar tax

We replaced a sidecar-per-pod service mesh with eBPF-based enforcement through Cilium. Identity is derived from the pod's service account, mTLS terminates in the kernel, and the policy surface is a set of CiliumNetworkPolicy resources that live alongside the deployment manifests they protect.

At 10M audit events per second the sidecar overhead had been costing roughly 18% of cluster CPU. Moving enforcement into the kernel recovered most of that, and — more importantly — removed an entire class of "the sidecar was not ready yet" startup races from the incident log.

What we would do differently

Start the third-party penetration test earlier. Ours passed, but the findings that mattered were architectural, and architectural findings are cheap in month three and expensive in month fifteen.