Vector search at 510M embeddings holding p99 under 40ms

The constraint

Retrieval looked fine on a two-million-vector staging index and fell apart at production scale. The team had been tuning for latency without holding recall fixed, so every improvement was quietly trading away answer quality.

What we changed

We fixed recall@10 at 0.95 and treated it as non-negotiable, then tuned index parameters, memory layout, and query concurrency against that constraint.

• Recall pinned before any latency work began
• Index rebuild budgeted as a standing operational cost
• Query concurrency capped to protect the tail

Where it landed

510 million embeddings serving at a p99 of forty milliseconds, with recall held at the agreed floor.