Where Retrieval Stops Being Enough for Agent Memory
Two failures that show up in every retrieval system once an agent has to remember across sessions, and the hybrid pipeline I built to work around them.
I have spent the last couple of years building retrieval for agent memory, and the most useful thing I have learned is where retrieval stops being enough. Similarity search is a good default and a bad ceiling. If you are building anything that has to remember across sessions, you reach that ceiling sooner than you expect.
The first failure is that relevance and recency pull against each other. A pure vector search will happily hand back something perfect from eight months ago while missing what happened yesterday. This is not an embedding quality problem. Better vectors do not fix it, because the failure lives in the ranking objective rather than in the representation. You asked for the most similar thing when what you wanted was the most useful thing, and those are not the same question.
The second failure is that a flat index has no notion of what supersedes what. Facts get revised. Decisions get reversed. A plain index returns both versions with equal confidence and leaves the model to sort it out. Agents are especially bad at this, because they will act on the stale version without ever noticing a newer one existed.
What I ended up building runs on Postgres. Lexical search through tsvector, dense search through pgvector, the two result sets fused with Reciprocal Rank Fusion, then the top candidates reranked with a BGE-M3 cross-encoder. The fusion step matters more than it sounds. Lexical search catches exact identifiers and rare terms that embeddings smooth away, dense search catches paraphrase, and RRF combines them without forcing you to tune a weight between two score scales that were never comparable in the first place.
Better retrieval quality still did not solve memory on its own. Underneath the search I put summary pyramids: rolled-up summaries at turn, day, week, and month granularity, each embedded and indexed. A question about last Tuesday lands on the day layer. A question about how a project evolved lands on the month layer. All of it is indexed against an ltree topic taxonomy, so you can traverse structurally instead of only searching by similarity. That combination is what finally produced year-old recall worth having.
I should be honest about the limit. I built that taxonomy by hand. It works, and hand-built structure beats no structure, but it does not scale past one person's discipline, and it models where a fact sits in a tree rather than how facts relate to each other. A real graph-native store with typed edges and proper invalidation would do better on both counts. That is the next problem, and I do not think retrieval by itself gets you there.