"Store this in your library" works better than "remember this". here's why | Sam Partee
Sam Partee (CTO + co-founder of Arcade) sat down with Mateo to walk through the design choices behind Agent Library — the open-source local-first agent memory we released last week. The conversation starts where the project started.
This is less of a launch pitch and more of a design interview: why the word "library" outperforms "memory" with modern agents, why the canonical RAG pipeline (BM25 + vector + RRF + MMR) is actually the right floor for local memory, and the one place where doing the math correctly produced the wrong product behavior.
In this episode, we cover:
Why naming it "library" (with sections, chapters, books, librarian) gets a higher success rate from agents than "memory"
"Semantic saturation" — when fine-tuning collides with overloaded words like "memory" and tool-calling gets worse
The Obsidian origin story: a typo that accidentally let the agent organize its own knowledge graph
Why agent-organized libraries beat human-curated ones once the corpus is big
The actual hybrid search recipe: BM25 + vector with reciprocal rank fusion, plus MMR for diversity
Local-first by default: every model (text embed, code embed, CLIP, OCR, generation) running on a 32GB MacBook Pro
Read-mostly by design: why deletes require a `--clobber` flag and why that matters for notes
Local context (Agent Library) vs remote context (Arcade tools to Google Docs, Drive, etc.) — when to use which
The hardest technical problem: weighting multimodal results when code, text, and images all "win" differently
Why the code embedding ended up off by default in multimodal search
What Sam wants from open-source contributors (vector DB swaps, SLM benchmarks, issues from real users)
A tease for the next open-source release that uses Agent Library as its memory building block
⏰ TIMESTAMPS
00:38 – Welcome and intro
01:03 – Origin story: from an Obsidian tool to Agent Library
04:18 – Why "library" semantics work better than "memory"
07:13 – Hybrid search and the RAG pipeline (BM25, RRF, MMR)
09:54 – Daily use cases and personal workflow
13:18 – Local vs remote context: Agent Library vs Arcade
16:23 – Why SQLite and the fully local stack
19:24 – Open-source contributions and future roadmap
22:42 – Hardest technical challenge: multimodal search weighting
25:29 – What users find most useful, and wrap-up
🔗 LINKS & RESOURCES
Agent Library
GitHub (Apache 2.0): https://github.com/ArcadeAI/agent-lib...
Install: `uv tool install agent-library`
Referenced in this episode
Thinking, Fast and Slow (the "tier one / tier two" book Sam alludes to): https://en.wikipedia.org/wiki/Thinkin...
Obsidian: https://obsidian.md/
sqlite-vec: https://github.com/asg017/sqlite-vec
SQLite FTS5: https://www.sqlite.org/fts5.html
BM25 / TF-IDF: https://en.wikipedia.org/wiki/Okapi_BM25
Reciprocal Rank Fusion (RRF) paper: https://doi.org/10.1145/1571941.1572114
Maximal Marginal Relevance (MMR): https://www.cs.cmu.edu/~jgc/publicati...
CodeBERT: https://huggingface.co/microsoft/code...
Tiangolo / FastAPI (the contributor model Sam aspires to): https://github.com/tiangolo/fastapi
Companion launch episode
Agent Library: Own Your Agent's Memory in a Single SQLite File: • How to setup Agent Library to own your Age...
About the guest
Sam Partee, CTO + Co-founder at Arcade
LinkedIn: / sampartee
GitHub: https://github.com/Spartee
X: https://x.com/SamPartee
🚀 Try Arcade: https://account.arcade.dev/register?u...
Looking for the fastest way to build MCP servers?
https://docs.arcade.dev/en/home/custo...
💬 Join the conversation: / discord
🔧 Build with Arcade: https://arcade.dev
📖 Arcade Docs: https://docs.arcade.dev
#AgentMemory #AgentLibrary #RAG #HybridSearch #SQLite #LocalFirst #AIAgents #ArcadeDev