The BEST Way to Chunk Text for RAG
To try everything Brilliant has to offer—free—for a full 30 days, visit https://brilliant.org/AdamLucek/ You’ll also get 20% off an annual premium subscription!
Resources:
Chunking Notebook: https://github.com/ALucek/chunking-st...
ChromaDB Technical Report: https://research.trychroma.com/evalua...
ChromaDB Report Repo: https://github.com/brandonstarxel/chu...
OpenAI Token Visualizer: https://platform.openai.com/tokenizer
Greg Kamradt 5 Levels of Text Splitting: https://github.com/FullStackRetrieval...
Jaccard Index: https://en.wikipedia.org/wiki/Jaccard...
Chapters:
00:00 - Background on Text Chunking
02:28 - Brilliant!
03:47 - Character Text Splitting
06:28 - Token Text Splitting
10:26 - Recursive Character/Token Splitting
16:07 - Kamradt & Modified Semantic Chunking
20:43 - Cluster Semantic Chunking
24:46 - LLM Semantic Chunking
27:56 - Chunking Metrics & Comparison
30:00 - Overall Findings
#ai #programming #datascience
This video is sponsored by Brilliant