Production-scale RAG for Real-Time News Distillation - Robert Caulk | Vector Space Talks
🎤 Meet our guest speaker:
Robert Caulk, Founder at Emergent Methods
Title: "Production-scale Retrieval Augmented Generation for Real-Time News Distillation"
Abstract:
This talk aims to map out the cluster architecture at Emergent Methods for our AskNews.app deployment. We aim to explain how we process over 1 million news articles per day for fast real-time RAG consumption. In particular, we will discuss the roles and interactions of four open-source software packages, including a cluster orchestration software (Flowdapt), a vector database (Qdrant), an LLM server (vLLM), and an embedding server (TEI). This combination of services allows us to infuse a mixture of LLMs with the latest news, and to deploy an enriched chat to thousands of users simultaneously in a public-facing application. We will highlight the role of Qdrant for some of its indispensable features, including efficient batch upserting, fast search on millions of vectors, easy filtering, and multi-node scaling. Finally, we will discuss the key roles that Flowdapt plays for orchestrating production-scale RAG across a variety of services in real-time, including the importance of configurability, scheduling, and resource management.
We will conclude with our thoughts on why we believe the incorporation of these tools at the foundation level of a start-up is a huge advantage in comparison to the big tech companies from the 2010s.
Talk Takeaways:
How we process over 1 million news articles by combining Flowdapt, Qdrant, vLLM, and TEI.
Why Qdrant is indispensable for production-scale RAG.
How Flowdapt improves deployability, configurability, scaling, scheduling, and resource management.
Why new start-ups have a huge tech-stack advantage over 2010 era tech companies.
Robert's Bio:
Robert is a scientist by trade, dedicating his career to a variety of open-source projects that range from large-scale artificial intelligence to discrete element modeling. He is currently working with a team at Emergent Methods to adaptively model over 1 million news articles per day, with a goal of reducing media bias and improving news-awareness.
--------------------------------------------------------------------------------
Connect with Robert and Emergent Methods:
/ rcaulk
https://emergentmethods.ai
https://asknews.app
Connect with Demetrios:
/ dpbrinkm
Follow Qdrant:
/ qdrant
/ qdrant_engine
Join the Qdrant Discord server:
/ discord