LLM Client with Spacy
Spacy will be joining Monarch and Anthony to talk about their JS/TS library, LLM Client, and how it makes it easy for developers to build LLMs.
LLM Client is an open-source TypeScript framework that abstracts over various large language models, providing a unified interface for prompting, tracing, and tuning. It supports features like prompt signatures, semantic routers, agent frameworks, and integrated tools for working with vector databases and document processing. The framework aims to make working with LLMs more efficient, consistent, and production-ready.
00:00 - Introduction and Beginning of Episode
The episode begins with Anthony Campolo introducing the guest, Spacey, and asking about their background in software development and how they got into the LLM AI space.
02:56 - Spacey's Background and Journey into LLMs
Spacey shares their background, discussing their experience founding companies, working at LinkedIn, and eventually getting into the LLM AI space through projects like 42 Papers and exploring early models like BERT.
14:37 - The Importance of BERT and Its Role in LLM Development
Anthony and Spacey discuss the significance of BERT in the history of LLM development, highlighting its generalization capabilities, bidirectional transformer architecture, and the techniques it introduced, such as pre-training and fine-tuning.
26:27 - Introducing LLM Client and Its Motivation
Spacey introduces LLM Client, discussing the motivation behind creating an abstraction layer over various LLM APIs and the challenges of working with prompts and examples in a programmatic way.
40:08 - Demonstrating LLM Client: Signatures, Prompts, and Examples
Spacey demonstrates the basic usage of LLM Client, explaining concepts like prompt signatures, the chain of thought prompting strategy, and the role of examples in guiding LLM outputs.
47:06 - Tracing and Tuning in LLM Client
The discussion moves on to tracing and tuning capabilities in LLM Client, with Spacey explaining how the framework captures input/output traces and uses them to optimize prompts and examples for better performance.
59:53 - Agent Frameworks and Semantic Routers in LLM Client
Spacey showcases additional features of LLM Client, such as the agent framework for building complex workflows and the semantic router for efficient request routing based on embeddings.
67:27 - Running LLMs Locally and the Hermes Model
The conversation shifts to running LLMs locally, with Spacey discussing the Hermes model by Nous and its capabilities for function calling and JSON output.
70:01 - Working with Documents and Vector Databases in LLM Client
Spacey demonstrates how LLM Client integrates with tools like Apache Tika for document processing and provides abstractions for working with vector databases for efficient retrieval and querying.
75:44 - Using Traces to Improve LLM Performance
The discussion revisits the topic of traces, with Spacey explaining how captured traces can be used to improve LLM performance by providing high-quality examples for fine-tuning and optimizing prompts.
77:17 - The Future of LLM Client and Closing Thoughts
Spacey shares their vision for LLM Client as an open-source project, discussing potential contributions, community building, and the possibility of a renaming. The episode concludes with final thoughts from Anthony and Monarch.
Guest
https://github.com/dosco
/ dosco
/ vikramr
Links and Resources
https://github.com/dosco/llm-client
https://dosco.github.io/llm-client/
https://en.wikipedia.org/wiki/BERT_(l...)
https://x.com/42papers
https://arxiv.org/pdf/2310.03714
https://github.com/stanfordnlp/dspy
https://platform.openai.com/docs/guid...
https://unsloth.ai/
https://github.com/unslothai/unsloth
https://arxiv.org/pdf/2301.00234
https://arxiv.org/abs/2201.11903
https://github.com/dosco/llm-client/b...
• What's next for AI agentic workflows ft. A...