Dagster Deep Dive - Stop Writing Prompt Spaghetti: Production LLM Engineering with DSPy & Dagster
Transform your LLM development from chaotic prompt strings to production-ready, self-optimizing pipelines.
In this Dagster Deep Dive, Alex Noonan and Colton Padden demonstrate how to escape the "prompt spaghetti" trap that plagues most LLM applications. Learn how combining DSPy's declarative framework with Dagster's orchestration creates maintainable, testable, and automatically optimized AI systems.
What You'll Learn
Why Traditional LLM Development Fails
The problems with hard-coded prompts and how they fall outside the software development lifecycle. Why LLM applications need proper orchestration just like data pipelines.
DSPy Fundamentals
Signatures that define what you want without specifying how. Modules as building blocks for language model calls. Optimizers for automatic prompt tuning. Type-safe inputs and outputs using Pydantic models.
Dagster for Production
Asset-centric orchestration for LLM pipelines with built-in observability and metadata tracking. Dagster Components for declarative pipeline setup. Moving models from development to production.
Real-World Demo: NY Times Connections Puzzle
Complete walkthrough using actual game data. Model evaluation and optimization workflows comparing performance across different LLM providers.
Key Takeaways
Automatic prompt optimization without manual tuning. Production-ready pipelines with full observability. Complete model lineage tracking. Continuous evaluation workflows. Self-healing systems with proper error handling.
Timestamps
0:00 Introduction: The Problem with Prompt Spaghetti
2:00 Why LLM Development is Like Data Engineering
4:00 Introducing DSPy: Declarative LLM Framework
7:00 DSPy Core Abstractions (Signatures, Modules, Optimizers)
9:00 Dagster Components & Asset-Centric Approach
11:00 Automatic Optimization Deep Dive
12:00 Demo: NY Times Connections Puzzle
17:00 Code Walkthrough: Modeling the Game
20:00 Dagster Asset Factory & Components
23:00 UI Demo: Observability & Metadata
25:00 Evaluation Results Across LLM Providers
27:00 Q&A and Wrap-up
Resources
Full code example available in the Dagster examples repository: https://docs.dagster.io/examples/dspy
Complete evaluation results across all major LLM providers: https://pedramnavid.com/blog/dspy-evals/
Join our Community Slack to try Dagster Compass, our AI data analyst agent: https://dagster.slack.com/archives/C0...
GitHub: dagster-io/dagster
Website: https://dspy.ai/
#Dagster #DSPy #LLM #MachineLearning #DataEngineering #AI #MLOps #Python