Finding the Right Datasets and Metrics for Evaluating LLM Performance
Evaluation is one of the more difficult challenges to bringing an LLM-based application to production. Traditional evaluation approaches require a labeled dataset representative of the data you’ll see in production. Because labeled datasets are hard to come by, many teams settle for using a handful of manually created examples or rely on off-the-shelf academic datasets that rarely align with their specific production use case.
In this 30-minute webinar, we’ll discuss the tradeoffs and risks of evaluating LLMs using labeled data and unlabeled data. For supervised (labeled) evaluation, we'll compare traditional metrics on off-the-shelf academic datasets versus small manually-created tailored examples. These techniques should be combined with evaluation approaches that work on unlabeled data that we'll compare: performance estimation techniques and unsupervised evaluation using semantically meaningful metrics, both appropriate for production data.