Challenges with unsupervised LLM knowledge discovery
"Challenges with unsupervised LLM knowledge discovery" is a recent research paper by S. Farquhar et al. This video walks through the paper and highlights some of the key results and implications.
Timestamps:
00:00 - Challenges with unsupervised LLM knowledge discovery?
01:41 - Eliciting Latent Knowledge
02:34 - Contrast-Consistent Search (CCS)
05:29 - Critique of CCS
07:26 - Theoretical results
09:55 - Experiment: Discovering random words
11:24 - Experiment: Discovering an explicit opinion
12:11 - Experiment: Discovering an implicit opinion
13:30 - Experiment: Prompt template sensitivity
14:10 - Experiment: Agreement between unsupervised methods
14:58 - Does the criticism extend to future methods?
The paper can be found here: https://arxiv.org/abs/2312.10029
The CCS paper can be found here: https://arxiv.org/abs/2212.03827
Topics: #elk #ai #ccs
For related content:
Twitter: / samuelalbanie
personal webpage: https://samuelalbanie.com/