LaMini-LM: A diverse herd of distilled models from large-scale instructions
Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, one can perform knowledge distillation from instruction-tuned LLMs into much smaller ones. To this end, the authors carefully develop a large set of 2.58M diverse instructions based on both existing and newly-generated instructions. Extensive analysis of the instruction dataset confirms its diversity, and responses for these instructions are generated using gpt-3.5-turbo. Leveraging these instructions, a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes, is fine-tuned. Performance of these models is evaluated using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The model is also assessed for hallucination and toxicity, and for the former, a new benchmark dataset for hallucination-inducing QA is introduced. LaMini-LM models are comparable to strong baselines while being much smaller in size.
In this video, I talk about the following: What is LaMini-LM? How is the instruction set generated? How does LaMini-LM perform? How much do LaMini-LM models hallucinate?
For more details, please look at https://arxiv.org/pdf/2304.14402
Wu, Minghao, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Fikri Aji. "Lamini-lm: A diverse herd of distilled models from large-scale instructions." arXiv preprint arXiv:2304.14402 (2023).