SOLAR 10.7B: Scaling LLMs with Depth Up-Scaling
SOLAR 10.7B is a large language model (LLM) with 10.7 billion parameters, demonstrating superior performance in various natural language processing (NLP) tasks. Inspired by recent efforts to efficiently up-scale LLMs, SOLAR has a new method for scaling LLMs called depth up-scaling (DUS), which encompasses depthwise scaling and continued pretraining. In contrast to other LLM up-scaling methods that use mixture-of-experts, DUS does not require complex changes to train and inference efficiently. DUS is simple yet effective in scaling up high-performance LLMs from small ones. SOLAR 10.7B-Instruct is a variant fine-tuned for instruction-following capabilities, surpassing Mixtral-8x7B-Instruct. SOLAR 10.7B is publicly available under the Apache 2.0 license.
In this video, I will talk about the following: Why does SOLAR 10.7B matter? What is Depth-Up Scaling? How is SOLAR trained?
For more details, please look at https://arxiv.org/pdf/2312.15166.pdf and https://huggingface.co/upstage/SOLAR-...
Kim, Dahyun, Chanjun Park, Sanghoon Kim, Wonsung Lee, Wonho Song, Yunsu Kim, Hyeonwoo Kim et al. "Solar 10.7 b: Scaling large language models with simple yet effective depth up-scaling." arXiv preprint arXiv:2312.15166 (2023).