Apple's OpenELM models
OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of ~1B parameters, OpenELM exhibits a 2.36% improvement in accuracy compared to OLMo while requiring 2× fewer pre-training tokens. Diverging from prior practices that only provide model weights and inference code, and pre-train on private datasets, OpenELM release includes the complete framework for training and evaluation of the language model on publicly available datasets, including training logs, multiple checkpoints, and pre-training configurations. They also release code to convert models to MLX library for inference and fine-tuning on Apple devices.
In this video, I talk about the following: What is OpenELM’s architecture and how is it trained? How does OpenELM perform?
For more details, please look at https://arxiv.org/pdf/2404.14619v1 https://github.com/apple/corenet https://huggingface.co/apple/OpenELM
Mehta, Sachin, Mohammad Hossein Sekhavat, Qingqing Cao, Maxwell Horton, Yanzi Jin, Chenfan Sun, Iman Mirzadeh et al. "OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework." arXiv:2404.14619 (2024).