Video
PALO is a Large Multilingual Multimodal Model. PALO offers visual reasoning capabilities in 10 major languages, including English, Chinese, Hindi, Spanish, French, Arabic, Bengali, Russian, Urdu, and Japanese, that span a total of ~5B people (65% of the world population). PALO involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model, thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi, Arabic, Bengali, and Urdu. The resulting models are trained across three scales (1.7B, 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines.
In this video, I talk about the following: What is PALO? How is PALO trained? How does PALO perform?
For more details, please look at https://arxiv.org/pdf/2402.14818.pdf and https://github.com/mbzuai-oryx/PALO
Maaz, Muhammad, Hanoona Rasheed, Abdelrahman Shaker, Salman Khan, Hisham Cholakal, Rao M. Anwer, Tim Baldwin, Michael Felsberg, and Fahad S. Khan. "PALO: A Polyglot Large Multimodal Model for 5B People." arXiv preprint arXiv:2402.14818 (2024).