Deploying a LLM Chatbot using BigDL-LLM and Llama2 on an Intel Laptop
Guoqiong Song & Jiao (Jennie) Wang, Intel
Due to the ongoing expansion of large language models (LLMs), leading to performance degradation, heightened memory requirements, and increased computational demands, there is a growing urgency for efficient quantization to compress LLMs into a more compact form. (https://stackoverflow.blog/2023/08/23.... Additionally, optimization across platforms is pivotal for enhancing the accessibility of LLMs. BigDL-LLM (https://github.com/intel-analytics/BigDL) is designed to make efficient LLM development more accessible all Intel platform users, spanning from CPUs to GPUs, from clients to the cloud.
BigDL-LLM is an open source library designed to run large language models (LLMs) using low-bit optimizations (FP4/INT4/NF4/FP8/INT8) on Intel XPU , for any PyTorch model with very low latency and small memory footprint. BigDL-LLM incorporates a variety of low-bit technologies including llama.cpp, gptq, bitsandbytes, qlora, and more. With bigdl-llm, users can build and run LLM applications for both inference and fine-tuning, using standard PyTorch APIs (e.g., HuggingFace Transformers and LangChain) on Intel platforms. Meanwhile, a wide range of models (such as LLaMA/LLaM2, ChatGLM2/ChatGLM3, Mistral, Falcon, MPT, Dolly/Dolly-v2, Bloom, StarCoder, Whisper, InternLM, Baichuan, QWen, MOSS, etc.) have already been verified and optimized on bigdl-llm.
The presentation will walk the audience the process of optimizing a Llama 2 model unitizing the BigDL-LLM library, and offers a practical session on deploying a chatbot through Llama2 on an Intel laptop. Subsequently, A detailed walkthrough of the material will be covered as part of a broader workshop on LLM Agents. We invite everyone to join us in exploring this exciting journey with the Intel BigDL-LLM.
SPEAKER BIOs
Jiao (Jennie) Wang is a AI Framework Engineer on the Machine Learning Platform team at Intel working in the area of AI and big data analytics. She is key contributor in developing and optimizing distributed ML/DL framework and provide customer support for end-to-end AI solutions.
Guoqiong Song: An AI Frameworks Engineer at Intel, with a focus on building end-to-end AI applications within the AI Software Engineering department. She has a PhD degree in Atmospheric and Oceanic Sciences from UCLA, specializing in quantitative modeling and data analysis. Previously, Guoqiong worked at Verizon as a data scientist.
/ guoqiong-song-903aa759
https://www.meetup.com/sf-bay-acm/eve...
(Minor audio issues in the beginning)
0:00 Chapter Intro
1:40 Speaker Intro
3:00 Presentation
35:22 BigDL-LLM demo
1:01:16 Q&A