Blazing Fast Local LLM Web Apps With Gradio and Llama.cpp
In this video, we'll run a state of the art LLM on your laptop and create a webpage you can use to interact with it. All in about 5 minutes. Seriously!
We'll be using Llama.cpp's python bindings to run the LLM on our machine and Gradio to build the webpage.
Resources mentioned in the video:
Llama.cpp python: https://github.com/abetlen/llama-cpp-...
Gradio: https://github.com/gradio-app/gradio
Qwen-2 0.5B Instruct Model (GGUF): https://huggingface.co/Qwen/Qwen2-0.5...
Llama.cpp's chat completion API: https://github.com/abetlen/llama-cpp-...
Gradio Chatbot Guide: https://www.gradio.app/guides/creatin...