Run Local Llm On Raspberry Pi 5 With A Single Executable Llamafile!
First, starting early April, Raspberry Pi OS comes with Vulkan driver support natively, I use tinyllamafile f16 and Q8_0 to run sample Q&A. The results coming from the set up of raspberry pi 5 is different from what Justine showed. I hope I can do something to make the throughput (tokens/sec) run as fast as what Justine showed.
https://justine.lol/matmul/