Local Tool Calling with llamacpp
Tool calling allows an LLM to connect with external tools, significantly enhancing its capabilities and enabling popular architecture like agents. But, tool calling with local LLMs has been a challenge. Now, in part driven by the release of Llama 3, we've seen more efforts to fine-tune local LLM to support tool use. We've recently updated our integration with Llama.cpp, a popular library for working with local LLMs, to support tool calling and structured outputs. Here we quickly walk through its usage with a fine-tuned 8b parameter llama3 model (from Nous Research).
Documentation:
https://python.langchain.com/v0.2/doc...