OpenVINO GenAI API inferencing
Install openvino for conda
https://docs.openvino.ai/2023.3/openv...
VStudio 2022 with C++ extensions
https://visualstudio.microsoft.com/th...
genAI API OpenVINO
https://github.com/openvinotoolkit/op...
Make sure you follow the same way
(Optional) Clone OpenVINO GenAI repository if it does not exist
git clone --recursive https://github.com/openvinotoolkit/op...
cd openvino.genai
Install python dependencies
python -m pip install ./thirdparty/openvino_tokenizers/[transformers] --pre --extra-index-url https://storage.openvinotoolkit.org/s...
python -m pip install --upgrade-strategy eager -r ./samples/requirements.txt
optimum-cli export openvino --model "TinyLlama/TinyLlama-1.1B-Chat-v1.0" --trust-remote-code "TinyLlama-1.1B-Chat-v1.0"
Python script
import openvino_genai as ov_genai
model_path = r"C:\openvino.genai\TinyLlama-1.1B-Chat-v1.0"
pipe = ov_genai.LLMPipeline(model_path, "CPU")
print(pipe.generate("The Sun is yellow because", max_new_tokens=100))