Benchmark embedding models #4 - Generate question-answer pairs from text chunks
In this video, you'll learn how to generate question-answer (QA) pairs from text chunks.
This video is part 4 of the "Benchmark Embedding Models" series. I'll walk you through the entire process of using Large Language Models (LLMs) to create a high-quality QA dataset from a document we chunked in the previous video.
First, I'll show you how to use a proprietary model, Google's Gemini, to generate questions. We'll build a detailed system prompt that instructs the model on its role, the required JSON output format, and provides clear examples to ensure relevant, high-quality results.
To guarantee the Gemini API returns valid, structured data, we'll define a Pydantic object. I'll then show you the Python code to load our text chunks from a JSON file, loop through each one, and call the Gemini API to generate questions. We'll see how to link each generated question back to its original chunk_id and handle potential API rate limits.
Next, I'll demonstrate how to achieve the same result using a powerful open-source model, Qwen3-30B-A3B-Instruct-2507, running locally. We'll find the quantized GGUF model on Hugging Face and serve it using llama.cpp's llama-server. I'll show you the exact terminal command to get the server running.
Here is the link to the GitHub repository: https://github.com/ImadSaddik/Benchma...
Don't forget to like, subscribe, and leave a comment if you have any questions or feedback!
⭐ Contents ⭐
(00:00) Introduction
(02:02) The GitHub repository containing the source code
(02:25) The first notebook that uses Gemini
(09:30) Switching to the open-source model (Qwen) notebook
(09:59) Finding the Qwen GGUF model on Hugging Face
(10:53) Starting the local llama-server with llama.cpp
(11:57) Adapting the code to use the llama.cpp server endpoint
(15:32) Conclusion