How to build your own ChatGPT with custom knowledge base using OpenAI, Pinecone, LangChain, Estuary
!!!IMPORTANT: There has been a change since this video was made.
In the Python script (@11:00), replace:
text_field = "input"
with:
text_field = "flow_document"
Leave a comment if there's any issue!
***
This video teaches how to build your own free ChatGPT with custom knowledge base within minutes, using Estuary, OpenAI, Pinecone, and Langchain.
Scripts used in the video can be found here:
https://www.estuary.dev/chatgpt-custo...
🏄🏻♂️ If you're interested in Surfshark, sign up via this link for an exclusive discount and free months:
https://get.surfshark.net/aff_c?offer...
🍺 If you've learned something new from this video, buy me a coffee to support this channel =)
https://bmc.link/jennyman
0:00 Intro
0:28 ChatGPT Gaps
1:37 Solution
2:10 Prerequisites
3:14 Use Case Ideas
4:51 Set up input
5:20 Set up Estuary Flow Capture
6:42 Pinecone Materialization
6:51 Create Pinecone Index
8:51: Break time
10:03 Python
11:51 Testing
#chatgpt #chatgpt4 #chatgptprompt #ai #artificialintelligence #data #dataengineering #dataengineeringessentials
============================================================
Large Language Models (LLMs) like ChatGPT are great for answering questions that are covered by the datasets they have been trained on, but what if you want to use them for more recent data, or data that is very specific to your business that wasn't part of their training dataset? Asking these kinds of questions will inevitably lead to incomplete or inaccurate responses.
A technique called Retrieval Augmented Generation can be used to solve this problem. This involves passing additional related documents to the LLM along with the question itself. The LLM then uses that related information along with its trained "understanding of the world" to produce a response. Vector databases like Pinecone allow for storing and retrieving related documents based on their vector embeddings, and in this article we'll see how how to use Estuary Flow's Pinecone Materialization Connector to build a real-time data pipeline for AI using retrieval augmented generation.
We'll build our pipeline step-by-step, using an example scenario to illustrate the problem and how to solve it.