Data fidelity using ADB Property Graph (Lineage) and GenAI RAG models
In this Autonomous Database Learning Lounge session, the Oracle Data Services, Graph PM and Autonomous Database PM teams explained how the Oracle Data Catalog service analyzes Data Fidelity using Autonomous Database Property Graph (Lineage) and GenAI RAG models.
Register for our bi-weekly sessions and watch recordings at https://bit.ly/adbll-main-y
Slides for this session at: https://bit.ly/adbll-slides-datafidelity
Outline
Data Fidelity is a critical factor for any data-driven business. It is characterized by accuracy , relevancy , consistency, timeliness and completeness of Data. Decision makers rely on these quality metrics of data to understand their business trends, exploit opportunities and to mitigate risks.
Melliyal Annamalai, Distinguished Product Manager and Ganesh Seetharaman, Architect, Oracle Data catalog explained how data lineage harvested across the data real estate , property graphs in Autonomous Database and GenAI RAG models are leveraged in building a framework to analyze fidelity of data used across applications.
Data Catalog and Governance platforms help organizations to maintain data fidelity through well defined access policies, traceability and audit logs, curation and collaborative tools. As businesses deploy more AI/ML models in their decision making systems, it becomes imperative to address the inherent opacity of these systems by providing details on their reasoning process. Among other things , substantiating the use of high fidelity data is a key requirement for model explainability.
Video highlights:
00:38 Agenda
01:40 Important links on Autonomous Database
02:22 What are Graphs?
05:13 Graph use cases
06:10 Represent your data as Graph
06:16 Graph Studio in Autonomous Database
08:09 Data Fidelity using ADB Property Graph (Lineage) and RAG models
09:30 Data Fidelity definition
14:32 Data Catalog use case
18:05 Data Lineage and Property Graph
22:50 Data Lineage: typical usage
24:10 Data Lineage: harvesting - how do we get lineage metadata
25:46 Data Lineage analysis - how we consume lineage metadata today
26:12 Data Lineage: the need for property graph
33:34 To summarize: property graph = knowledge graph
33:44 Data Lineage using Property Graph
36:02 Leveraging Lineage Metadata for Data Fidelity analysis using Search and RAG
39:06 Benefits of using Lineage Graph for context infusion in RAG
41:49 Lineage Visualization
42:30 Lineage Properties used in Data Fidelity
42:36 Missing Data - Fidelity
44:26 Compliance - Fidelity
46:03 Generating code based on lineage pipeline metadata
47:15 Q&A
47:34 Q&A Can we ingest live data streams into the service?
48:16 Q&A Can the Data Lineage differentiate between AI Generated content and non-AI content?
49:30 Q&A Is everything that you showed in this demo Generally Available?
50:06 Q&A Do we gather pipeline information from services like AWS Glue? Does the pipeline have to call Data Catalog with the OpenLineage REST API?
51:30 Q&A What are the plans for including the Golden Gate data dictionaries and all its pipelines?
51:53 Q&A How is the lineage graph used for ranking documents in a RAG system?
54:11 Important links on Autonomous Database
54:30 Final thoughts