Validate Common Records Between Two Tables Using iceDQ: Prevent Overlapping Data
In today’s video, we demonstrate how to validate that there are no overlapping records between two tables: the permanent employee table and the temporary employee table. The goal is to ensure that there are no employees who are both temporary and permanent, and we achieve this by using iceDQ's powerful reconciliation rule.
We walk through the process of comparing these two tables using the employee ID as the primary join condition, and then we focus on finding the intersection (common records) between the two datasets.
Key Highlights
Finding Common Records: Use iceDQ’s reconciliation rule to check for overlapping records between two tables.
Intersection of Records: Focus on identifying the intersection (common records) between permanent and temporary employee tables.
Efficient Data Validation: Ensure that there is no data overlap between the two datasets, using diff join conditions for better accuracy.
Data Investigation: Explore the results to see which records are common in both tables and easily investigate discrepancies.
With iceDQ, you can automate data validation, ensuring that your ETL processes, data migration, and data integrity remain intact.
Ready to automate your data validation?
Request a demo today and see how iceDQ can help you streamline your data validation and improve your data quality.
0:00 - Introduction: Validating Unique Employee Records
0:15 - Understanding the Use Case: No Overlap Expected
0:33 - Creating the Reconciliation Rule
0:55 - Setting Up Source and Target Tables
1:40 - Defining Join Conditions and Checks
2:03 - Running the Rule and Analyzing Results
2:23 - Investigating Common Records and Final Thoughts
Request a Demo: https://icedq.com/request-a-demo
-------------------------------------------------
About iceDQ: Ensuring Reliable Data From Development to Production with iceDQ.
iceDQ is a one-stop platform for data reliability with unified data testing, monitoring, and observability. Large banks, insurance, healthcare, and other enterprises rely on iceDQ in both development and production environments, ensuring data reliability and robust processes.
Streamlined Data Testing in Development: iceDQ is used to automate data migration testing, ETL data pipeline testing, big data lake testing, BI report testing, and more. It helps identify and fix data issues early in the data development lifecycle.
Proactive Monitoring and Observability in Production: iceDQ is used by operations to establish checks and controls for their data pipelines, and the AI-based observability engine ensures anomalies are detected and incidents are reported.
-------------------------------------------------
Request a Demo: https://icedq.com/request-a-demo
Data Testing: https://icedq.com/product/data-testin...
Data Monitoring: https://icedq.com/product/data-monito...
Data Observability: https://icedq.com/product/data-observ...
Data Reliability: https://icedq.com/data-reliability-en...
LinkedIn: / icedq
Facebook: / icedq.toranainc
X: https://x.com/iceDQ_Toranainc
Reddit: / icedq
-------------------------------------------------
Don't forget to like this video, subscribe to our channel for more informative content, and hit the notification bell to stay updated with our latest uploads. Thank you for watching.
#iceDQ #DataValidation #ETLTesting #DataReconciliation #DataIntegrity #AutomatedTesting