Building automated data profiling into your data workflows for better data quality
Building data profiling steps into your data pipelines can help prevent processing suspect data, and can also help you detect trends in data or data quality decay over time.
In this data workflow built in CloverDX we can see the number of files flowing through the process at each step, as well as being able to drill into individual records.
In this example we're simply checking for null records. If we're getting less than 10% null records we'll continue processing, but when we hit that threshold we log an error and alert the user.
Pulling those results in to a spreadsheet shows is the data quality is trending in the wrong direction, so you can address the errors even before they reach that critical status - fixing data before it even fails.
Clip from the full video - Data validation in data ingestion: https://www.cloverdx.com/ty/webinar/d...
Request a demo: https://www.cloverdx.com/demo/
Start a free trial of CloverDX: https://www.cloverdx.com/trial/
Follow us on LinkedIn: / cloverdx