Data ingestion pipeline with built-in error handling
A step-by-step case study of a data ingestion pipeline built in CloverDX to onboard client data to a SaaS platform, with support for logging and retries of the ingest pipeline, so you can re-run easily (without needing the dev team).
The pipeline in this case needed to:
Handle client data in a variety of formats
Land the data as it comes in to some staging table in a relational database
Handle ingestion without any knowledge of incoming file formats
Inspect incoming files and detect what the structure is
Use that structure to actually read and load the data into a data lake
And then make retries simple - if something fails, we can quickly make adjustments and re-run
This clip is taken from a longer video: Making Data Ingestion Faster, More Reliable and Easier to Scale. Watch the whole video here: https://www.cloverdx.com/ty/webinar/d...
** Read the related blog post: https://www.cloverdx.com/blog/making-...
** More on data ingestion with CloverDX: https://www.cloverdx.com/solutions/da...
** Request a demo: https://www.cloverdx.com/demo/