Boost your ETLs with Data in The Fast Lane - DataFriday 1x23
Extract, transformations, and load (ETL) operations can become quite complex. In this short video, I'll go over what is an ETL process and how you can simplify those processes with a new tool: Data in the Fast Lane (DFL).
On 10/19/2020, Advance Auto Parts announced the Advance Open Source Garage, and part of this announcement, a new tool has been released. Data in the Fast Lane (DFL) is an easy-to-use ETL based on Apache Spark. See it as a way to script Spark.
I will first remind everyone of what is an ETL, then I will go over two demos: the first demo simply displays the content of a CSV file as well as some other information like the schema and the number of records. The second demo will take two CSV files and upload them in a PostgreSQL database.
DFL leverages simple recipes written in YAML, allowing you to easily write data pipelines.
Data in the Fast Lane can be downloaded from GitHub: https://github.com/AdvanceAutoParts/d....
Demo #1:
GitHub link: https://github.com/AdvanceAutoParts/d...
Demo #2:
GitHub link: https://github.com/AdvanceAutoParts/d...
Advance Open Source Garage is on Twitter at @AdvanceOSS. More information and documentation are coming to / advanceoss .
This event is supported by La French Tech Raleigh-Research Triangle ( / frenchtechraleigh .
#DataFriday #LifeInAdvance