🔍 Harnessing the Power of Airflow: S3 File Sensor + Talend Integration Demystified!
In this video, we are going to explore Apache Airflow's powerful FileSensor.
Explore how this sensor can revolutionize your workflow management system by monitoring the presence of crucial files in a specified directory.
We'll guide you through a real-life example where we use the FileSensor to wait for the arrival of a success file, signaling the completion of data ingestion.
This video also explains , how to easily integrate Talend Jobs with Airflow ..
UserData Script:
------------------------------
https://github.com/SatadruMukherjee/D...
Airflow Dag Code:
--------------------------------
https://github.com/SatadruMukherjee/D...
Snowflake Query:
----------------------------------
--drop database if exists
drop database if exists s3_to_snowflake;
--Database Creation
create database if not exists s3_to_snowflake;
--Use the database
use s3_to_snowflake;
--Table Creation
DROP TABLE IF EXISTS s3_to_snowflake.PUBLIC.Iris_dataset1;
create table s3_to_snowflake.PUBLIC.Iris_dataset1 (Id number(10,0),sepal_length number(10,4),sepal_width number(10,4),petal_length number(10,4) ,petal_width number(10,4),class varchar(200));
--create the file format
create or replace file format my_csv_format
type = csv field_delimiter = ',' skip_header = 1
field_optionally_enclosed_by = '"'
null_if = ('NULL', 'null')
empty_field_as_null = true;
--create the external stage
create or replace stage s3_to_snowflake.PUBLIC.Snow_stage url="s3://{}/{}/"
credentials=(aws_key_id=''
aws_secret_key='')
file_format = my_csv_format;
show stages;
list @s3_to_snowflake.PUBLIC.Snow_stage;
select * from s3_to_snowflake.PUBLIC.Iris_dataset1;
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engineering with Cl...
Apache Kafka form scratch
• Apache Kafka for Python Developers
Messaging Made Easy: AWS SQS Playlist
• Messaging Made Easy: AWS SQS Playlist
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
https://doc.clickup.com/37466271/d/h/...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY YOUTUBE CHANNEL