End-to-End Data Engineering with Pandas | API to Postgres ETL | SQL Automation | Pipeline | Database
#dataengineering #automation #python #etl
Learn how to perform end-to-end ETL using Python Pandas.
In this demo, you will learn how to extract data from APIs using Python libraries, Transform the data using Python Pandas, and Load the data into the Postgres server using the psycopg2 library.
00:00 - Introduction
01:20 - How to Query #restapi using python requests library and extract response text.
02:45 - How to write incoming data into files using the Python open method.
03:40 - How to convert a raw file containing python #dictionary into #pandas dataframe.
05:10 - How to slice and visualize a subset of pandas dataframe using sample.
05:26 - Slicing pandas dataframe and selecting column data.
06:2- - How to transform pandas dataframe. Select specific columns and apply filters to perform string manipulation and comparison.
07:36 - Pandas inbuilt functions to validate data (count, unique and unique)
08:15 - How to reset the dataframe index and set the rows sequentially.
09:50 - how to Sort pandas dataframe column values alphabetically in ascending order.
11:50 - how to apply multiple arithmetic operations on pandas columns to compare and filter data.
13:20 - how to dynamically and securely read secrets using Python configparser.
14:00 - How to establish Postgres connection from Python using psycopg2 connection string.
14:25 - Create a programmatic cursor to connect and execute SQL statements on the database from Python.
15:15 - How to create a Postgres database table.
17:37 - How to iteratively read pandas dataframe as tuple and extract the elements to insert into SQL table.
LET'S CONNECT!
🐦 Gumroad➔ https://databracket.gumroad.com/
📰 LinkedIn ➔ / jayachandra-sekhar-reddy
📖Medium ➔ / jay-reddy
📲 Substack➔ https://databracket.substack.com
💁Fiverr ➔ https://www.fiverr.com/jayreddy9
#bigdata #bigdatatutorialforbeginners #python #automation #dataanalytics #data #cloudstorage