Docker for Data Engineering: Postgres, Docker Compose, and Real-World Workflows - Alexey Grigorev
In this session, Alexey Grigorev walks through essential concepts and practical workflows that power modern data engineering. Drawing from real world project experience, he demonstrates how Docker, Python, Postgres, and efficient data processing techniques form a unified toolkit for building reliable data pipelines and production ready environments.
You will learn about:
Foundational Docker concepts including containerization, isolation, stateless design, and image management
Practical use of core Docker commands and volume mapping for persistent data
Building custom images with Dockerfile and configuring automated execution with ENTRYPOINT
Running and managing Postgres inside containers with correct port exposure for host access
Managing Python environments with UV for reproducible development
Processing CSV and mixed type data using Pandas
Strategies for optimized ingestion with chunked reading
Creating command line interfaces with Click
Solving container connectivity through Docker networks and Docker Compose
Links:
Telegram: https://t.me/dezoomcamp
Workshop: https://github.com/alexeygrigorev/wor...
Course: https://github.com/DataTalksClub/data...
TIMECODES:
00:00 Docker and Python prerequisites for the workshop
09:33 Core definition of Docker containerization
11:42 Docker container isolation and stateless behavior
12:12 First Docker command testing Docker run hello world
14:45 Understanding stateless containers and Docker image snapshots
20:46 Data persistence using Docker volume mapping
28:01 Data engineering overview for building data pipelines
38:17 Python environment management with UV
44:16 Creating a custom Docker image with a Dockerfile
48:08 Setting the ENTRYPOINT for automated container execution
56:30 Running Postgres in Docker for database management
1:01:36 Host access to Postgres via container port mapping
1:17:01 Pandas data processing for CSV schemaless data types
1:24:51 Optimized data ingestion with chunked reading
1:36:50 Building a command line interface with Click
1:48:19 Resolving container communication using Docker networks
1:59:39 Multi container orchestration with Docker Compose
This talk is ideal for aspiring and intermediate data engineers, machine learning practitioners, analysts transitioning to engineering roles, and software engineers who want a practical introduction to container based workflows and reproducible data processing environments.
Connect with DataTalks.Club:
Join the community - https://datatalks.club/slack.html
Subscribe to our Google calendar to have all our events in your calendar - https://calendar.google.com/calendar/...
Check other upcoming events - https://lu.ma/dtc-events
GitHub: https://github.com/DataTalksClub
LinkedIn - / datatalks-club
Twitter - / datatalksclub
Website - https://datatalks.club/
Connect with Alexey
Twitter - / al_grigor
Linkedin - / agrigorev
Check our free online courses:
ML Engineering course - http://mlzoomcamp.com
Data Engineering course - https://github.com/DataTalksClub/data...
MLOps course - https://github.com/DataTalksClub/mlop...
LLM course - https://github.com/DataTalksClub/llm-...
Open-source LLM course: https://github.com/DataTalksClub/open...
AI Dev Tools course: https://github.com/DataTalksClub/ai-d...
👉🏼 Read about all our courses in one place - https://datatalks.club/blog/guide-to-...
👋🏼 Support/inquiries
If you want to support our community, use this link - https://github.com/sponsors/alexeygri...
If you’re a company, reach us at [email protected]
#docker #python #postgres #dataengineering #dockercompose #dockernetworks #pandas #clicklibrary #uvpython #containerization #datapipelines #softwareengineering #datascience #codingtutorial #devops #cloudengineering #programming #etl #bigdata #datatalksclub