Cloud Functions - How to Read PDF Files on GCS Events and Store in BigQuery
In this tutorial, you will learn "How to create an event-driven Cloud Function that reads PDF files from Google Cloud Storage (GCS) and pushes their contents into BigQuery" in GCP.
Google Cloud Functions offer a serverless environment to run event-driven code, making them suitable for various data engineering use cases.
Topics:
0:00 🌿Introduction and Overview
0:34 🌿Roles and Permission
1:14 🌿Cloud Function Configuration & Trigger
3:01 🌿Runtime, Build, Connections and Security Settings
3:40 🌿Runtime Environment Variables
5:00 🌿Cloud Function Code
5:28 🌿Supportive Python Packages/Libraries
7:25 🌿Elaborate Code Line By Line
15:25 🌿Deployment, Testing and Quality Checks
Ensure that you have appropriate permissions for accessing GCS and BigQuery in your Cloud Function's service account.
🌿Roles and Permission
👍Cloud Functions Invoker: to execute Cloud Function
👍Service Account User: to interact with other Google Cloud services
👍BigQuery Data Editor: to edit data in BigQuery datasets
👍Storage Object Admin: to read, write, update, and delete on GCS Bucket
🌿Cloud Function Configuration & Trigger
Trigger Type, Event Type, Bucket, Retry on failure
🌿Runtime, Build, Connections and Security Settings
Memory Allocated, CPU, Timeout, Concurrency, Autoscaling
🌿Runtime Service Account
🌿Runtime Environment Variables
🌿Cloud Function Code
🎯Runtime Language and Entry Point
🎯Supportive Python Packages/Libraries
🚀PyPDF2 - To interact with PDF files
🚀google-cloud-storage - to interact with GCS
🚀google-cloud-bigquery - to interact with BigQuery
🚀Read PDF File from GCS bucket as a Byte Format
🚀Convert Byte Object to File Like Object
🎯Conditions
🚀Verify the Source Directory Path
🚀Verify the File Extension
🎯Reading PDF File from GCS
🚀Download file as Byte Format
🚀Call PDF Convert Method
🚀Get the PDF Data as Text in Json Array List
🚀Deleting PDF file from GCS Bucket
🎯Load Data Into BigQuery
🚀Initializing client variable from BigQuery
🚀Create Schema variable to meet BQ Table
🚀BigQuery Job Configuration
🚀Load BigQuery Job
🌿Save and Deployment and Testing-
🎯Upload PDF files into GCS Bucket,
🎯Log Check and Verification
🎯Verify the data in BigQuery Table
⭐To learn more, please follow us -
http://www.sql-datatools.com
⭐To Learn more, please visit our YouTube channel at -
/ sql-datatools
⭐To Learn more, please visit our Instagram account at -
/ asp.mukesh
⭐To Learn more, please visit our twitter account at -
/ macxima
⭐To Learn more, please visit our Medium account at -
/ macxima