How to upload Pandas DataFrame to GCP Bucket with Python script
This video demonstrates how to upload Pandas DataFrame to Google Cloud Console (GCP) storage bucket. For this purpose we will use a very simple Python script, which uses the following dependencies:
Pandas: pip install pandas(https://pandas.pydata.org/docs/)
Google Cloud Storage: pip install google-cloud-storage (https://pypi.org/project/google-cloud...)
Also, we will need to create a Service Account dedicated to communicate between our Python script and GCP bucket. For this Server Account me must to apply Storage Admin privileges. Furthermore, you need to create a new JSON based key which will be specified in your Python script. Use IAM dashboard in GCP to do it.
The Python script itself defines the main logic how our Pandas DataFrame is setup before uploading to the GCP bucker. There you must to specify:
The the JSON key you generated in GCP IAM dashboard.
Bucket name (created earlier).
Source data file (in this example we are using CSV data file, where our Pandas DataFrame is saved on).
Output directory and filename.
Python script template we used in the tutorial: https://github.com/vb100/pandas-to-gc...
The content of the tutorial:
0:00 - Intro
0:27 - Create a new Google Cloud Bucket
1:05 - Create a new Service Account
2:13 - Create new JSON key for Service Account
2:40 - Create a new Python Virtual Environment
3:18 - Install dependencies on Virtual Environment
4:19 - Create a simple Pandas DataFrame
5:06 - Set-up a Python script for GCP
7:17 - Fix the Bucket name error
7:45 - Test the Python script
8:20 - Outro
See you on the next video!
#python #googlecloud #pandas