Conversion of Data Frames | Spark to Pandas & Pandas to Spark
In this video, I will show you how to convert a Spark dataframe to Pandas dataframe and pandas dataframe to Spark dataframe. In addition this, we also see how to sample a large dataframe and pass custom schema while reading/converting a pandas df to spark df.
Link to the playlist "Getting started with PySpark" : • Getting started with PySpark
Link to "Setting up the PySpark environment on Google Colab" video: • Setting up the PySpark environment on Goog...
Link to the GitHub repo: https://github.com/Abhishekmamidi123/...
Check out my "Data Science guide for freshers and enthusiasts" playlist: • My path to becoming a Data Scientist | Abh...
I have put my 3 years of learning experience into this playlist.
Contents of this video:
00:00 - Setting up the PySpark environment
00:40 - Initialize Spark Session object
01:02 - Read data from UCI into Google Colab
02:51 - Convert Spark Dataframe to Pandas Dataframe
03:46 - Sampling a Spark Dataframe
05:18 - Convert Pandas Dataframe to Spark Dataframe
06:21 - Pass Schema while reading Spark Dataframe
07:54 - Summary and Subscribe :)
Please do like, share and subscribe to this channel and share this video with your friends. Keep learning :)
Follow me here:
LinkedIn: / abhishekmamidi
Blog: https://www.abhishekmamidi.com/
GitHub: http://github.com/Abhishekmamidi123
Kaggle: http://www.kaggle.com/abhishekmamidi
Tags:
abhishek mamidi, data science, machine learning, deep learning, artificial intelligence, internship, career, college, job, experience, krish naik, ai engineering, fresher, data science enthusiasts, pyspark, apache spark, python, pysparkling