Pyspark for sql developers - statistical functions - Part 1
Hello and welcome back to the series PySpark for SQL Developers,
where we explore how to use PySpark to perform data analysis and manipulation tasks that are common in SQL.
In the previous video, we learned how to perform joins in PySpark and SQL Server and compare similarities and differences between them.
In this video, we will focus on pyspark statistical functions and how they compare with the equivalent functions in SQL Server.
Statistical functions are useful for computing descriptive statistics,
such as standard deviation, correlation, and so on.
They can help us understand the distribution, variability, and relationship of our data.
We will also learn how to use the summary method to get a quick overview of the statistics of a DataFrame.
Finally, we will compare the syntax and output of pyspark statistical functions with the corresponding functions in SQL Server, and highlight the similarities and differences between them.
By the end of this video, you will have a better understanding of how to use pyspark statistical functions and how they relate to SQL Server functions. You will also be able to apply these functions to your own data and gain insights from them.
00:00 - Introduction
02:00 - Requirements
08:00 - The Standard Deviation
14:00 - The Variation Function
24:00 - The Correlation Function