NO MORE NULLS - how to handle missing values in Spark DataFrames + Fabric (Day 10 of 30)
10+ hours of FREE Fabric Training: https://www.skool.com/microsoft-fabri...
Learn Apache Spark in Microsoft Fabric in the 30 days of September.
Here's the playlist for this series if you want to catchup: • Learn Apache Spark in Fabric in 30 days
Link to the GitHub for this series: https://github.com/LearnMicrosoftFabr...
Imputer documentation: https://spark.apache.org/docs/latest/...
Spark is the engine behind both the Data Engineering AND the Data Science experiences in Microsoft Fabric, so in September I'll be walking you through Apache Spark: what it is, why you should learn it, how to use it, and how it integrates into Microsoft Fabric.
No previous Spark knowledge is required, some basic Python would be useful!
#pyspark #microsoftfabric #apachespark
Here's the schedule:
1 Welcome - • DAY ONE - Learn Apache Spark in Micro...
2 Why Spark? • Why you should learn Spark in Microso...
3 Components of Spark - • FIVE components of the Spark engine i...
4 Spark DataFrame - • Introducing the Spark DataFrame in Mi...
5 Read Files into DataFrame - • How to read CSV, JSON, PARQUET into S...
6 Read/Write to Lakehouse Table - • How to write data to Lakehouse Table ...
7 Basic DataFrame Operations - • Learn THESE important pySpark operati...
8 DataFrame Filtering - • 14 ways to filter your Spark DataFram...
9 GroupBy and Aggregate Functions - • How to use GROUPBY and AGG functions ...
10 Handling missing values - • NO MORE NULLS - how to handle missing...
11 Joining DataFrames - • How to get better insights by JOINING...
12 Time-series data - • Analyzing DATES and TIMES in Spark in...
13 Spark SQL - • Using your SQL knowledge in Spark in ...
14-16 Spark Machine Learning - • Spark Machine Learning End-To-End Pro...
20 Configuring Spark - • Customising Fabric Runtime and Spark ...
21 Autotuning Spark Configuration - • Autotuning your Spark config in Micro...
22 Library Management - • Manage your Python packages (the old ...
23 High-concurrency mode - • Running multiple notebooks on same se...
24 Spark Scala - • Writing Spark Scala in Microsoft Fabr...
25 Fabric MSSparkUtils - • Powerful notebook utility functions (...
26 Monitoring Spark - • Monitoring Spark (errors and cost) in...
30 FINALE QnA - • How to use Spark in Microsoft Fabric ...
Timeline
0:00 Intro
0:22 Why handle missing values?
2:14 Getting some data
3:03 Simple drop null values
3:38 How parameter
4:34 Threshold parameter
6:15 Only on a subset of columns
7:16 Filling missing values
9:55 Mean Imputation of missing values
12:24 Summary
-BROWSE MY OTHER FABRIC PLAYLISTS-
DATA ENGINEERING • Data engineering (Microsoft Fabric)
END-TO-END FABRIC PROJECT • Playlist
INTRO TO MICROSOFT FABRIC • Intro to Microsoft Fabric
DATA FACTORY • Data Factory (Microsoft Fabric)
-LINKEDIN-
Not following the LinkedIn page yet? Here's the link: / learnmicrosoftfabric
-ABOUT WILL-
Hi, I'm Will! I'm hugely passionate about data and using it to create a better world. I currently work as a Consultant, focusing on Data Strategy, Data Engineering and Business Intelligence (within the Microsoft/Azure/Fabric environment). I have previously worked as a Data Scientist. I started Learn Microsoft Fabric to share my learnings on how Microsoft Fabric works and help you build your career and build meaningful things in Fabric.
-SUBSCRIBE-
Not subscribed yet? You should! There are lots of new videos in the pipeline covering all aspects of Microsoft Fabric.
/ @learnmicrosoftfabric