PySpark Interview Question: Calculate Total Transaction Amount for Each Customer
In this PySpark interview question, we explore how to calculate the total transaction amount for each customer using PySpark. We'll walk through the code implementation step-by-step and discuss the key concepts involved. This question assesses your understanding of PySpark DataFrame operations, data aggregation, and basic data manipulation. Watch the video to learn how to solve this problem in PySpark
from pyspark.sql import SparkSession
from pyspark.sql.functions import sum
Create a SparkSession
spark = SparkSession.builder \
.appName("TotalTransactionAmount") \
.getOrCreate()
Read the dataset
data = spark.read.csv("path/to/dataset.csv", header=True)
Convert TransactionAmount column to float
data = data.withColumn("TransactionAmount", data["TransactionAmount"].cast("float"))
Calculate the total transaction amount for each customer
total_amount = data.groupBy("CustomerID").agg(sum("TransactionAmount").alias("TotalAmount"))
Show the result
total_amount.show()
Stop the SparkSession
spark.stop()