Changing Column Types in Spark using cast and withColumn | Data Type Conversion | Data Engineering
Hello Everyone, Welcome to my channel :
Directly connect with me on : https://topmate.io/mantukumardeka
My Essential Gear items:
Camera :- https://amzn.to/3ZN02PB
Microphone:- https://amzn.to/3DtN166
Tripod:- https://amzn.to/41PnA9d
Lighting Kit:- https://amzn.to/3ZN7oCS
Gimbal/Camera Stabilizer: -https://amzn.to/3Bt3CXi
External Monitor:- https://amzn.to/4gMeXjZ
Macbook:- https://amzn.to/3VMYcgD
Backdrop/Green Screen:- https://amzn.to/3BJ0438
Best Mobile:- https://amzn.to/403aOmj
My PC Components:-
intel i7 Processor:- https://amzn.to/3P6so2i
G.Skill RAM:- https://amzn.to/4grWdGD
Samsung SSD:-https://amzn.to/49RFMRq
WD blue HDD:- https://amzn.to/3DpB3uh
RTX 3060Ti Graphic card:- https://amzn.to/41MTlQ9
Gigabyte Motherboard:- https://amzn.to/3BShxpL
Printer InkJet: https://amzn.to/4iG0cRw
Printer InkTank:- https://amzn.to/49U5t3V
Others:
RO Purifier: https://amzn.to/3ZQs6BS
Best TV : https://amzn.to/3BK7VgP
geyser 8+ litre :- https://amzn.to/3VPaghn Changing Column Types in Spark usinf cast and withColumn | Data Type Conversion
package Pack
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql._
import org.apache.spark.sql.types._
import org.apache.spark.sql.functions._
import org.apache.spark.sql.expressions._ //WINDOW FUNCTION
object objnove24 {
def main(args:Array[String]):Unit={
println("Hello Guyes")
val conf = new SparkConf().setAppName("first").setMaster("local[*]").set("spark.driver.host", "localhost")
.set("spark.driver.allowMultipleContexts", "true")
val sc = new SparkContext(conf)
sc.setLogLevel("ERROR")
val spark = SparkSession.builder.getOrCreate()
import spark.implicits._
val data =Seq(
("A","2024-09-01", "100"),
("A","2024-10-01","150"),
("A","2024-11-01","170"),
( "B","2024-09-01","100"),
( "B","2024-10-01","115"),
("B","2024-11-01","185")
)
val df=data.toDF("transaction_id", "date", "sale")
df.show()
df.printSchema()
val castdf=df.withColumn("sale",col("sale").cast("Integer"))
castdf.show()
castdf.printSchema()
val castdf1=castdf.withColumn("date", col("date").cast("Date"))
castdf1.show()
castdf1.printSchema()
}
}
=============Python=========
df.withColumn('column', df['column'].cast('new_type'))
===========-Scala=======
val castdf1=castdf.withColumn("date", col("date").cast("Date"))
SEARCH QUERIES:
pyspark tutorial for data engineers
what is spark in data engineering
apache spark for data engineering
spark tutorial data engineer
learn python for data engineering
data engineering on microsoft azure
kafka tutorial for data engineer
data engineering architecture interview questions
python for data engineering
advanced python for data engineering
spark interview questions for data engineer
spark architecture in big data
data engineering life cycle
kafka data engineering project
how to start data engineering career
spark projects for data engineer
data engineering project using pyspark
data pipeline in data engineering
how much python is needed for data engineer
python libraries for data engineering
data engineer scenario based interview questions
data engineer scenario based interview questions
data engineering coding interview questions
databricks data engineering associate
data engineer roles and responsibilities
data flow modeling in software engineering
apache airflow tutorial for data engineer
senior data engineer interview questions
fundamentals of data engineering masterclass
data engineer system design interview questions
#DataEngineering, #ApacheSpark, #Scala, #PySpark, #BigData, #Python, #ETL, #DataScience, #SparkSQL, #DataPipeline, #ScalaSpark, #MachineLearning, #CloudComputing, #SparkStreaming, #BigDataTools