How to Remove Duplicates in Python | Data Cleaning with Pandas 🐍📊
Simple rule:
duplicated() = Find duplicate data
drop_duplicates() = Remove duplicate data
Find duplicates
df.duplicated()
Show duplicate rows
df[df.duplicated()]
Remove duplicates
df.drop_duplicates()
Remove duplicates based on a column
df.drop_duplicates(subset=["Employee"])
Keep the last occurrence
df.drop_duplicates(subset=["Employee"], keep="last")
Remove ALL records that have duplicates
df.drop_duplicates(subset=["Employee"], keep=False)