🚿Clean data beats fancy algorithms🚿
Let’s break down the key aspects of how to keep your data clean:
✅ Validity:
This is about how well your data sticks to the rules.
Data-Type Constraints: Each column needs to have values in the correct format, like numbers or dates.
Range Constraints: Values should be within a certain range—like a date of birth that makes sense.
Mandatory Constraints: Some fields just can’t be left blank.
Unique Constraints: Some data needs to be unique, like a social security number.
Set-Membership Constraints: Values should come from a predefined list, like “male” or “female” for gender.
Foreign-Key Constraints: If you’re referencing another table, the data needs to match up correctly.
Regular Expression Patterns: Certain fields, like phone numbers, need to follow a specific format.
Cross-Field Validation: Related fields should make sense together, like a discharge date that’s after the admission date.
✅ Accuracy:
This is about how close your data is to the truth. Just because data is valid doesn’t mean it’s accurate. For example, a valid street address might not actually exist. Also, being precise is important; “Earth” as an address is true, but not very helpful.
✅ Completeness:
How much of the necessary data is actually there? Missing pieces are inevitable, but you can try to fill in the gaps by double-checking with your sources.
✅ Consistency:
This checks if your data tells the same story throughout. Inconsistencies show up when data contradicts itself, like a 10-year-old listed as “divorced.”
✅ Uniformity:
Data should be in the same format across the board. For instance, weights should all be in pounds or kilograms—not a mix of both. Dates and currencies should also follow a consistent format.
Anything to add?
#data #dataengineer #dataanalytics #datascience #bigdata