SCD Type1 with joins Implementations in azure databricks
The idea of slowly changing dimensions often pertains to data warehousing and database management. In this context, "slowly changing dimensions" (SCD) refer to the changes that occur in dimensional data over time.
There are typically three types of slowly changing dimensions:
Type 1: In this type, the old data is simply overwritten with the new data. No history is maintained.
Type 2: Here, a new record is added to the dimension table to represent the new data, while the old record is retained, often with a surrogate key and an effective date range. This allows for historical analysis.
Type 3: In this type, both the old and new values are kept in the same dimension table, usually in separate columns. This allows for a compromise between storage space and historical analysis.
In a Type 1 SCD, changes to the dimension attribute simply overwrite the existing data. This means that the old data is replaced with the new data, without any historical tracking. As a result, there is no history preserved for the dimension attribute. This method is straightforward and easy to implement but doesn't allow for historical analysis.
Example:
Consider a customer dimension table where the "Customer Name" attribute is updated from "John" to "Jonathan". In a Type 1 SCD, the existing record for John would be directly updated to reflect Jonathan without retaining any information about the previous name.