EDA: exploratory data analysis
Enroll in the full Machine Learning in Python course at [email protected]
EDA, or Exploratory Data Analysis, is typically performed simultaneously with data acquisition and cleaning and largely determines how exactly the data should be cleaned. Essentially, EDA is the identification of statistical features in the data, such as normal distribution, heterogeneity, correlations, or outliers, which may lead to model simplification or the need for additional data cleaning.
Exploratory data analysis involves several key steps:
1. Missing Data
Gaps in the data prevent machine learning methods from fully exploiting the dataset: in this case, partially missing data must either be excluded from the sample, or missing values must be somehow "calculated"—replaced with boundary values or interpolated from adjacent data.
In any case, the data that machine learning models will operate on must be free of gaps. All values in all data tuples must be defined. Therefore, the first step is to check for gaps in each data series.
2. Distribution Parameters
After resolving data issues, you can move on to the next step: assessing the sample distribution for normality, searching for anomalies (outliers), and identifying any features (e.g., dominant values, bimodality, or a specific distribution pattern).
Information about anomalies, such as the 1st and 99th percentiles, highest and lowest values, can be taken into account when cleaning the data if this will improve the quality of the model. Also, in some cases, the distribution pattern determines the mean estimate (this may not be the arithmetic mean, but the median or mode) for use in models.
3. Data Correlation
An important part of exploratory data analysis is identifying relationships (correlations) between both independent parameters and between the predicted (dependent) value and the independent parameters.
The presence of correlations allows you to select a set of parameters for constructing a first approximation of the model and evaluating different models based on the most significant parameters. Thus, properly conducted exploratory data analysis can significantly reduce the time spent searching for the optimal model.
4. Data Examples
At the conclusion of exploratory data analysis, one or more data tuples are typically examined as a whole to understand the situation. Sometimes, certain data features may go unnoticed during the first three steps but become apparent when examining several data examples.
The results of exploratory data analysis are often presented in the form of a series of graphs—histograms, correlation matrices, or scatter plots—that display the characteristic features of the data and the data itself.