In the previous post, I introduced the idea of class imbalance; a problem in which the target distribution of a binary classification problem was at 95:5, or even 99:1. This presents challenges in developing performant ML models, because the model can sometimes struggle to create an effective decision boundary (see here, Fig. 1) because the minority class contributes relatively little to the definition of the decision boundary. As a result, models may not be as accurate as we want them to be, for the metrics we care about.
To counter this, practitioners often rebalance the dataset so the model sees and even(ish) number of the minority and majority class, and thus is able to learn the distinction between each class and arrive at a more accurate model. We may undersample the majority class, oversample the minority class, or generate synthetic data through complex methods. These techniques can be effective in some circumnstances, and are the standard responses to class imbalance.
However, class imbalance isn't really the fundamental problem, and more often it's a symptom of a greater issue: limited observation of the process that generates the minority class in the first place. Rebalancing modifies the data the model is exposed to, but it does not create new information about the phenomenon we are trying to model, and ultimately, predict.
In this post, we'll examine the popular sampling techniques and see how they fare.
Random Sampling: Undersampling
This seems the most intuitive thing to do at the outset, it’s fairly simple, and seems like it can solve our problem quickly. Here, we sample from the Majority class until the classes are balanced in our dataset. One obviously problem here is we can significantly reduce the size of our dataset, depending on the degree of class imbalance.
If we have a dataset of 1000 observations, but at a 1:9 minority:majority class imbalance, and we Undersample, it reduces the total dataset size to 100 (old minority) + 100 (new “majority”); 200, which means discarding 80% of the original data. As such, some consider this approach to be throwing away data, but that depends on your unique situation, and how "redundant" the Majority class is (how variable observations in the Majority class tend to be).
Random sampling: Oversampling
Here we duplicate records from the minority class until class balance is reached, or is less sever, This could mean entirely duplicating the minority class a few times, depending on our class imbalance.
If we take our above dataset of 1000 rows and a class imbalance of 1:9 then we effectively duplicate every minority class 9 times in order to balance the classes. Minority classes thus have a greater influence on model training, but, no new information is shown to the model; it sees the minority class more often but doesn't see anything new. This can lead to overfitting.
SMOTE (Synthetic Minority Oversampling TEchnique)
Here, we don’t just randomly sample, rather, we try and sample sophisticatedly, making assumptions about how the data are generated.
Under-and Over-sampling may seem very crude methods to create balance in the dataset. A more sophisticated version of class rebalancing is SMOTE. Instead of simply under/over-sampling, we create new minority class datapoints. The mechanics of SMOTE are better described elsewhere, but effectively, the method works like this: we pick a datapoint in the minority class, find its nearest neighbours in n-dimension space, and sample from some point along that line. We do this until we have created enough new samples to balance the classes. Implicitly, we assume that the features in our dataset lie on what's called a "locally-smooth segment" of the feature space, meaning that we expect the distribution of the features in the n-dimensional space to approximately take the form of a smooth "plane" in n-dimensions, and additionally, that all our minority class observations are clustered around one point in this plane, and it is from this region we sample and generate new minority class observation. If neither of these assumptions are true, SMOTE won't generate useful synthetic data, and as a result, the model performance will degrade.
Cluster-based SMOTE
As its name suggests, cluster-based oversampling assumes clusters of minority class records exist in clusters or segments along the feature space, and not in one continuous cluster. We relax the assumption around the smooth plane in the feature space here, and we assume that the minority class occupies clusters within that space, rather than being localised to a single point. The approach is two-fold: firstly, identify local clusters of minority class observations using something like k-means, and then perform SMOTE within these clusters to generate synthetic observations that preserve local structure while increasing class balance.
Let’s use some real data and give this a bash and see if we can get the data to behave using the above methods. We’re using data from the well-known German Credit dataset, where the target is imbalance at a 70:30 ratio (good:bad credit rating), with 20 already-engineered features. Whilst this isn’t terrible class imbalance, it’s sufficient to show the differences in each data sampling approach. We'll first run a simple XGB classification model on the data and treat the resulant model as the "baseline model". Thereafter, run the same XGB model, but applied to rebalanced datasets using each of the abovementioned methods, and compare the metrics to better understand how each method performs.
In the table below, the "_0" subscript refers to the "bad credit" group, and the "_1" to the "good credit" group. Our model's objective is to learn the distinction between the two groups, but, we could assume that identifying a bad credit person is more valuable (in terms of monetary loss) than a "good" person. So here, I've broken the metrics up into how well it models each class. Recall_0 is probably the most important measure here (out of all "bad credits" how many did we get right), followed by Precision_0. I've gone with a 70:30 training:test data split.
Table 1. Performance metrics of a standard XGB model, applied to datasets created by using each one of the data-sampling methods mentioned above, including a baseline model (no class-imbalance correction.)
Our baseline model does a pretty good job at predicting "Good credit" (Recall_1 & Precision_1), but doesn't do a good job of predicting "Bad Credit", and from our Recall_0 score, we see that we correctly identify just less than 50% of all "Bad Credit" cases (recall is the proportion of total bad credit cases our model correctly identifies). In general, employing class rebalancing techniques improves this metric, but at the expense of the other (particularly Recall_1), except for Cluster SMOTE, which doesn't meaningfully improve on the baseline model. Whilst the exact specifics of which model performed better at what metric are interesting, the general take-home message here is that no class rebalancing method significantly improved upon all metrics from the baseline model. Simply put: rebalancing the classes in the dataset increased some metrics, whilst decreased others. Effectively, sampling gave us trade-off options, rather than a single, well-performing model across all metrics.
After using the sampling techniques, why don’t we see increases in all metrics? It seems intuitive that if the baseline model shows good scores for good credit precision and recall, and we over sample the minority class, a new model applied to these data should not only retain the good credit scores but then also improve scores for the bad credit metrics? The reason for this isn’t really related to sampling methods, but more to do with data-generating processes: this is the process behind the events that generate the data we record and use in our models.
We can treat the process that generates the data we record as unknown, but constant, and that forms a huge distribution in n-dimensions (20 in this case, as we have 20 features). So, in our case, we have some conditional distribution P(X∣Y), where 𝐗 is the vector of our features, but we think of these as distributed on a “manifold” (or "plane") of the features in 20 dimensional space, and Y is the target (“good credit, bad credit”). We know this is theoretically present, but, we can’t observe it. In practice, our model only observes sparse regions of that entire distribution, and it is our model’s job to learn the mappings from X→Y. Class imbalance only becomes a problem when the minority class occupies regions in that distribution, but are poorly represented in our dataset (i.e., we only partially observe the full distribution of minority class in the 20-d space.)
Over-or-under-sampling does not reveal new regions of P(X∣Y); it simply reinforces observations we have already seen. This may strengthen existing decision boundaries, but it does not improve coverage of unseen minority-class structure and we don’t learn anything new about P(X∣Y), and we see this in our experiments -- which is ultimately a fundamental limitation.
SMOTE et al. (including cluster-based SMOTE) are different in their approach; they try to create new data by interpolating between existing observations. This implicitly makes assumptions about the distribution of P(X∣Y) in your dataset, that our features can lie in locally-smooth segments in X (in the case of SMOTE), or, in clusters (as is the case with cluster-based SMOTE). If this assumption turns out to be true, SMOTE et al. can be very successful in rebalancing the minority class, and I have had good performance in other modelling projects with this method. The problem comes in where P(X∣Y) is not smooth: in this case it's almost impossible to generate new data, and SMOTE et al. won't work. In our case, we saw that the cluster-based SMOTE model actually performed as well as the baseline model -- this is a clear indication that the minority class on X (the feature space) is not positioned in clusters: cluster-based SMOTE may have provided more (synthetic) data, but it did not produce more information, as can be seen in the model performance metrics being almost identical to the baseline model's.
Class imbalance is not primarily a problem of frequency, but a problem of incomplete representation of the conditional feature space. Sampling changes the training distribution, but it does not create new signal. Collecting more data to fully observe this feature space would be the ideal response, but sometimes this isn’t possible and we remain limited to the available data. In these cases, the sampling techniques here may help you; one never truly knows ahead of time which method may prove the most successful, because each method assumes something different about the feature space. Owing to this, there is unfortunately no one-size-fits-all approach to class imbalance, it’s more a case of experimenting and trying new things. Sampling can help rebalance training distributions, but it cannot manufacture information that does not exist in the data -- which may be ultimately limiting to you given your modelling objectives.
Ultimately, class imbalance is rarely solved by sampling alone; it is solved through better representation of the underlying feature space. For this reasons, I view sampling as a tool -- like a bandage to put on the wound, but that it doesn't address the wound itself. Often, in practice, class imbalance is solved through sampling and model-based imbalance solutions (which we'll explore in the next blog).
If you've found this useful or interesting, please consider subscribing, or share to interested colleagues. I publish long-form, in-depth articles on ML, AI, and Data Science (just like this one) every two weeks or so, as well as practical business-oriented coding problems, like this one. If you'd like to get in touch, drop me a line.