07 July 2026 Weighting Model Observations
1. Introduction
In the last two articles we introduced the idea of class imbalance in datasets, how it arises and the problem it creates, and then focussed on how we can rectify this problem by using sampling techniques to create a more balanced dataset. Now, in the third and final part of the Class Imbalance series, we are going to think about how to approach and solve class imbalance by using the ML model itself. However, before we get into that, we need to spend a bit of time thinking about how ML models learn.
2. How do ML models learn?
The job of a ML model is learn the optimal decision boundary (see fig. 1) between classes in the target, and it creates these decision boundaries by learning the relationship between the input features (for example, “age”, “outstanding credit”, “years of work experience”) and the target (for example, "Good" or "Bad" Credit). The objective of nearly all ML models is for its predictions to be as accurate as possible; meaning that we compare its predictions with known-truth (in the testing dataset), and from this we can calculate the model metrics such as accuracy, F1 score, and so on.
2.1 Loss functions
Now, let’s say we are dealing with a discrete target (that is, let’s say we are modelling something that can be represented as either “good” or “bad”, for example “Good Credit” (0) or “Bad Credit” (1) of a banking customer. Obviously 1 and 0 can represent anything you are trying to model). Accurate model would mean that our model is making predictions on 1s and 0s as accurately as possible. The difference between what our model predicts and the known-truth (in the training data) can be quantified: for each row, we calculate the difference between the known truth (1/0), and our model’s probabilistic prediction (p). Thus, a quantification of the “error” or “wrongness” for the whole dataset can be calculated. This is essentially known as Loss:
Summing up the Loss for each row (prediction of the model) for all our observations (n) gives us our overall Loss score. How do we get each prediction's loss? This may differ depending on each model’s objective. But, in the simplest case, a simple simple logistic regression’s loss function is as follows:
Where y is the target we are trying to model ("Good Credit", "Bad Credit"), p is the probability prediction made by the model, for each record/observation (i). Remember, p is the output of your model’s input features and the parameters (or weights/coefficients) your ML model is trying to estimate (through some application of calculus).
2.2 The process of learning
When you are training an ML model based off some feature data, what it’s really doing is, for most models is the following:
→ initialising the model parameters/weights (let’s assume these start off at 0) make predictions with current parameter values (and we know these predictions will be a little better than random)
→ calculate overall loss
→ use gradient descent / optimisation techniques to recalculate optimal parameters that minimise loss
→ recalculate loss
→ end if no further decrease in loss is achievable.
2.3 How does this relate to class imbalance
You may be asking yourself “What does this have to do with how models account for class imbalance!”. And you’d be right; it is distantly related, but it’s coming. You may have noticed that in the calculation for Loss, we sum up the loss per row (observation). This effectively (without explicitly being stated) considers that each row is as important as each other row in the total loss calculation. In other words, each row is weighted evenly and contributes evenly to the total loss. But what if we were to change each row's contribution to the total loss before total loss is calculated? What would we think the impact would be? Let’s try this out with a simple spreadsheet exercise.
2.4 Loss calculation
Let’s pretend that we have some information on Good Credit and Bad Credit, and that the class imbalance is 4 to 1: for four Good Credit observations, we only have 1 Bad Credit observation. Let’s further say we are going to run a simple logistic regression whereby we want to predict the probability of Bad Credit as a function of Age, in the following form:
The job of the model is to estimate the optimal values for b0 and b1. Now, let’s set b0 and b1 to some random number, run the model and store its predictions (Probability). We know these are no better than random guesses. Nevertheless, we end up with the following loss values (where 0 in the "Outcome" column represents "Good Credit", and 1 "Bad Credit").
So the model is now going to try and tweak the parameters (b0 and b1) to lower the total loss of 11.56. That looks good and it’s standard. But, what if we modify the weighting of each row? In particular, what if we conceptually decide to make the total weighting of the good credit and bad credit cases similar, across the whole dataset? We could do this by weighting each observation by the degree of class imbalance; for example, we know that for each Bad Credit observation we have 4 Good Credit observations; that’s a ratio of 1:4.
By upweighting each Bad Credit observation to 4, and keeping the weighting of the Good credit cases to 1, we’ve effectively ‘balanced’ the weighting of each observation into the loss calculation. Let’s see what the total loss looks like now that we’ve included the weighting per observation. For the weighted loss calculation, we simply take the regular loss equation in above and multiply by a scalar c, depending on the class. This form is simply:
Let's now work out the new loss, with our weighted classes.
We can see immediately that the weighted loss is greater than the equally weighted loss, and that this difference is directly attributable to the 2 Bad Credit Cases in the last 2 rows of the table. The optimisation algorithm will now try to modify the parameters in a way that minimises the overall loss and create a more accurate decision boundary; since the overall loss is different for the “weighted loss” compared to the “equally weighted loss”, the resultant model itself will also be different in its estimated coefficients/parameter values.
2.5 Mechanisms for better modelling
In essence, we’ve just told the algorithm “getting a bad credit prediction wrong is more important than getting a good credit prediction wrong”. That may seem quite trivial, but it’s effectively the entire approach we have when using models to account for class imbalance. The models themselves don’t really care about metrics like accuracy, and F1 score, they’re simply trying to minimise the total loss of all predictions and real-world targets. By changing the observational weights, we inherently change the shape of the decision boundary, and consequently the model itself with all its associated metrics. It’s obviously not clear ahead of time how these will change, however.
2.6 Practical Implementation
Now that we know how models account for class imbalance, how can you practically go about implementing this in your models? Most, if not all, different modelling libraries (Linear Models, Stats Models, SVM, etc) in Python have some argument by which you can input a parameter which scales the loss calculation as a function of the target class.
There are approximately four flavours of this: class weight, sample weight, and weighted loss, but they essentially do the same thing. XGB has a specific flavour of this, called “scale_pos_weight”, but it’s recursively doing the same thing, albeit in a complicated implementation (given the complexities of gradient-boosted trees).
There is unfortunately no “default” value to input as the class weight. A general “rule of thumb” is to input the ratio of good:bad classes as this argument, as this at least balances the weighting, but I recommend you try different values for this (for example, in a hyperparameter grid) and see which holds best for your particular situation.
3. Experiments
Let’s again turn to experimentation to prove our point. Again, we are utilising the “German Credit Dataset” to act as our imbalance dataset. Here, we have 1000 observations of Credit Status which is imbalanced at 70:30 (Good Credit to Bad Credit), with 20 already-engineered features. Our goal is to model the Credit Status. Let’s use a simple XGB to model. Here are the input features, notably, “scale_pos_weight”, which we leave at 1 (all observations equally weighted).
model = XGBClassifier(
objective="binary:logistic",
eval_metric="logloss",
n_estimators=200,
max_depth=4,
learning_rate=0.05,
random_state=42,
scale_pos_weight=1,
)
Accuracy : 0.775
ROC AUC : 0.770
precision recall f1-score support
0 0.79 0.92 0.85 140
1 0.70 0.43 0.54 60
This model has good Accuracy (0.775) and ROC (0.77) scores. But, we note that the recall for 1 (Bad Credit) is pretty poor; meaning that of all the Bad Credit cases, or model only identified 43%. This probably won't do if you work in a bank and predicting Bad Credit customers is the primary objective. Let’s see if we can do better with weighted observations to the class:
model = XGBClassifier(
objective="binary:logistic",
eval_metric="logloss",
n_estimators=200,
max_depth=4,
learning_rate=0.05,
random_state=42,
scale_pos_weight= 2.3 # (7/3; n_good / n_bad)
)
The results of this model look like:
Accuracy : 0.695
ROC AUC : 0.759
precision recall f1-score support
0 0.81 0.74 0.77 140
1 0.49 0.58 0.53 60
We note immediately the improvement to this model’s Bad Credit recall score. However, this has come at the expense of the Bad Credit (1) Precision, and the Good Credit (0) Recall. So once again we are confronted with the uncomfortable truth: that class imbalance strategies don't solve the problem outright, but give us options. Class Imbalance is utlimately limiting and no amount of data sampling or model gymnastics will make up for that.
Rather, these sampling and modelling strategies simply afford us the option of choice: instead of a single model that performs well for most metrics but poor for the minority class recall, we can now choose between another option in which the minority class recall is higher, but at the expense of the other model. These decisions ultimately reflect what it is (in terms of business objectives) we are trying to do with our model – in the case of Bad Credit, most Product Managers would prefer a model that’s better at predicting Bad Credit customers, as these represent a huge financial risk. Predicting Good Credit customers could be useful, but ultimately, Good Credit customers don't cost the bank millions in unpaid loans. To that end, we could even go further, and increase the weighted contribution of Bad Credit loans up to 5 in the model to get an even better recall for Bad Credit Recall:
Accuracy : 0.675
ROC AUC : 0.754
precision recall f1-score support
0 0.85 0.65 0.74 140
1 0.47 0.73 0.58 60
4. Beyond "Regular Supervised Learning"
When Class Imbalance is relatively moderate, you can typically use class weights in all the regular supervised learning approaches and arrive at a model that fits your needs. But when Class Imbalance gets to 1:1000 (0.1%) or more extreme, then you will have little luck in using any approach mentioned here, or in this series. At that degree of class imbalance you need to be using specialised methods to account for the severe imbalance. Some approaches I've used (albeit infrequently) include isolation forests, anomaly detection, or one-class identifiers. The trade-off with these methods is that they typically learn what "normal" looks like, and anything outside of that may be considered "anomalies". Often, however, anomalous behaviour is legitimate (think of spending and fraud detection), so the cases of false-positives is often quite high. Again, no silver-bullet, but rather more options to choose from and different trade-offs to consider. Deep-learning approaches also exist and have shown good performance -- but these are more involved.
5. Conclusion
In this series, we’ve explored why class imbalance exists and why it’s a problem for businesses. We also then focussed on data sampling methods to alleviate class imbalance, and this article focuses on how we can do this with the model itself. Throughout the series, we’ve come to realise that there is no single approach that almost always alleviates class imbalance. A greater take away is that we are approaching this as it’s a trade off; class imbalance is ultimately a limiting factor; the decision we as data scientists need to make is not “what technical tools do we have at our disposal to overcome class imbalance”, but rather “what trade-offs am I willing to make in this particular modelling exercise”.
If you've found this useful or interesting, please consider subscribing, or share to interested colleagues. I publish long-form, in-depth articles on ML, AI, and Data Science (just like this one) every two weeks or so, as well as practical business-oriented coding problems, like this one. If you'd like to get in touch, drop me a line.