I am using machine learning to approach a balanced binary classification task.

Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate this, I decided to do two things.

  1. Sample weighting (to emphasize more important points in the training process)
  2. Custom evaluation/scoring metric (to reflect whether model got important rows correct)

The problem is this: When I use my custom evaluation/scoring metric for purposes of model selection during cross validation, there appears to be overfitting to the validation every time. That is, the performance during cross validation (for model selection/hyperparam tuning) is considerably better than on the test set.

Context: I am doing nested cross validation. So for 5 different outer folds, am I able to compare the inner cross validation performance (vs. baseline) against the outer fold test set performance (vs. baseline).

When I score the models (in model selection and on the test set) using my metric, there seems to be a consistent and considerable worsening of performance from cross validation to test set. This does NOT happen when I use a regular evaluation metric, like accuracy for model selection and test set evaluation. Why might this be?

My evaluation metric is defined as follows: Every data point is assigned an importance score between 0 and 1, exclusive (it's never 0, and never 1). These importance scores are used as sample weights AND they are the same scores that are counted by the evaluation metric. The metric is simply the sum of all importance scores of every row predicted correctly minus the importance scores of every row predicted incorrectly, divided by the total number of rows. So, it's sort of like "average importance correctly predicted per row".

I would greatly appreciate any insight here, as I have never used a custom evaluation metric before, and I'm happy to give a bounty to an answer.

  • 1
    $\begingroup$ Have you considered any other regular metric than accuracy? Accuracy is a rather poor metric. Are the sample weights balanced as well (on average they are the same for both groups)? What do you do with the importance weights for the test set? I assume that in prediction time you won't be having them, so I guess, you are not using them in the test set as well? $\endgroup$
    – Tim
    Mar 10, 2022 at 22:38
  • $\begingroup$ Can you please clarify how the importance weights are created? I suspect that the evaluation is inadvertently biased if the same data are used to predict "in-sample" (for training as well as validation evaluations) but then to predict "out-of-sample" (for testing evaluation). $\endgroup$
    – usεr11852
    Mar 11, 2022 at 14:03
  • $\begingroup$ @Tim I have not considered any other "baseline" metric - I just wanted some point of comparison. The sample weights are pretty close to balanced for the two groups. You're right, I don't use the importance weights in any way (besides scoring) since I don't have the weights ahead of time in real time. $\endgroup$ Mar 11, 2022 at 15:43
  • $\begingroup$ @usεr11852 Absolutely! The data is a financial time series, and the importance weights are based on the magnitude of the price movement (bigger moves get more weight). I'm not sure I understand your suspicion/concern, and why (in such case) this problem wouldn't appear for accuracy metrics. $\endgroup$ Mar 11, 2022 at 15:45

1 Answer 1


It sounds like what you want to optimize are the "importance scores". With your metric, you are verifying if your model is able to correctly classify the samples as important vs not. In such a case, why not make the "importance" your target variable? You could use as a target -1 * importance for the negative class and +1 * importance for the positive class and treat this as a regression problem (or classification, if your algorithm allows for a fuzzy target), whereas for making hard classifications, you would use some threshold over the predicted scores. In such a case, you would be directly optimizing the scores. There wouldn't be a problem with designing custom metrics because you could just use standard loss like logistic loss (for $\pm 1$ labels), squared, or absolute error (unlike the two previous ones, absolute error is not a proper scoring rule, you probably would like to stick to the proper ones).

On another hand, if you really care only about using importance weights when training and validating the results, why not just use standard weighted metrics? Weights can be introduced for standard metrics (e.g. squared error) by replacing averaging the individual errors with a weighted average.

Finally, as about the metrics themselves, accuracy is a poor metric. Accuracy doesn't care if the predicted scores are well callibrated, so it also would not be able to measure also the kind of changes that you intend to measure with your metric. In the case of standard classification, it can be the case that different metrics disagree because they measure different things, that is why the general advice is to stick to a single metric that you optimize.

  • 1
    $\begingroup$ @VladimirBelik I don't follow why can't you predict the importance * label if you care about predicting the importance? You are optimizing your algorithm to predict one thing, but validate the algorithm using completely different criteria. You are making things harder for yourself, while the solution is pretty simple, as stated above. With using importance * label as target, you are optimizing for the same criteria as measured using your custom metric. $\endgroup$
    – Tim
    Mar 11, 2022 at 16:09
  • 2
    $\begingroup$ @VladimirBelik ok, so you don't want to accurately predict the importance, but use it as a weight when training, but then, why won't you use just a standard loss but weighted using the weights (i.e. instead ov averaging it, you would use weighted average)? $\endgroup$
    – Tim
    Mar 11, 2022 at 17:07
  • 1
    $\begingroup$ @VladimirBelik not sure about ready implementations in R, but they are trivial to implement, also R has a lot of packages, so I'd expect that you should find it. Yes, oversampling and weighting are technically same. $\endgroup$
    – Tim
    Mar 12, 2022 at 21:39
  • 1
    $\begingroup$ @VladimirBelik I can't give you the exact answer without access to the data. But I'll repeat myself: you’re optimizing completely different criterion than using as a evaluation metric, so it's not surprising that the results aren't remarkable. $\endgroup$
    – Tim
    Mar 14, 2022 at 19:49
  • 1
    $\begingroup$ So either change your loss function (as suggested above) or the metric. $\endgroup$
    – Tim
    Mar 14, 2022 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.