Why custom evaluation/scoring metric is causing overfitting in (cross) validation?

Question

I am using machine learning to approach a balanced binary classification task.

Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate this, I decided to do two things.

1. Sample weighting (to emphasize more important points in the training process)
2. Custom evaluation/scoring metric (to reflect whether model got important rows correct)

The problem is this: When I use my custom evaluation/scoring metric for purposes of model selection during cross validation, there appears to be overfitting to the validation every time. That is, the performance during cross validation (for model selection/hyperparam tuning) is considerably better than on the test set.

Context: I am doing nested cross validation. So for 5 different outer folds, am I able to compare the inner cross validation performance (vs. baseline) against the outer fold test set performance (vs. baseline).

When I score the models (in model selection and on the test set) using my metric, there seems to be a consistent and considerable worsening of performance from cross validation to test set. This does NOT happen when I use a regular evaluation metric, like accuracy for model selection and test set evaluation. Why might this be?

My evaluation metric is defined as follows: Every data point is assigned an importance score between 0 and 1, exclusive (it's never 0, and never 1). These importance scores are used as sample weights AND they are the same scores that are counted by the evaluation metric. The metric is simply the sum of all importance scores of every row predicted correctly minus the importance scores of every row predicted incorrectly, divided by the total number of rows. So, it's sort of like "average importance correctly predicted per row".

I would greatly appreciate any insight here, as I have never used a custom evaluation metric before, and I'm happy to give a bounty to an answer.

Answer 1

It sounds like what you want to optimize are the "importance scores". With your metric, you are verifying if your model is able to correctly classify the samples as important vs not. In such a case, why not make the "importance" your target variable? You could use as a target -1 * importance for the negative class and +1 * importance for the positive class and treat this as a regression problem (or classification, if your algorithm allows for a fuzzy target), whereas for making hard classifications, you would use some threshold over the predicted scores. In such a case, you would be directly optimizing the scores. There wouldn't be a problem with designing custom metrics because you could just use standard loss like logistic loss (for $\pm 1$ labels), squared, or absolute error (unlike the two previous ones, absolute error is not a proper scoring rule, you probably would like to stick to the proper ones).

On another hand, if you really care only about using importance weights when training and validating the results, why not just use standard weighted metrics? Weights can be introduced for standard metrics (e.g. squared error) by replacing averaging the individual errors with a weighted average.

Finally, as about the metrics themselves, accuracy is a poor metric. Accuracy doesn't care if the predicted scores are well callibrated, so it also would not be able to measure also the kind of changes that you intend to measure with your metric. In the case of standard classification, it can be the case that different metrics disagree because they measure different things, that is why the general advice is to stick to a single metric that you optimize.