I am using machine learning to approach a balanced binary classification task.
Some of rows are more important/valuable than others, so getting them right is extra important. Therefore, to accommodate this, I decided to do two things.
- Sample weighting (to emphasize more important points in the training process)
- Custom evaluation/scoring metric (to reflect whether model got important rows correct)
The problem is this: When I use my custom evaluation/scoring metric for purposes of model selection during cross validation, there appears to be overfitting to the validation every time. That is, the performance during cross validation (for model selection/hyperparam tuning) is considerably better than on the test set.
Context: I am doing nested cross validation. So for 5 different outer folds, am I able to compare the inner cross validation performance (vs. baseline) against the outer fold test set performance (vs. baseline).
When I score the models (in model selection and on the test set) using my metric, there seems to be a consistent and considerable worsening of performance from cross validation to test set. This does NOT happen when I use a regular evaluation metric, like accuracy for model selection and test set evaluation. Why might this be?
My evaluation metric is defined as follows: Every data point is assigned an importance score between 0 and 1, exclusive (it's never 0, and never 1). These importance scores are used as sample weights AND they are the same scores that are counted by the evaluation metric. The metric is simply the sum of all importance scores of every row predicted correctly minus the importance scores of every row predicted incorrectly, divided by the total number of rows. So, it's sort of like "average importance correctly predicted per row".
I would greatly appreciate any insight here, as I have never used a custom evaluation metric before, and I'm happy to give a bounty to an answer.