Do you know any technique for checking if a machine learning regression model gives different performance for different values of features?

Question

Let’s say I have fitted a regression model on a large dataset, and I obtain an estimation performance metrics (R-squared, RMSE) for the whole dataset, via cross validation or via a test set. However, this estimation is relative to the whole dataset, with no difference in different parts of the feature space. It would be possible that the model perform well for certain values of the features, and bad for certain other values.

How can I detect that? Is there any technique?

The only thing I can imagine is to build several test sets containing data with different feature values. But that is gonna be very complicated and arbitrary if I have a large number of features.

Do you have any idea?

Thanks a lot in advance,

All the best

Davide

Answer 1

your crossvalidation estimates of each point are valid out of sample estimates. so with any summable metric eg logloss,MSE, but not RMSE, you can just breakdown the total loss along each feature. eg if one of the features is age_bracket then overall MSE = weighted mean of Squared error in each age_group

so you can analyse your loss with a pivot table of dimensions and squared error on each datapoint so if your crossvalidation mse is 85, you might break it down by age and see that the 40 -80 year olds have the largest source of error.

age	proportion	Squared error
20-30	50%	20
30-40	25%	100
40-80	25%	200
Total	100%	.5x20 + .25x100 + .25x50=85