0
$\begingroup$

For model hyper parameter tuning, standard k-fold cross validation is being performed. Scoring is done using coefficient of determination, with higher test scores preferred. Dataset size is (100K, 30).

  • Model A train score is 0.80 and test score is 0.50
  • Model B train score is 0.95 and test score is 0.52

Which model should I use? I understand that Model B produces a higher test score but the high train - test deviance is concerning. Is there a name for this train - test score statistic? How do I account for this deviance? I have tried to related this problem to the bias vs variance tradeoff but finding it difficult to make the answer concrete and any help will be appreciated.

$\endgroup$
0
$\begingroup$

I dont believe this is the question at hand. So high differencing training scores, may indicate an 'influential instance' in the training data, or in other words, possible outliers. I would first try to detect if you have outliers in your data, that sneak into every fold. There are some techniques available for that, e.g. cook distance, that means delete one instance of influential instance and measure all again (it is the same when ppl look at mean income, the high income of millionaires distorts the real distribution, in case of the mean, deletion of this outlier would show a more real value or in our case model fit).

This deletion approach is only one approach there exists many more.

How to deal with that in machine learning we can see here: https://christophm.github.io/interpretable-ml-book/influential.html

Be advised that to the keen eye it is not always possible to see what is an outlier. In my case I had an observation with 4 times bigger a value as the other. but that may not alwys be the case.

In Summary, first make sure to check for outliers. And btw. it would be great if we could know which estimator and tagret outcome you are analyzing.

Update due to the wish of the OP

If you just want the short answer. You cant use one of them. The test score is too low. Your model is overfitted, in short your RF learned the trained data, but works poorly on new seen data.

If you still want one of both models from the kfold, I would look into the complexity of both models. That means choosing the lower R² model might be a better idea, because you gain the same predicted value but with less complexity. Higher complexity can derived from higher r² and may induce overfit, as it is in your case. In statistics we prefer simple models. And in your case it seems the simpler model (R² .8) has a much similar prediction power than the .95 model. Perhaps you should check prmutation importance in addition to the folds.

$\endgroup$
2
  • $\begingroup$ While I appreciate the link, my question is more around how to pick the model between different choices that yield train and test performance as stated in the question. The model is using RandomForest with different depths. $\endgroup$
    – Atlas
    Mar 9 at 11:56
  • $\begingroup$ I updated my post, i hope it helps $\endgroup$ Mar 9 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.