I am working on model selection problem for noisy data sets. I am having non-parametric models like SVR, regression splines etc. which have can overfit if the hyperparameters are not tuned properly. I have added gaussian distributed noise to the response (0 mean, 5% of response as standard deviation). Can I say that a model whose "RMSE of predictions" on trained data set (goodness of fit) or on test set (prediction accuracy) is less than "standard deviation of noisy response with respect to the true noise-free response" is actually overfitting (by capturing noise) on train or test set, accordingly?

Mathematically, I mean to say this: If $ \sqrt((y_p-y_n)^2/K)<\sqrt((y_n-y_t)^2/K)$ , where $y_n, y_t, y_p$ are noisy response, true underlying response and model predicted response for all K data points in either train or test set; then I can say that my model is fitting noise in that data set. Is this correct?

P.S: AFAIK a model that overfits on train set will always generalize poorly on test set i.e. give an RMSE > standard deviation in noisy response from true response. I feel that this is how we check overfitting - if a model has very small training error and generalizes poorly on new data points, it means it is overfitting and has learnt noise. I dont know if it is possible for a model to overfit on test set as well as on train set, because there is no way to learn the (random) pattern in noise at unseen locations. Please let me know if my understanding is correct.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.