Ok, i was just introduced to the ideas of overfitting and underfitting, and its method of detection, that is splitting the dataset we have into two parts into training(80%-90%) and testing(10%-20%) dataset. But my question is havent we collected the main dataset from the same place at the same time. So maybe even if our model did overfit, it would still do good on the testing data set, and we would be deceived. Because the noise that the training dataset had would be present with the testing dataset owing to the fact I previously stated. Have i got it correct?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
The whole idea of the approach of Splitting into Testing/Training Set is based on the assumption, that the observations are Independent and identically distributed random variables. So their is no shared noise specific to the situation. Under this premise Cross Validation Techniques work quite well. In your question your are implicitly assuming that this assumpution isn't meet. Here we have another case: you are implying that the n+1 data that you want to predict with your model comes from another population - of course Cross Validtion can't tell you with MSE you have to expect in this case.
Depending on the concret circumstances, it might be a case for Multilevel Modelling.
