3
$\begingroup$

So, I was thinking of a situation of when to not split up your data into training and testing and to just train on the entire dataset, at the risk of "overfitting". If my dataset has let's say 10 columns of which 9 are categorical (which have no particular order, and one is numeric). It could happen that your training set split may not capture observations that include some of the original categorical observations. When you then go and try to evaluate your model on the test set, which will have these new categorical variables that were not trained on, your model would potentially not know what to do, and in R, or wherever your model is implemented, you would get an error.

Another situation could be where you are not interested in predicting "new data", rather, just getting a feel for the relationships/patterns between the predictors and the response variable.

Would these not be situations that even though your model will potentially be overfitted, that you would want to train on the entire dataset?

Thanks!

$\endgroup$
  • 2
    $\begingroup$ The categorical situation would be an argument for being very careful about how the split is done rather than not spliitting at all (e.g. rather than splitting purely at random, some more sophisticated sub-sampling might be called for), unless the sample was so small - such s for a designed experiment perhaps) that no such approach could work. $\endgroup$ – Glen_b -Reinstate Monica Jul 11 '17 at 10:21
4
$\begingroup$

I agree with your first point: if the amount of data is limited, then we can use the entire data for model building, instead of split the data into two sets.

However, I do not completely agree with your second point: even there are no new data to predict, and all we want is trying to discover patterns. Sometimes testing data is still needed.

For example, we want to use K-means to "discover patterns" in data. How many clusters to choose can be derived using a testing data set.

Always keep in mind, data can contain noise, and if we just do whatever we can to overfit one data set. The discovery can be false and not generalized well.

$\endgroup$
0
$\begingroup$

Would like to add my comments to the second point.

In the situation that we are using a regression model to discover pattern in the data, I would think that we do not need a training, testing and validation data set to select the model. There are other methods available to assist us in eliminating "noise" to prevent over-fitting. For instance, we can use metrics such as AIC, leave one out cross-validation (jackknife) or adjusted R-square. Using these metrics above, it will help us to eliminate the possibility of overfitting and the aim of interpolation as oppose to using a testing and validation set which is more related to extrapolation.

Feel feel to add in your views on this.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.