Predict using trained model on dataset

Question

I am very new to machine learning. I have a question about running predict() on data used for training set. Here are details: I took a portion of my initial dataset and split that portion into 80% (train) and 20% (test). I trained the model on 80% of training set

model <- train(name ~ ., data = train.df, method = ...)

and then run the model on 20% test data:

predict(model, newdata = test.df, type = "prob")

Now I want to predict using my trained model on entire initial dataset which also includes the training portion. Do I need to exclude that portion that was used for the training?

Answer 1

Do not test your model on the training data, it will give over-optimistic results that are unlikely to generalize to new data. You have already applied your model to predict the 20% held out test data, which gives an unbiased estimate of classifier performance. Don't go back to the training data.

If you want a larger test dataset, you can do cross-fold validation, in which you repeatedly hold out a different 20% of the data, learn a new model each time, and test on the held out data. This lets you effectively put every sample in your test data, without including it as part of the training data.

Answer 2

It really depends on what your goal with this model is at the moment. I can see basically 2 scenarios:

1. You are trying to understand how your model makes predictions:

   In this case, yes, go ahead predict the entire data set and see how the predictions perform based on variables values and other interesting data manipulations, plotting and etc. Just keep in mind that part of your data is not known to the model, so keep track of those observations when you study them since this is where your model is highly likely to fail and you should focus on how it fails.

2. You are trying to optimize some hyper-parameter:

   If your model relies on some hyper-parameter, you will not want to check the fit of your model against known data (i.e. the training data) to help optimize the hyper-parameter, since this leads to overfitting very fast. Instead, you will want to rely on out of training data and some re-sampling procedures such as cross-validation on jackknife. Keep in mind that this is usually done programmatically.