13
$\begingroup$

Suppose I've got an 80/20 split between modeling/validation observations. I've fit a model to the modeling data set, and I'm comfortable with the error that I'm seeing on the validation data set. Before I roll out my model for scoring future observations, is it appropriate to combine the validation back with the modeling data to get updated parameter estimates on the 100% data? I've heard two perspectives on this:

  1. The validation that I've performed was a validation of the model structure, ie the set of predictors and transformations that I've applied. It doesn't make sense to leave 20% of my data on the table when I could use that to update my estimates.

  2. The validation that I've performed was partly a validation of the parameter estimates that I calculated on my modeling data set. If I update the model fit, I've changed the estimates and I no longer have an objective way to test the performance of the updated model.

I've always followed argument #1, but more recently I've heard several people argue #2. I wanted to see what others think about this. Have you seen any good discussions in the literature or elsewhere on this topic?

$\endgroup$
6
$\begingroup$

Option 1 is the correct one. You can add the validation data set to update the parameters of your model.

That is the answer. Now let us discuss it. The fact that what you did was your option 1. above is clearer if you did either k-fold cross-validation of bootstrapping (and you should have done that - it is not clear from your question).

In 5 fold cross validation, you divide the data you have into 5 random sets of equal size. Let us call them A, B,C, D and E. Then you learn the parameters of your model (of the model itself) in 4 of the sets, say A,B,C and D, and test it or validate it in the fifth model E. (This you did). But then you select another set as the test/validation (say D) and learn using the other 4 (A,B,C, and E). Test it on D, repeat.

The error you your predictive model is the average error of the 5 tests - and you have some understanding on how the predictive error depends on the learning and testing sets. In the best case scenario all 5 measures of error are similar and you can be reasonable sure that your model will perform at that level in the future.

But what model?? For each set of learning sets you will have a different parameter for the model. Learning with A,B,C,D generates a parameter set P1, learning with A,B,C,E, the parameter set P2, up to P5. None of them is your model.

What you tested is the expected error of a procedure to construct models, the procedure that you followed when the learning set was A,B,C,D and when it was A,B,C,E and so on. Is this procedure that generates a model with that expected error.

So what is the final model? It is the application of the procedure in all the data you have available (A,B,C,D and E). A new model with parameter set P0, which you never generated before, you have no data to test it (since you "used" all the data in determining the the parameters P0) and yet you have a reasonable expectation that it will perform on future data as the other models (P1, P2...) constructed using the same procedure did.

What if you did not perform a cross-validation or bootstrap (bootstrap is somewhat more complex to explain - I leave it out from this discussion)? What if you only performed one learning/validation split and one measure of error. Then, argument 2. may be somewhat correct but you have a bigger problem - you have only one measure of the error of the model and you do not know how dependent that error is on the data used to validate it. Maybe, by luck, your 20% validation set was particularly easy to predict. Not having done multiple measures of error, it will be very risky to assume that the expected error rate of your predictive model will remain the same for future data.

Which is the "bigger risk"? To assume that that error will remain basically the same for future data, or assume that adding more data to learn your model will in some way "screw up" the model and increase its error rate in the future? I don't really know how to answer this, but I would be suspicious of models that get worse with more data....

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.