0
$\begingroup$

I trained a linear regression model using 3000 data points. (OLS regression, no regularization.) Then I trained another model with the same predictors (about 25), but with a subset ($n=700$) of the 3000. Is there a systematic reason why the model trained with the subset of the training data performs significantly better on a test set ($n=1300$)?

$\endgroup$
  • 3
    $\begingroup$ I imagine this is luck and an interesting way to test my suspicion would be to do this 100 more times with the subset of data and see whether training with the subset continues to outperform training with the whole sample. My suspicion is the answer will be "no, it doesn't"... $\endgroup$ – the_scheining Oct 23 '18 at 17:06
  • $\begingroup$ Thanks for the suggestion. To clarify, where does the variation come from? You're suggesting 100 more subsets of the same size? Or bootstrapping the same subset? $\endgroup$ – raid_the_inarticulate Oct 23 '18 at 17:10
  • $\begingroup$ I thought 100 randomly chosen subsets of the same size. $\endgroup$ – the_scheining Oct 23 '18 at 18:33
0
$\begingroup$

You didn't really give enough information for a good answer. For instance, you didn't tell us how that subsample of 700 observations were chosen. So for this answer I will assume it was taken randomly. If that is incorrect, you should tell us.

So then, the reason was probably only luck. To see if that is true, repeat the subsampling randomly 100 times (say), and plot a histogram with the errors on the validation set. You should be able to conclude.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.