I was wondering about model selection problem. To be more specific, how to split the data and use cross validation. So let's imagine situation:

We want to create some predictive model on data set D. Let's say we're considering $\bf only$ simple linear regression and partial least squares (so no tuning parameters). So, what we can do now is:

  • we're dividing $D$ on two sets - $D_{test}$ (i.e. 30%) and $D_{train}$ (i.e. 70%)
  • we want to select model. So we're using k-fold cross validation on $D_{train}$ and calculate $k$ times some performance measures, i.e. $RMSE$, take average (optionally standard deviation) and basing on that we make decision about model selection.
  • So let's assume that basing on aforementioned step finally we have chosen simple linear regression. So we build model basing on whole $D_{train}$ and check its performance on new, unseen data, that means $D_{test}$. Let's say that performance is good enough and we accept model. So we take $\bf all$ our data, build simple linear regression model and go to manager with results.

My question:

  • Is it good that we only have one test set? I mean maybe the first division we made (70/30) was kind of special - for example there are lots of outliers in $D_{test}$. Wouldn't it be reasonable to do many $\bf initial$ splits of data?
  • $\begingroup$ you should try to ask only one question per thread $\endgroup$ – Antoine Nov 15 '16 at 20:46
  • 1
    $\begingroup$ ok, it's edited already $\endgroup$ – jj_konan Nov 15 '16 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.