# model selection using cross validation

I was wondering about model selection problem. To be more specific, how to split the data and use cross validation. So let's imagine situation:

We want to create some predictive model on data set D. Let's say we're considering $\bf only$ simple linear regression and partial least squares (so no tuning parameters). So, what we can do now is:

• we're dividing $D$ on two sets - $D_{test}$ (i.e. 30%) and $D_{train}$ (i.e. 70%)
• we want to select model. So we're using k-fold cross validation on $D_{train}$ and calculate $k$ times some performance measures, i.e. $RMSE$, take average (optionally standard deviation) and basing on that we make decision about model selection.
• So let's assume that basing on aforementioned step finally we have chosen simple linear regression. So we build model basing on whole $D_{train}$ and check its performance on new, unseen data, that means $D_{test}$. Let's say that performance is good enough and we accept model. So we take $\bf all$ our data, build simple linear regression model and go to manager with results.

My question:

• Is it good that we only have one test set? I mean maybe the first division we made (70/30) was kind of special - for example there are lots of outliers in $D_{test}$. Wouldn't it be reasonable to do many $\bf initial$ splits of data?
• you should try to ask only one question per thread – Antoine Nov 15 '16 at 20:46
• ok, it's edited already – jj_konan Nov 15 '16 at 20:56