Multiple linear regression approach: difference in set-up and dataset split

Question

Firstly, I am new to modelling linear regression models. I want to build a linear regression model to predict building energy use based on building parameters. I'm having a dataset of 100000 buildings with about 40 parameters that have correlation with the building energy use I want to predict. I plan on using a step-wise multiple regression as a starting point of the analysis towards a good regression model.

The literature tells us to always split the dataset into a training and test-set.

• However I found that some people split the dataset in 2 (training and testset) and some people split it in 3 (training, validation and testset). Which approach is better and why?
• What is a good way to divide the data over this training and testset? -> Equally 50-50 or do other other weighted divisions occur (e.g., 80-20)?

Answer 1

Splitting into three sets is preferred by some.

• Training set: This is the data we use to infer the model. This might be parameters in a linear regression or number of nodes in a neural net or whatever

• Validation set: Once we have a model we see how it performs in the validation set. We might see a pattern in the residuals when performing a regression; this tells us that we might need to change something about the model (e.g. include polynomial terms). We then retrain the new model using the same training set at before. Keep validating until we are happy

• Test set: Once we are happy with the model we should aim to see how it "performs in the wild". Although out validation set wasn't explicitly used in the training of the model, it had influenced the final model choice. The test set allows us to assess the pros and cons of the final model.

How to split into train/val/test kind of depends on your sample size. You have a pretty big dataset so train:val:test = 60:20:20 would be reasonable.

Note that an alternative method would be cross validation. Here you randomly choose lots of sets of train/val data. This can be tricky when it is computationally expensive to infer the model parameters, but with linear regression this is straight forward.