1
$\begingroup$

Firstly, I am new to modelling linear regression models. I want to build a linear regression model to predict building energy use based on building parameters. I'm having a dataset of 100000 buildings with about 40 parameters that have correlation with the building energy use I want to predict. I plan on using a step-wise multiple regression as a starting point of the analysis towards a good regression model.

The literature tells us to always split the dataset into a training and test-set.

  • However I found that some people split the dataset in 2 (training and testset) and some people split it in 3 (training, validation and testset). Which approach is better and why?
  • What is a good way to divide the data over this training and testset? -> Equally 50-50 or do other other weighted divisions occur (e.g., 80-20)?
$\endgroup$

1 Answer 1

2
$\begingroup$

Splitting into three sets is preferred by some.

  • Training set: This is the data we use to infer the model. This might be parameters in a linear regression or number of nodes in a neural net or whatever

  • Validation set: Once we have a model we see how it performs in the validation set. We might see a pattern in the residuals when performing a regression; this tells us that we might need to change something about the model (e.g. include polynomial terms). We then retrain the new model using the same training set at before. Keep validating until we are happy

  • Test set: Once we are happy with the model we should aim to see how it "performs in the wild". Although out validation set wasn't explicitly used in the training of the model, it had influenced the final model choice. The test set allows us to assess the pros and cons of the final model.

How to split into train/val/test kind of depends on your sample size. You have a pretty big dataset so train:val:test = 60:20:20 would be reasonable.

Note that an alternative method would be cross validation. Here you randomly choose lots of sets of train/val data. This can be tricky when it is computationally expensive to infer the model parameters, but with linear regression this is straight forward.

$\endgroup$
2
  • $\begingroup$ Alright that's helpful! And in cross-validation, I suppose you apply that then also on part of the dataset but keep a part aside for testing when the best model is found in cross-validation? $\endgroup$
    – Matthi9000
    Sep 23, 2020 at 14:07
  • $\begingroup$ Yeah with cross validation you could split your data into a CV set (which would then be randomly split into subsets) and a test set. Your final model is usually trained on the "full" CV set $\endgroup$
    – jcken
    Sep 24, 2020 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.