0
$\begingroup$

I am looking to buy a house, and I therefore look at a database of houseprices up to today (some are sold, some ads are live). I want to do a regression so i can see if a house for sale is over or underpriced. Lets assume i have a simple model: Houseprice = sqm + renovated + swimmingpool + area

Now I want to do a random forest to see what the predicted price for a house should be. The normal wisdom is to split the dataset in a training and testing set. However I dont get that.

For my purpose, isn't it better to use all the data to estimate the model, and then use the residual to see if a house is over or underpriced?

I dont care about future predictability. Is now I need to estimate the house that is the best bargin.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The rationale for splitting the dataset into training, (validation) and testing subsets is to try to reduce overfitting (by getting an estimate of it). The concern for overfitting implies that you want to generalize results to a broader population.

In your scenario, it seems your database of house prices today is the total population you care about, i.e. you want to choose one of the houses in your dataset, and you do not concern yourself with future hypothetical houses that might come up. If that is the case, you are right in that there is no need to subdivide your dataset and you should estimate your model with all the observations you have.

In other words, what you are trying to do is to describe your dataset as a function of certain attributes you care, hence no inference is involved.

Improve this answer
$\endgroup$
2
  • 3
    $\begingroup$ Your answer reads a little bit like: "data splitting avoids overfitting", which is wrong. It helps to measure the amount of overfitting and supports model selection. But the model can still be overfitting on the training data, like e.g. a random forest. $\endgroup$
    – Michael M
    Commented Nov 1, 2020 at 20:26
  • $\begingroup$ I am not sure I follow your comment. The model can overfit the training data even without the data split? 'Over'-fitting with regards to what? If the sample is the population of interest, to what are you comparing the fit? $\endgroup$
    – Kuku
    Commented Nov 1, 2020 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.