0
$\begingroup$

I'm building a model that predicts house prices in order to learn some regression techniques. Currently I'm trying to engineer features that might be significant when predicting prices.

I got a hold of some historical data which includes the amount of sold houses, amount of constructed homes, etc.

I was thinking that, in addition to simply merging the data on the actual date that the houses were sold, it could be useful to feed some historical data to the model. For example: amount homes sold/constructed 1, 3, 5 years ago, etc.

My initial though was to compare the difference of amount of homes sold/constructed at the time of the sale with the previous periods. But this leads to many negative values and quite non-normal distributions.

A quick test (using a keras neural network), tells me that the predictions end up worse than before. So my question is: is this type of data useful at all? Does it make sense to keep the actual historical values instead of the differences? What are some techniques which I can approach this problem with?

Any leads on resources for me to read about this kind of feature engineering would be helpful.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Generally speaking, the more the training data the more accurate the model could be. Yet, it seems that it could be a hyperparameter tuning issue. It could (also) be a noise in the new data, therefore, it would more useful if you performed EDA analysis on the two datasets: one for the legacy in addition to new, and another one for the new fetched data. This could help you find any noisy data points that make the model worse perform.

It can be seen that there is no one ultimate optimal solution that can be valid for all (or most) cases. In other words, each case should be carefully discussed and investigated in the analysis phase. For instance, If we build a regressor to predict the rent price of a property, and we use "Home Last Sale Price" as a feature, so, it could be better to remove the legacy data of the same property if the value of such a mentioned feature changed. The old observation about the same property could introduce noise with information does not reflect reality of the meanwhile in the market.

In forecasting models, legacy data, on the other hand, could be useful. That is, data that changes over time will be necessary to forecast future values based on the changes over the variations of the previous readings. For example, predicting the rent price values next year based on the changes in rent price over the past time.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "the more the training data the more accurate the model is" Not really: adding junk data will not improve your model, it will confuse it! $\endgroup$
    – David
    Commented Apr 1, 2019 at 15:09
  • $\begingroup$ Definitely I agree .. it is about "clean data" .. EDA and Hyperparameter tuning processes are always introduced in each iteration $\endgroup$
    – Rami Azmi
    Commented Apr 2, 2019 at 5:31
  • $\begingroup$ Thanks! Regarding usage of the difference between now and last year vs. the actual data values of last year: what are your thoughts here? $\endgroup$
    – Void
    Commented Apr 2, 2019 at 16:35
  • $\begingroup$ Good point actually.. To get to the point, a dataset is known as a set of observations. Each observation represents an instance or an entity about/of something. Take home prices as a case. If we receive new information about a home with new information differ from the information in the previous observation, then, we have a new observation; e.g: the number of bathrooms could be changed.; in case the bathroom variable viewed as significant. $\endgroup$
    – Rami Azmi
    Commented Apr 2, 2019 at 16:55
  • $\begingroup$ Actually it is an interesting question and a good point by @asdf .. such a discussion helped me improve my ansers by supporting it with examples from the industry I work for. $\endgroup$
    – Rami Azmi
    Commented Apr 4, 2019 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.