Data preparation for regression

Question

I am trying to predict real estate sales prices.

• In my dataset there are independent variables that are both nominal and numeric (square meters, prices etc.)
• Before feeding the data to any regression algorithm I'd like to preprocess it correctly (binning, normalizing mean / std deviation, discretization etc.)
• I am overwhelmed by the many methods listed in various textbooks and try to find out what works well in practice

Although the most reasonable answer to this question is probably 'it depends', could you maybe give me some rules of thumb / war stories / general advice?

• How do you usually preprocess data for regression?
• What methods do you usually apply?
• Which regression algorithms need a special treatment?

As my tools I am using weka and R.

Many Thanks!

Answer 1

(You may start from the after line section, for a shorter answer) To begin with, you are absolutely right saying that it firstly depends on the purposes of your analysis: forecasting of average price (at macro level) or a particular price (at micro level), causal analysis of consumer preferences (district, size, age, number of bedrooms, gas, travelling to the job, level of noise, etc.). This verbal specialization secondly will guide you to an appropriate choice of a model and, finally, requirements for your data.

From what you have written, I assume, that you deal with the real estate pricing models. After quick googling showed there are many ways to specify a model. Quite good starting reference could be Simon P. Leblond's article Comparing predictive accuracy of real estate pricing models: an applied study for the city of Montreal. From practical point of view you have to choose between additive or multiplicative regression models. The latter has several advantages as opposed to additive models:

• parameter estimates (but intercept term, junk regression parameter anyway) are not affected by the changes in scale
• parameters for log-transformed variables have a nice elasticity interpretation, that ...
• naturally allows for diminishing returns to scale restrictions (in real estate this one could be crucial restrictions)
• if one studies average prices, more robust averaging is weighted geometric mean than average (this will not be relevant too much at micro level though)
• you may set price to zero, if, for instance, your apartment has no bedrooms at all (it is hard to do so with additive models)

One more important thing before you proceed is to think about each of your observation as a unique data point that was jointly set on the market by a decision maker on the basis of utility maximizing behaviour. Jointly meaning here that you can't separate the variables from each other (for instance, the value of apartments without a bedroom is zero for most of the consumers), but a consumer may or may not like all bundle of the attributes together, after that his or her budget (money in the pocket) is all that matters. Therefore standardization is useful for analysis of relative importance of explanatory variables, but be careful judging what variables are not significant (all factors may be important). Heterogeneity of preferences and budgets (buyers are different households) in each case of your observation also shows why regression at micro level (not averaging or so) could also be misleading. Finally, you have cross-sectional (static) data. Trying to predict prices for different years (than the year of your observations), static pictures work poorly for different periods of time (say you build model based on 2009 year's data, it will be not very useful retrospectively predicting prices for say 2007, or for 2011). At least try to correct the outcomes on the basis of change in average price for a particular year in this case.

---

Regarding your particular questions (what I personally do for my projects, or at least pretend to do):

1. List all the variables you have and their measurement units
2. Check and re-check the data for imputation errors
3. Make additional imputation for the points with missing values (you may also simply exclude the observations if you have large dataset with not so many missing values)
4. Make all measurement units the same across similar variables (sq. meters, currency units, etc.)
5. Think of a simple data frame structure at once (you need to communicate with $R$ conveniently)
6. Bring only raw data to $R$, make all log, differences, fractions transformations in $R$ directly (logarithms are important for multiplicative models, some pros for one are in the prelude above; fractions are also nice for you may want to eliminate the scale (size) effect at once, and emphasize the differences caused by other factors)
7. Leave dummies as they are but always leave one level of qualitative attribute for intercept term (if not this would be a source for pure multicollinearity problem in your model)
8. For your purposes you may apply ordinary least squares (OLS), though in pricing models I would also consider tobit or Heckman models, that do need a special treatment (one of my early may-be-not-so-successful post's on pricing was about this)
9. OLS is straightforward and usual residual analysis (found in textbooks on econometrics) is done. Violating some of the assumptions you may go for generalized methods, instrumental variables, ridged regression, cures for autoregressive residuals, but... What you really need to know: are the parameter estimates theoretically reasonable (values, signs, etc.)?
10. Just a nice number... any additions from the community are welcome.

Answer 2

The real estate prices that you are tying to predict , are they consecutive/chronological values i.e. time series data or are they prices for different classes e.g. this years prices for different classes for the same time frame. You might want to read something I wrote on these two kinds of problems as it warns that if you are dealing with longitudinal data ( time series) then the tools of ordinary cross-sectional regression will not ordinarily apply. It is entitled "Regression vs Box-Jenkins" http://www.autobox.com/pdfs/regvsbox.pdf .

Answer 3

Binning your data is usually a bad idea because it will cause you to lose information, which will likely result in loss of power. Also, I would rarely standardise variables before doing regression, although some people may like to.

A really good book to read, if you can get it, is "Regression Modeling Strategies" by Frank Harrell.

Answer 4

For preprocessing I always like to include outlier detection, and removing bad data. If your data is of different scales, normalizing the data is a good idea (standardization). As far as technique goes, it always pays to graph and plot all of your variables with each other, as well as with the predicted variable. That will tell you a lot about which assumptions you can make about the data such as linearity, equality of variances, normality and can better help you choose a technique.