Correct application of linear regression models from different types of features

Question

I want to build a predictive model of a value which changes over time from features which can be partitioned into 2 types - 9 features which describe the history of the value (And could be used to build some sort of predictive function of the future value), and one feature which describes a semantic feature of the current state which is supposed to also effect the predicted value.

Is the correct thing to learn one linear model based on all these 10 features, despite the fact the first nine features are of a different nature from the last one? If so, would the same answer be true had there been 1 million features of the first type, and still only one of the second? I'm worried that the second one would "disappear" (i.e. get a small weight) since it's only one feature against 1 million other ones - despite the fact that it might be very significant.

Answer 1

Linear regression excels when it comes to taking into account different kinds of predictors. Unless the number of predictors gets to be large relative to the number of observations, you should be fine. (The "small sample adjustment" for inflating standard errors in OLS is $n/(n-k)$, which only gets you into trouble when $k$ is large relative to $n$.)

However, the bigger problem you're going to run into is that for time series data, the errors are likely to be correlated over time, which violates the independence assumption of OLS. To "fix" OLS, you should look into the Newey-West estimator, although your best bet may be to make a full-on autoregressive model with nine AR terms and a covariate, and let a time-series program do the heavy lifting.

For a good introduction to time series analysis I recommend Bowerman et al's book:

http://www.amazon.com/Forecasting-Time-Series-Regression-CD-ROM/dp/0534409776

For a more in-depth technical discussion with all the matrix math you'll ever need, I recommend the book by Box, Jenkins, and Reinsel:

http://www.amazon.com/Time-Analysis-Forecasting-George-Box/dp/0470272848

On the software side of things, I know Stata has "arima" and "newey" commands, and there are also dedicated time-series programs such as EViews. I'm almost certain there are R packages as well, but I am not familiar with them.