52 variables after backward variable selection on logistic regression on 160 variable at beginning, whether it is illusion or good modeling

Question

Thank you for you guys effort on building such a nice community and I learned lot by reading and asking.

Background about my logistic regression:

1. Target Variable: Binary target about volatility of market. "1" for trading days with top 15% percentile volatility day which indicates the extreme day on market. "0" for rest of normal and kind of calm day for market.

2. Predictor Variable: frequency of 160 trading related key words from famous trading online forum. For example, the word “trader” in 2008-01-01 we found it occurred for 50 times and we found in 2008-01-01 we have totally 500 posts on that forum. Therefore, we kind of normalize the appearance of "trader" by dividing 50 of "trader" by 500 of "number of posts". We got 0.1 here for 2008-01-01. We have 160 words dictionary and the training data is from 2008-01-01 to 2011-12-31. Each row has 160 column with number indicating the frequency of specific words. That is all. Nothing variable of price, volatility here. Kind of pure sentiment data based on frequency counting.

3. The Goal of Modelling: I think it is quite obvious that we would want to see whether we could have the sentiment words to be good predictor on whether tomorrow would be a EXTRAME VOLATILE DAY?

4. Software I use: SAS Enterprese Miner 3 in SAS 9.1.3. Here is definition and details of modeling:
   A. I select logistic regression
   B. No data partition because I only have 1000days as my training data so I select to use complete data which means no sampling process and only training data.
   C. I select the backward as we want to see what are those significant predictor out of 160, here is some shortage if using forward.
   D. There is no validation data set so I chose the Crossvalidation-error as my ultimate criteria. Significant level for variable to stay is 0.05.

5. Result: thankful for the sas easy usage, I got the following graph directly from the the result and modeling manager:

First one is the result: as it shows how many variable left:

wow, so many variables... scaring.

Second one about the response chart which indicates how much we improve the prediction compared to pure random pick or guessing about tomorrow, here, in my case, is about 15% chance you guess it right about whether tomorrow is a volatile day. Response chart looks nice and smooth, right?

WORRIING:

TOO MANY VARIABLE LEFT HERE. KIND OF SCARING. HOW DO I IMPROVE MODELLING AND INTERPRET THE RESULT?

Answer 1

First of all, thank you for providing context.

Next: Why dichotomize volatility? This makes little sense to me. By doing this, you are saying that the most volatile day was just like the day that was the 149th most volatile (of 1000) and the 151st was just like the 1000th (presumably a day with just about 0 volatility). Dichotomizing increases both type I and type II error, and makes the result less meaningful. Use volatility as a continuous variable instead

Then, don't use backward elimination. The p-values will be too small, the coefficients will be biased away from 0, the model will be too complex. In SAS, you can use GLMSELECT to implement LASSO or LAR. For more on this, see my paper Stopping Stepwise (written with David Cassell). It's here

Even more important, though, unless the stock market acts very differently from the way I think it acts, you will violate the assumption that the residuals are independent. That is, the volatility today depends in part on the volatility yesterday. You need to account for this. There are two broad ways of doing this: Time series analysis and multi-level modeling. I think the latter is probably more suitable here, with the goal of modeling a bunch of independent variables. In SAS, if you take my above advice and don't dichotomize, you can look at PROC MIXED. If you decide to ignore that advice, look at PROC GLIMMIX.

Answer 2

+1 to @PeterFlom for an excellent (and concise) answer that hits the main points. As Peter correctly points out, using backward elimination as a model selection technique will lead to a lot of problems. It may be helpful to you to understand why that is true. My answer to a similar question here might be useful in that regard.

I wanted to say a couple other things to complement Peter's answer. First, since your predictors are counts, it may be worthwhile to take the log before using them. This is often done with counts. Basically, the idea would be that there are diminishing returns: going from 5 mentions in a given day to 10 is a bigger jump than going from 155 to 160. It is always possible that this isn't true, but it's something to think about. There is a lot of good information about the use and effects of the log transformation here.

My other note is that reducing the number of predictor variables in the manner you go about it may not be strictly necessary. A different approach is to combine predictors before looking at the response variable. For example, it may be possible to conduct a factor analysis on your predictor variables and then use just a handful of robust factors as the predictors in your model. This could provide some of the benefits you hope for without the problems Peter mentions.