Model and variable selection are fairly popular topics. I recently asked a question on logistic regression interpretation but I'm also considering the broader question informing my inquiry. Is it possible to compare models for which variable selection will be the ultimate goal?
Background:
I have a survey with over 600 opinion questions. It's a great way to see what people are thinking but way too many for respondents to complete. I would like to reduce the number of questions asked. I have a few ways of doing this,
There is an analytical department that thinks about the psychology of each question and makes a judgement call on the most important ones. I call this the qualitative approach.
I have usage statistics on how often those who use the survey results search for each question. Which questions are clients usually interested in seeing the responses of. This is the usage approach.
Statistical approach. Which questions are the most predictive of behaviors I am interested in. The survey asks these questions on attitudes, but it also asks pragmatic questions like, do you use this product? There are hundreds of products asked about. This approach will see which question responses (scale of 1 to 5) predict brand choices most (binary 0,1). The questions are in groups like attitudes on food, technology, fashion, etc...
Ultimate Goal:
In the end, the goal is to reduce the survey but not so much that the insight is lost. I will take care of the statistical approach. I'm thinking of using logistic regression to classify the brand choices using the questions as predictors. Taking the coefficients and seeing which questions have larger coefficients, keeping those.
Alternatives
I've also considered Lasso/Ridge Regression for question selection. There is SVM, Random Forest, and many others to choose from. However, it is hard to say which one is the best at selecting great questions.
Let's say Lasso says remove questions 1, 3, and 5. And Logistic regression says questions 2, 4, and 6 have little to no affect on brand choice. Can I look at a shared measure like AIC to compare the two, or is that comparing apples and oranges?
Accuracy?
One thought was to look at the prediction results. Whichever model does the best in overall predictive accuracy, should be the one chosen. But I think this confuses predictive accuracy with variable selection. A model may be great at predicting the outcomes and not very good at variable selection. Non-linear relationships come to mind. Let's say the true relationship between brand choice and questions is non-linear, logistic regression may do horribly at prediction, while random forests is great. And at the same time logistic regression may do a better job at variable selection.
One Final Point: Confounding Bias
If I am only looking at the questions against brands I am forgetting other factors that may be causing both question and brand choice. Like gender, it may explain the fashion question and the brand choice. By not including it, I may misattribute the effect of that question. Are you afraid of this confounding when selecting variables? How could one overcome this?
