What is the best way to use a discrete variable with a lot of classes in linear regression? I have a data set with a few discrete variables having a lot of classes(>100,000). This is too large to encode using a one-hot method. What is the best way to deal with such variables? Should I be using some other model?
I have yes/no responses to a set of questions and am trying to predict features based on these. But each user answers only a very small random subset of questions, so in order to make a prediction I need to determine the probability of answering yes/no for the rest of the questions.
 A: Maybe you would like to try regularisation that works on the levels of the categorical variable, for example a type of group LASSO approach. See e.g. Gertheiss, J. and Tutz, G. (2010): Sparse Modeling of Categorial Explanatory Variables, The Annals of Applied Statistics, 4, 2150-2180 http://arxiv.org/abs/1101.1421
This will automatically cluster the categories in a LASSO path and conduct variable selection.
Additionally, please take care not to use linear regression in the case of binary responses or their probabilities, it is suboptimal (rather use a glm for binary data or proportions).  
Another possibility would be to use recursive partitioning (classification trees), which will greedily search for a stepwise function through the whole space spanned by the categories of the predictors and select one at each step. If interpretation of the tree structure is of interest, use an unbiased algorithm such as GUIDE or ctree, if prediction is more important CART or C4.5 will do.   
A: maybe you can try reducing the number of classes. More classes for relatively less number of observations creates big problems for linear regression. For example, if your variables have classes 1 to 5, introduce one variable for classes 1-3 and some other for 4-5. This thing of course depends upon the number of your classes, number of variables and most crucially upon number of observations.
As for other models, it depends. Dummy variables work fine in normal least squares method. Or you can also use a logistic model. Just make sure that number of observations is considerably more than number of variables and number of classes.
If you want to use data as it it, partial least squares may be useful-it is designed to be used when number of observations are comparable with number of variables. But for a huge number of classes, I don't know whether it will work or not.
