When dependent variable is categorical (more than 2 categories) and independent variable is continuous, is there any way i can check the predictive power of independent variable? Can i apply one-way ANOVA for variable selection? I read somewhere - one-way can be applied when the situation is opposite = Dependent Variable (Continuous) and Independent - categorical having more than 2 categories. I know i can use chi-square when both variables are categorical. My task is to reduce variables from 1000 to some top 100. Note : I am not looking for variable selection method to remove multicollinearity.
One way to check whether a single continuous variable $X$ is predictive for a categorical variable $Y$ is to fit a multinomial logistic regression model for $Y$ using $X$ and one without any predictors, and to perform a nested likelihood-ratio test to check if $X$ is predictive for $Y$.
A simple way to generalize it to $n$ variables by repeating it for each variable separately and selecting only the most important ones.
The obvious drawbacks is that this method is not able to capture non-linear relationships and that it may select redundant variables, but it may be a good way to start.
An alternative method is to use L1 regularized logistic regression and to select the best feature set using cross-validation or some other model selection method (like BIC or AIC), similarly to LASSO for continuous outcome variables.
I suggest a boosted multinomial logit modeling strategy as implemented in the
mboost package in R, and possibly in other libraries or packages in R and other platforms. This method iteratively fits a set of base-learner generalized linear models that the user specified by the set of predictors to be included. Base learners are usually specified as very simple models with one or two effects, but can include as many effects as the user desires. The base learner that leads to the smallest negative loss is chosen in each iteration to be incorporated into the averaged model. At the end, you are awarded both the average effects of each parameter as well as the probability that each base learners was selected in a given iteration, which you could consider a sort of base-learner importance score. The
mboost package allows you to specify smoothed effects in a base learner. Another benefit of this package is that the boosted GLM method is resistant to multicollinearity or separation issues, as well as the curse of dimensionality. Moreover, there is only one parameter to tune: the number of iterations. You can do so using bootstrap or k-fold cross-validation methods for estimating the generalization error. I believe that
mboost and the packages that depend on it are among the most useful R packages that exist.