Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a large set of predictors (more than 43,000) for predicting a dependent variable which can take 2 values (0 or 1). The number of observations is more than 45,000. Most of the predictors are unigrams, bigrams and trigrams of words, so there is high degree of collinearity among them. There is a lot of sparsity in my dataset as well. I am using the logistic regression from the glmnet package, which works for the kind of dataset I have. My problem is how can I report p-value significance of the predictors. I do get the beta coefficient, but is there a way to claim that the beta coefficients are statistically significant?

Here is my code:

library('glmnet')
data <- read.csv('datafile.csv', header=T)
mat = as.matrix(data)
X = mat[,1:ncol(mat)-1] 
y = mat[,ncol(mat)]
fit <- cv.glmnet(X,y, family="binomial")

Another question is: I am using the default alpha=1, lasso penalty which causes the additional problem that if two predictors are collinear the lasso will pick one of them at random and assign zero beta weight to the other. I also tried with ridge penalty (alpha=0) which assigns similar coefficients to highly correlated variables rather than selecting one of them. However, the model with lasso penalty gives me a much lower deviance than the one with ridge penalty. Is there any other way that I can report both predictors which are highly collinear?

share|improve this question
    
Sounds to me like this is more so a multivariate classification problem, as opposed to being a Linear Model. –  ADP Dec 8 '12 at 23:54
    
thanks @ADP what will be your suggested model for this kind of a situation? –  tan Dec 11 '12 at 17:34
add comment

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.