I need to do a high dimensional biological data analysis. My data consists of hundreds of thousands of dimensions. I am looking for an implementation of multinomial logistic regression that will scale well to data of this size.

Ideally, it should allow me to also do Ridge and Lasso regressions also. Which software should I be using?

  • 1
    $\begingroup$ @Andy, can you clarify your question a little bit? Do you have hundreds of thousands of predictors for each response? Or hundreds of thousands of responses? Or perhaps, a data matrix with hundreds of thousands of entries? Or maybe you meant "hundreds or thousands"? The answer to this will provide some guidance. Also, how many class categories do you have? What sort of computing resources are available? Are your features sparse or dense? $\endgroup$ – cardinal Mar 1 '11 at 1:52
  • $\begingroup$ @Andy, for binary logistic regression, Paul Komarek built a package as his dissertation research at Carnegie Mellon. It has some optimizations for sparse features. I don't believe it does multinomial logistic regression, but I could be mistaken. He actually uses a ridge version of logistic regression, but with a fixed ridge parameter. His claim is that that is good enough for (most) all problems. The fitting method uses conjugate gradients if I recall. I'm not sure it's amenable to parallelization, though. $\endgroup$ – cardinal Mar 1 '11 at 1:55
  • $\begingroup$ @cardinal : Thanks for your answer. Good questions ! I have 12 classes, and the number of predictors can go from 80k to 200k depending on the source of the data. And yes this is a highly sparse sparse data set. Regarding computing resources, I think I might be able to get access to a cluster (~10nodes or so), but I think if push comes to shove, I can try and get more computing resources also. $\endgroup$ – Andy Mar 1 '11 at 2:39
  • $\begingroup$ @Andy, how much information are in those features? Can't be much. Are the features continuous-valued or binary? There might be a quick-and-dirty approach where, e.g., you can train 12 binary-logistic regressions separately and then use there predictions in a final step to get a proper distribution for the twelve-class case. (For example, train a 12-feature multinomial logistic regression when you're done training each of the individual ones. Obviously, that's suboptimal, but depending on the application, it might be both computationally feasible and good enough.) $\endgroup$ – cardinal Mar 1 '11 at 2:56
  • $\begingroup$ @Andy, I just checked and Komarek does provide some very simple suggestions for multi-class extensions. They're simpler than what I was suggesting. One reason mine might be better is that if you fit 12 separate binary logistic regressions, the associated parameters aren't necessarily on similar scales. By having a final step that is a full multinomial logistic regression, the parameters in the final fit can compensate for those scale differences that a simple renormalization wouldn't. Besides, simple renorming wouldn't change your predicted class anyway. $\endgroup$ – cardinal Mar 1 '11 at 3:04

I've had good experiences with Madigan's and Lewis's BMR and BBR packages for multiple category dependent variables, lasso or ridge priors on parameters, and high dimensional input data. Not quite as high as yours, but it might still be worth a look. Instructions are here: http://bayesianregression.com/bmr.html

  • $\begingroup$ Thanks for your answer. I have heard about it also. I downloaded them but could not find any documentation at all. For ex. what should the input file format be? Am I just not seeing it? $\endgroup$ – Andy Mar 2 '11 at 0:05
  • $\begingroup$ Just here: bayesianregression.com/bmr.html $\endgroup$ – conjugateprior Mar 2 '11 at 18:53
  • $\begingroup$ Would be interested in hearing how you found the bayesian regression software BMR/BBR @Andy $\endgroup$ – Will Beauchamp Sep 17 '13 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.