I have been using R to calculate logistic regression with many independent variables for a Ruby on Rails web application. However, I can no longer import data from the database to R using RPostgreSQL. The web host has stopped allowing insecure connections to the database. The point is, I either need to get a new web host, or write my own logistic regression algorithm in Ruby. Ruby probably isn't the best programming language for that kind of thing, but I don't really have a choice. Is there an easy to implement algorithm for multiple logistic regression?

  • $\begingroup$ Some of the responses here might be informative: stackoverflow.com/questions/13673769 $\endgroup$ – smillig Apr 24 '13 at 9:59
  • 1
    $\begingroup$ @smillig That's an interesting reference. Gradient descent seems like a poor strategy for solving the ML equations, though: it's just too easy for it to lock into a local maximum that is not a global one. $\endgroup$ – whuber Apr 24 '13 at 13:34
  • $\begingroup$ I actually contacted that guy about his Ruby algorithm months ago. I don't remember what came of it, but I didn't come out with anything I could use. He had an interesting website though. $\endgroup$ – Eric Apr 24 '13 at 21:20
  • $\begingroup$ @whuber the negative conditional log likelihood is convex, so every local maxima is a global maxima. See for example this short writeup. $\endgroup$ – alto Apr 25 '13 at 0:25
  • 1
    $\begingroup$ @user603 I was mainly commenting on the global minima local minima. Thanks for the additional info though. I likely haven't encountered these issues as the only time I've rolled my own logistic/multinomial regression, I just used SGD and some appropriate learning rate schedule tweaking. $\endgroup$ – alto Apr 26 '13 at 0:13

As regression problems go, it's actually a fairly complicated algorithm. The answer to your question depends a lot on whether you have access to a reliable general-purpose CG optimization algorithm. If you do, the problem becomes somewhat simpler. If you don't, I wouldn't recommend re-implementing logistic regression from scratch (though others have tried, see here for a minimal R implementation without a GC routine) for the reasons explained here.

At any rate, the underlying likelihood surface can be nearly flat so you have to be careful about the small prints of the implementation and test it on many corner cases (these are situations where the $X$ are highly correlated or when the two groups are nearly perfectly separable).

A possible (quick and dirty) alternative is to rescale all your $X$'s to be in $[0,1]$ --for example by using the inverse logit function on each of them individually (after they have been standardized first to have mean 0 and unit variance)-- and estimate a fit by OLS (this approach is called the linear probability model). It will not be the same model and the coefficients won't be comparable but the results will be better than doing OLS on the raw data. The advantage here is that implementing OLS is trivial, assuming you have access to a good ruby linear algebra library (googling around I have found quix/linalg)

  • $\begingroup$ Honestly, all of the Xs are already [0,1]. The Xs are a vector that is scaled by dividing everything by the square root of the sum of squares of all Xs. I haven't heard of doing such a thing before. Would I just predict Y as a numeric 1 or 0 based on the Xs? Also, wouldn't it be possible to have predictions greater than 1 or less than 0? $\endgroup$ – Eric Apr 24 '13 at 20:58
  • $\begingroup$ Do you have a link to a document that compares logistic regression to the OLS approximation? $\endgroup$ – Eric Apr 24 '13 at 21:10
  • $\begingroup$ This approach is called the linear probability model. I wouldn't think of it as an 'approximation' to the logit because it's a different model. The LPM has been studied a lot because it was long used as an alternative to logit regression is situations when good CG routines weren't available. Compared to logit, LPM will shrink the estimated OR towards the baseline which in some settings might even be desirable. $\endgroup$ – user603 Apr 25 '13 at 0:10
  • $\begingroup$ I think I will work towards implementing the LPM instead. No lives are on the line. Thanks for the suggestion. $\endgroup$ – Eric Apr 25 '13 at 0:40
  • 1
    $\begingroup$ oh my bad, here it is (non linear) conjugate gradient $\endgroup$ – user603 Apr 26 '13 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.