I have been using R to calculate logistic regression with many independent variables for a Ruby on Rails web application. However, I can no longer import data from the database to R using RPostgreSQL. The web host has stopped allowing insecure connections to the database. The point is, I either need to get a new web host, or write my own logistic regression algorithm in Ruby. Ruby probably isn't the best programming language for that kind of thing, but I don't really have a choice. Is there an easy to implement algorithm for multiple logistic regression?
$\begingroup$
$\endgroup$
7
-
$\begingroup$ Some of the responses here might be informative: stackoverflow.com/questions/13673769 $\endgroup$– smilligCommented Apr 24, 2013 at 9:59
-
1$\begingroup$ @smillig That's an interesting reference. Gradient descent seems like a poor strategy for solving the ML equations, though: it's just too easy for it to lock into a local maximum that is not a global one. $\endgroup$– whuber ♦Commented Apr 24, 2013 at 13:34
-
$\begingroup$ I actually contacted that guy about his Ruby algorithm months ago. I don't remember what came of it, but I didn't come out with anything I could use. He had an interesting website though. $\endgroup$– EricCommented Apr 24, 2013 at 21:20
-
$\begingroup$ @whuber the negative conditional log likelihood is convex, so every local maxima is a global maxima. See for example this short writeup. $\endgroup$– altoCommented Apr 25, 2013 at 0:25
-
1$\begingroup$ @user603 I was mainly commenting on the global minima local minima. Thanks for the additional info though. I likely haven't encountered these issues as the only time I've rolled my own logistic/multinomial regression, I just used SGD and some appropriate learning rate schedule tweaking. $\endgroup$– altoCommented Apr 26, 2013 at 0:13
|
Show 2 more comments
1 Answer
$\begingroup$
$\endgroup$
6
As regression problems go, it's actually a fairly complicated algorithm. The answer to your question depends a lot on whether you have access to a reliable general-purpose CG optimization algorithm. If you do, the problem becomes somewhat simpler. If you don't, I wouldn't recommend re-implementing logistic regression from scratch (though others have tried, see here for a minimal R implementation without a GC routine) for the reasons explained here.
At any rate, the underlying likelihood surface can be nearly flat so you have to be careful about the small prints of the implementation and test it on many corner cases (these are situations where the $X$ are highly correlated or when the two groups are nearly perfectly separable).
A possible (quick and dirty) alternative is to rescale all your $X$'s to be in $[0,1]$ --for example by using the inverse logit function on each of them individually (after they have been standardized first to have mean 0 and unit variance)-- and estimate a fit by OLS (this approach is called the linear probability model). It will not be the same model and the coefficients won't be comparable but the results will be better than doing OLS on the raw data. The advantage here is that implementing OLS is trivial, assuming you have access to a good ruby linear algebra library (googling around I have found quix/linalg)
-
$\begingroup$ Honestly, all of the Xs are already [0,1]. The Xs are a vector that is scaled by dividing everything by the square root of the sum of squares of all Xs. I haven't heard of doing such a thing before. Would I just predict Y as a numeric 1 or 0 based on the Xs? Also, wouldn't it be possible to have predictions greater than 1 or less than 0? $\endgroup$– EricCommented Apr 24, 2013 at 20:58
-
$\begingroup$ Do you have a link to a document that compares logistic regression to the OLS approximation? $\endgroup$– EricCommented Apr 24, 2013 at 21:10
-
$\begingroup$ This approach is called the linear probability model. I wouldn't think of it as an 'approximation' to the logit because it's a different model. The LPM has been studied a lot because it was long used as an alternative to logit regression is situations when good CG routines weren't available. Compared to logit, LPM will shrink the estimated OR towards the baseline which in some settings might even be desirable. $\endgroup$– user603Commented Apr 25, 2013 at 0:10
-
$\begingroup$ I think I will work towards implementing the LPM instead. No lives are on the line. Thanks for the suggestion. $\endgroup$– EricCommented Apr 25, 2013 at 0:40
-
1$\begingroup$ oh my bad, here it is (non linear) conjugate gradient $\endgroup$– user603Commented Apr 26, 2013 at 13:05