Differences Between Logistic Regression in Statistics and in Machine Learning

Question

I just found out that machine learning also has logistic regression as one of its methods. Can someone please tell me the differences between logistic regression in statistics and machine learning? I've seen lecture slides on logistic regression from a machine learning course, but I can't see the difference with the coverage of logistic regression in a statistics course.

Does logistic regression in machine learning have no need to check for multicollinearity?

The reason I asked this is because I've tried to run a dataset through R's glm function with binomial logit, and then I ran the same dataset through Apache Mahout's trainlogistic. But the resulting coefficients are different.

This is the command I use in R:

w1.glm <- glm(anw ~ cs, data = w1, family = "binomial")

This is the result of summary(w1.glm):

glm(formula = anw ~ cs, family = "binomial", data = w1)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.5400   0.1073   0.1924   1.0047   1.0047  

Coefficients:
            Estimate Std. Error z value Pr(>|z|)    
(Intercept)  0.42077    0.02588   16.26   <2e-16 ***
cs           1.89342    0.06427   29.46   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 11762.5  on 10660  degrees of freedom
Residual deviance:  9250.3  on 10659  degrees of freedom

And this is the command I use in Mahout:

/usr/local/mahout/bin/mahout trainlogistic --input w1.csv --output ./model --target anw --categories 2 --predictors cs --types numeric --features 20 --passes 100 --rate 50

Running on hadoop, using /usr/local/hadoop/bin/hadoop and HADOOP_CONF_DIR=
MAHOUT-JOB: /usr/local/mahout/mahout-examples-0.8-job.jar
20
anw ~ 
-19.553*cs + -7.512*Intercept Term
            cs -19.55265
      Intercept Term -7.51155
    0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000   -19.552646543     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000    -7.511546797     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000 
13/11/01 02:04:47 INFO driver.MahoutDriver: Program took 22118 ms (Minutes: 0.3686333333333333)

Edited: Added the reason I asked the question in the title. Added the commands used to execute glm in R and trainlogistic in Mahout.

Answer 1

Logistic regression refers to the same thing in both fields. It seems like Mahout does some things by default that make its implementation of logistic a little more than just logistic. First, Mahout seems to be regularizing the coefficients. If its doing this by default, I would also expect it to be standardizing (scaling and centering) the inputs. Passing it a value of lambda=0 should prevent regularization, but you still have to make sure that the inputs are not being standardized.

If you want to do regularized GLM in R check out the glmnet package.

Answer 2

Write logistic model as log(p/(1-p))= g(x1,x2,...) where g(.) is liner function of Xs' coefficients and p is the sigmoid function. The difference of coefficients between machine learning (ML) and statistics are the calculation process. ML use gradient based approach and statistics use mathematical equation solving methods. They have different properties such that different result. For multicollinearity, ML focus on precisely prediction rather than the estimation of coefficient, there is rare discussion for multicollinearity in this fild. And multicollinearity is an important issue in statistics field, not only logistic regression but also many statistical model have many discussion for multicollinearity detection, so statistics approach can conclude more reasonable coefficient by its theoretical properties rather than ML approach. As my experience, if you need meaningful coefficients to represent some importance or effect, please choose statistical version because there are some good properties to support. If you need more precise prediction, you can choose many classification model rather than ML logistical model unless you really need know the relative importance between x's using coefficient when you encounter big data.

Answer 3

My guess is that Mahout uses some sort of simplified first-order method for optimisation (similar to the linear perceptron's delta rule) whereas R's classic GLM from statistics uses ML estimation. I'm suspecting this from the fact that they're using the word "train" instead of "estimate". It would be very uncommon (not to say weird) to use the word "train" for OLS or ML estimation.

I've also noticed that, when you see terms from classical statistics such as "linear regression" and "logistic regression" used in a machine learning context, the difference is always in the optimisation process, i.e. estimation vs training (again, a good example is the linear perceptron). Training algorithms in machine learning are supposed to be more robust than parameter estimation methods in parametric statistics because they favour raw computational power over mathematical accuracy, though, they both have their pros and cons.

Anyway, I think it's good to pose such questions so that researchers, employers, and employees alike, can finally start understanding the differences between a machine learning expert (specialised computer scientist) and a statistician (specialised mathematician). They definitely have overlapping ground, such as the topics discussed in this question, but they're certainly not overlapping fields.

Answer 4

My intuition is that your learning rate is too high, which could cause your model to constantly overshoot the minima. Try out your code with small values of rate, maybe 0.05, to begin with and check if it tallies with your R output. A detail that may be relevant is confirming if the model is predicting the probability of 1 or 0. If R is predicting for 1's and Mahout for 0, then you could end up with coefficients of opposite signs.