# Differences Between Logistic Regression in Statistics and in Machine Learning

I just found out that machine learning also has logistic regression as one of its methods. Can someone please tell me the differences between logistic regression in statistics and machine learning? I've seen lecture slides on logistic regression from a machine learning course, but I can't see the difference with the coverage of logistic regression in a statistics course.

Does logistic regression in machine learning have no need to check for multicollinearity?

The reason I asked this is because I've tried to run a dataset through R's glm function with binomial logit, and then I ran the same dataset through Apache Mahout's trainlogistic. But the resulting coefficients are different.

This is the command I use in R:

w1.glm <- glm(anw ~ cs, data = w1, family = "binomial")


This is the result of summary(w1.glm):

glm(formula = anw ~ cs, family = "binomial", data = w1)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-2.5400   0.1073   0.1924   1.0047   1.0047

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)  0.42077    0.02588   16.26   <2e-16 ***
cs           1.89342    0.06427   29.46   <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 11762.5  on 10660  degrees of freedom
Residual deviance:  9250.3  on 10659  degrees of freedom


And this is the command I use in Mahout:

/usr/local/mahout/bin/mahout trainlogistic --input w1.csv --output ./model --target anw --categories 2 --predictors cs --types numeric --features 20 --passes 100 --rate 50

MAHOUT-JOB: /usr/local/mahout/mahout-examples-0.8-job.jar
20
anw ~
-19.553*cs + -7.512*Intercept Term
cs -19.55265
Intercept Term -7.51155
0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000   -19.552646543     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000    -7.511546797     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000     0.000000000
13/11/01 02:04:47 INFO driver.MahoutDriver: Program took 22118 ms (Minutes: 0.3686333333333333)


Edited: Added the reason I asked the question in the title. Added the commands used to execute glm in R and trainlogistic in Mahout.

• I tried with one predictor. Using R's glm, the coefficient is 1.89342 and the intercept is 0.42077. Using Apache Mahout's trainlogistic, the coefficient is -19.55625 and the intercept is -7.51155. The predictor is numeric, between 0 to 2.50. – Agus Dwikarna Nov 1 '13 at 9:09
• The dataset has 10661 records. I ran the Mahout's trainlogistic with parameters: --target anw --categories 2 --predictors cs --types numeric --features 20 --passes 100 --rate 50 – Agus Dwikarna Nov 1 '13 at 9:16
• The question and its title are not related. There is just one Statistics and ML did not reinvent it, or modify it. I don't know of any standard method that would have differing definitions in the fields. But you are not really asking this question anyway. You are asking whether the implementation of Logistic regression differs in Mahout and R to explain different results. I have no clue, but from looking at the Mahout regression code, they use stochastic gradient descent to minimize the (negative) log likelihood, while R uses IRLS. So both of these algorithms need convergence. – means-to-meaning Nov 1 '13 at 9:39
• @means-to-meaning, it was because of the difference in results that I asked if there is any difference between the implementation of LR in statistics and ML. Thank you for confirming that is not the case. I'm editing my question to put commands I use in R and Mahout. – Agus Dwikarna Nov 1 '13 at 10:18
• There is no difference, however in machine learning regularisation is typically used to deal with multi-colinearity (c.f. ridge regression). That also exists within statistics, but is less widely used as far as I can see. IMHO machine learning is essentially a computationally focussed branch of statistics. – Dikran Marsupial Nov 1 '13 at 12:17

Logistic regression refers to the same thing in both fields. It seems like Mahout does some things by default that make its implementation of logistic a little more than just logistic. First, Mahout seems to be regularizing the coefficients. If its doing this by default, I would also expect it to be standardizing (scaling and centering) the inputs. Passing it a value of lambda=0 should prevent regularization, but you still have to make sure that the inputs are not being standardized.

If you want to do regularized GLM in R check out the glmnet package.

Write logistic model as log(p/(1-p))= g(x1,x2,...) where g(.) is liner function of Xs' coefficients and p is the sigmoid function. The difference of coefficients between machine learning (ML) and statistics are the calculation process. ML use gradient based approach and statistics use mathematical equation solving methods. They have different properties such that different result. For multicollinearity, ML focus on precisely prediction rather than the estimation of coefficient, there is rare discussion for multicollinearity in this fild. And multicollinearity is an important issue in statistics field, not only logistic regression but also many statistical model have many discussion for multicollinearity detection, so statistics approach can conclude more reasonable coefficient by its theoretical properties rather than ML approach. As my experience, if you need meaningful coefficients to represent some importance or effect, please choose statistical version because there are some good properties to support. If you need more precise prediction, you can choose many classification model rather than ML logistical model unless you really need know the relative importance between x's using coefficient when you encounter big data.

My guess is that Mahout uses some sort of simplified first-order method for optimisation (similar to the linear perceptron's delta rule) whereas R's classic GLM from statistics uses ML estimation. I'm suspecting this from the fact that they're using the word "train" instead of "estimate". It would be very uncommon (not to say weird) to use the word "train" for OLS or ML estimation.

I've also noticed that, when you see terms from classical statistics such as "linear regression" and "logistic regression" used in a machine learning context, the difference is always in the optimisation process, i.e. estimation vs training (again, a good example is the linear perceptron). Training algorithms in machine learning are supposed to be more robust than parameter estimation methods in parametric statistics because they favour raw computational power over mathematical accuracy, though, they both have their pros and cons.

Anyway, I think it's good to pose such questions so that researchers, employers, and employees alike, can finally start understanding the differences between a machine learning expert (specialised computer scientist) and a statistician (specialised mathematician). They definitely have overlapping ground, such as the topics discussed in this question, but they're certainly not overlapping fields.

My intuition is that your learning rate is too high, which could cause your model to constantly overshoot the minima. Try out your code with small values of rate, maybe 0.05, to begin with and check if it tallies with your R output. A detail that may be relevant is confirming if the model is predicting the probability of 1 or 0. If R is predicting for 1's and Mahout for 0, then you could end up with coefficients of opposite signs.