20
$\begingroup$

As I understand, a perceptron/single-layer artificial neural network with a logistic sigmoid activation function is the same model as logistic regression. Both models are given by the equation:

$F(x) = \frac{1}{1-e^{-\beta X}}$

The perceptron learning algorithm is online and error-driven, whereas the parameters for logistic regression could be learned using a variety of batch algorithms, including gradient descent and Limited-memory BFGS, or an online algorithm, like stochastic gradient descent. Are there any other differences between logistic regression and a sigmoid perceptron? Should the results of a logistic regressor trained with stochastic gradient descent be expected to be similar to the perceptron?

Improve this question
$\endgroup$
1
  • 2
    $\begingroup$ Looks like this question is similar, and it seems to contain better responses :) $\endgroup$
    – Ralph Tigoumo
    Commented May 7, 2016 at 9:34

3 Answers 3

Reset to default
0
$\begingroup$

You mentioned already the important differences. So the results should not differ that much.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ This does not provide an answer to the question. To critique or request clarification from an author, leave a comment below their post. $\endgroup$
    – Xi'an
    Commented Mar 1, 2015 at 17:14
  • 1
    $\begingroup$ Actually I tried to answer both questions: 1) "Are there any other differences between logistic regression and a sigmoid perceptron?" and 2) "Should the results of a logistic regressor trained with stochastic gradient descent be expected to be similar to the perceptron?" $\endgroup$
    – Michael Dorner
    Commented Mar 1, 2015 at 17:27
  • 7
    $\begingroup$ That's a reasonable position, @MichaelDorner. Would you mind expanding your answer a little to make that clearer? $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 1, 2015 at 17:44
4
$\begingroup$

There is actually a big substantial difference, which is related to the technical differences that you mentioned. Logistic regression models a function of the mean of a Bernoulli distribution as a linear equation (the mean being equal to the probability p of a Bernoulli event). By using the logit link as a function of the mean (p), the logarithm of the odds (log-odds) can be derived analytically and used as the response of a so-called generalised linear model. Parameter estimation on this GLM is then a statistical process which yields p-values and confidence intervals for model parameters. On top of prediction, this allows you to interpret the model in causal inference. This is something that you cannot achieve with a linear Perceptron.

The Perceptron is a reverse engineering process of logistic regression: Instead of taking the logit of y, it takes the inverse logit (logistic) function of wx, and doesn't use probabilistic assumptions for neither the model nor its parameter estimation. Online training will give you exactly the same estimates for the model weights/parameters, but you won't be able to interpret them in causal inference due to the lack of p-values, confidence intervals, and well, an underlying probability model.

Long story short, logistic regression is a GLM which can perform prediction and inference, whereas the linear Perceptron can only achieve prediction (in which case it will perform the same as logistic regression). The difference between the two is also the fundamental difference between statistical modelling and machine learning.

Improve this answer
$\endgroup$
3
$\begingroup$

I believe one difference you're missing is the fact that logistic regression returns a principled classification probability whereas perceptrons classify with a hard boundary.

This is mentioned in the Wiki article on Multinomial logistic regression.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.