2
$\begingroup$

I'm studying neural networks, and I'm trying to decide why the default choice of cost function for a single neuron seems to be quadratic loss: $$\sum_i(y_i-f_i)^2,$$

instead of:

$$-\prod_ip_i^{y_i}(1-p_i)^{1-y_i},$$

as per logistic regression. Where both $f_i$ and $p_i$ are the sigmoid (activation) function.

I understand that the neuron does not classify per say but instead modulates/dampens the output based on its activation function, so that when many neurons are connected together to form the network, classification only needs to be done on the output node using some cut-off value.

Nevertheless, if classification is our goal, it seems like the right choice of cost function should be that which relates the strength of the firing directly to the probability that it would fire if it were only firing at total strength or not at all. And doing this in such a way that if we ran the neuron many times using this probability as the probability that it would fire, then the expected value of the neuron firing would equal the strength with which it fires every time upon minimizing our cost function. (sorry I may have worded that poorly)

Is there a reason then that the logistic regression cost function is not the default choice?

$\endgroup$
  • $\begingroup$ From what I heard, the use of the quadratic loss is more due to historical reasons than from practical ones. Interesting question! :) $\endgroup$ – agarie Apr 9 '14 at 21:10
  • $\begingroup$ Sounds like your reference is either outdated, or specifically assuming regression. All kinds of different losses (square, log, hinge, ranking, etc) get used with ANNs and any decent modern treatment is going to discuss what loss functions are appropriate for a given problem. $\endgroup$ – alto Apr 10 '14 at 16:59
3
$\begingroup$

The cost function derived in logistic regression is quite similar to the cross-entropy cost function in NN terminology. The difference is that in cross-entropy they took a log and divided it by the number of samples.

The reason for the log is for the ease of getting the gradient, and dividing by a constant does not affect the learning in terms of gradient decent(Except for Plain gradient descent).

It is used quite often. The default-introduction of the quadratic loss in text books may be due to historical reasons.

$\endgroup$
  • $\begingroup$ Aha! I feel vindicated by your cross-entropy link. $\endgroup$ – Thoth Oct 27 '14 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.