# Why is the default cost function choice of a neuron quadratic loss?

I'm studying neural networks, and I'm trying to decide why the default choice of cost function for a single neuron seems to be quadratic loss: $$\sum_i(y_i-f_i)^2,$$

$$-\prod_ip_i^{y_i}(1-p_i)^{1-y_i},$$

as per logistic regression. Where both $f_i$ and $p_i$ are the sigmoid (activation) function.

I understand that the neuron does not classify per say but instead modulates/dampens the output based on its activation function, so that when many neurons are connected together to form the network, classification only needs to be done on the output node using some cut-off value.

Nevertheless, if classification is our goal, it seems like the right choice of cost function should be that which relates the strength of the firing directly to the probability that it would fire if it were only firing at total strength or not at all. And doing this in such a way that if we ran the neuron many times using this probability as the probability that it would fire, then the expected value of the neuron firing would equal the strength with which it fires every time upon minimizing our cost function. (sorry I may have worded that poorly)

Is there a reason then that the logistic regression cost function is not the default choice?

• From what I heard, the use of the quadratic loss is more due to historical reasons than from practical ones. Interesting question! :) Apr 9, 2014 at 21:10
• Sounds like your reference is either outdated, or specifically assuming regression. All kinds of different losses (square, log, hinge, ranking, etc) get used with ANNs and any decent modern treatment is going to discuss what loss functions are appropriate for a given problem.
– alto
Apr 10, 2014 at 16:59
• This depends on the problem, but we can roughly generalise by using negative log-likelihood in all problems. In regression, the NLL is the quadratic error (different by some constants). In binary and multi-class clf, the NLL is the cross-entropy. See chapter 5 of Pattern Recognition and ML by Bishop for a summary and details. Sep 18, 2021 at 8:29