I'm studying neural networks, and I'm trying to decide why the default choice of cost function for a single neuron seems to be quadratic loss: $$\sum_i(y_i-f_i)^2,$$
as per logistic regression. Where both $f_i$ and $p_i$ are the sigmoid (activation) function.
I understand that the neuron does not classify per say but instead modulates/dampens the output based on its activation function, so that when many neurons are connected together to form the network, classification only needs to be done on the output node using some cut-off value.
Nevertheless, if classification is our goal, it seems like the right choice of cost function should be that which relates the strength of the firing directly to the probability that it would fire if it were only firing at total strength or not at all. And doing this in such a way that if we ran the neuron many times using this probability as the probability that it would fire, then the expected value of the neuron firing would equal the strength with which it fires every time upon minimizing our cost function. (sorry I may have worded that poorly)
Is there a reason then that the logistic regression cost function is not the default choice?