9
$\begingroup$

I observed that Caffe (a deep learning framework) used the Softmax Loss Layer SoftmaxWithLoss as output layer for most of the model samples.

As far as I know, Softmax Loss layer is the combination of Multinomial Logistic Loss Layer and Softmax Layer.

From Caffe, they said that

Softmax Loss Layer gradient computation is more numerically stable

However, this explanation is not the answer that I want, the explanation is just compare the combination of Multinomial Logistic Loss Layer and Softmax Loss layer instead of layer by layer. But not compare with other type of loss function.

However, I would like to know more what is the differences/advantages/disadvantages of these 3 error function which is Multinomial Logistic Loss, Cross Entropy (CE) and Square Error (SE) in supervised learning perspective? Any supportive articles?

$\endgroup$
  • 1
    $\begingroup$ Just a hint: I guess you will get a faster answer if you add the tag "caffe" to your questions. Also posting it on stackoverflow instead of stackexchange might give it more attention). $\endgroup$ – mcExchange Sep 9 '15 at 7:47
  • 1
    $\begingroup$ The combination makes the gradient easy to compute, just y-t. willamette.edu/~gorr/classes/cs449/classify.html $\endgroup$ – Jingpeng Wu Sep 16 '15 at 18:20
11
$\begingroup$

In my opinion, loss function is the objective function that we want our neural networks to optimize its weights according to it. Therefore, it is task-specific and also somehow empirical. Just to be clear, Multinomial Logistic Loss and Cross Entropy Loss are the same (please look at http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression). The cost function of Multinomial Logistic Loss is like this $J(\theta) = -\frac{1}{m} \left[ \sum_{i=1}^m y^{(i)} \log h_\theta(x^{(i)}) + (1-y^{(i)}) \log (1-h_\theta(x^{(i)})) \right].$

It is usually used for classification problem. The Square Error has equation like $\frac 1 {2N} \sum_{i=1}^N \| x^1_i - x^2_i \|_2^2.$

Therefore, it is usually used for minimize using some construction errors.

EDIT: @MartinThoma The above formula of multinomial logistics loss is just for binary case, for general case, it should be $J(\theta) = -\left[ \sum_{i=1}^{m} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log P(y^{(i)} = k | x^{(i)} ; \theta) \right]$, where K is number of categories.

$\endgroup$
  • 2
    $\begingroup$ In Caffe, MultinomialLogisticLoss is $\frac{-1}{N}\sum_{n=1}^{N}\log(p_{n,l_n})$, so who's wrong here? $\endgroup$ – moi Jun 6 '16 at 18:45
  • $\begingroup$ Not wrong, $y^i$ is binary variables, in the end, it can be reduced into your formulation. $\endgroup$ – beahacker Jun 8 '16 at 1:49
  • $\begingroup$ I thought multinomail logistic loss was without the second summand, so $J(\theta) = - \frac{1}{m} [\sum_{i=1}^m y^{(i)} \log h_\theta(x^{(i)})]$ $\endgroup$ – Martin Thoma Dec 26 '16 at 12:14
  • 1
    $\begingroup$ @MartinThoma My formula is just for binary case, for general case, it should be $J(\theta) = -\left[ \sum_{i=1}^{m} \sum_{k=1}^{K} 1\left\{y^{(i)} = k\right\} \log P(y^{(i)} = k | x^{(i)} ; \theta) \right]$ $\endgroup$ – beahacker Dec 28 '16 at 5:34
  • $\begingroup$ @beahacker Could you please tell me why the second summand isn't included in multinominal case as pointed out by Martin Thoma. I'm trying to understand why is it done like that. At-least could you point me to some resource to look into. $\endgroup$ – Nandeesh Jun 29 '18 at 8:41
2
$\begingroup$

I would like to know more what is the differences/advantages/disadvantages of these 3 error function which is Multinomial Logistic Loss, Cross Entropy (CE) and Square Error (SE) in supervised learning perspective?

The multinomial logistic loss is acturally the same as cross entropy. Look at this function(the cost function in softmax): $$ J( \theta ) = - \sum^m_{i=1} \sum^K_{k=1} 1 \{ y^{(i)} = k \} \log p(y^{(i)} = k \mid x^{(i)} ; \theta) $$ where m is the sample number, K is the class number.

The indicator function($1 \{ y^{(i)} = k \}$) determines whether the $p(x)$ bellow is 0 or 1 in the cross entropy definition, which is labelled as one hot in the training data, and $ p(y^{(i)} = k \mid x^{(i)} ; \theta) $ is the conditional likelihood of the softmax(q(x) as shown bellow). $$-\sum_x p(x) \log q(x)$$

And MSE is mostly for the situation where the link function is the unity function(the response distribution follows a normal distribution), the standard linear regression, while the cross entropy is normally for where the link function is the logit function. Here is an awesome comparison you can refer to.

Any supportive articles?

Except those in the links, recommend you this illustrating one: https://github.com/rasbt/python-machine-learning-book/blob/master/faq/softmax_regression.md

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.