0
$\begingroup$

I'm trying to implement cross-entropy as an error function in RBF neural networks instead of hinge loss error function. I need to find cross-entropy error for each output neuron, like hinge loss error function as you can see in the formula below

enter image description here

but when it comes to cross-entropy, it seems meaningless. because the formulation of cross-entropy returns 1 scaler. I look at some of the recent papers using cross-entropy and soft-max but can't find any relative information to my problem.

Is there any way to finding cross-entropy value for each neuron of output?

$\endgroup$
  • $\begingroup$ The given hinge loss formula also returns a single scalar. What exactly is the issue here? $\endgroup$ – jkm Jun 18 '18 at 9:27
  • $\begingroup$ @jkm assume n as the number of Classes so output layer has n output neuron. in the formula, I previously mentioned the j index, points output neurons and as you can see the formula calculate the error for each neuron. the output of equation is a vector $ {e_1, e_2 ,...e_n}$. the problem is I can't find any same formula for each output neuron when I use cross-entropy. $\endgroup$ – mkafiyan Jun 18 '18 at 9:38
  • $\begingroup$ ...just substitute the two conditional formulas in the bracket with a single standard cross-entropy formula over each j? $\endgroup$ – jkm Jun 18 '18 at 9:57
  • $\begingroup$ @jkm if I do so, it becomes meaningless because in the first condition of formula the multiplication must be more than one. but in cross-entropy, we use soft-max which is the normal distribution. assume that we change the first condition, the formula of cross-entropy is $-\sum_{j=1}^{n}y_j \log{\widehat{y}_j^~}$ . in this formula just one term of sum have to value cause $y_j$ is a vector with just an element with the value of one. so it seems meaningless to me if separate the error of each neuron. maybe I don't get correctly waht you exactly mentioned. $\endgroup$ – mkafiyan Jun 18 '18 at 10:14
  • $\begingroup$ @jkm do you mean using something like this : $\begin{cases}0 & y_j*\widehat{y}_j > 1\\z_j-y_j & other\end{cases}$ assume that z is softmax function ? $\endgroup$ – mkafiyan Jun 18 '18 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.