1
$\begingroup$

I'm trying to find if two hidden neurons in RBF Network overlap with each other or not? It's an online classification problem, it means data come to our network one-by-one and then discard completely. so I already know the mean and covariance of each neuron. I want to measure the distance between every two neurons. as long as this network is RBF and each neuron show a distribution, I want to use KL divergence as a measure of distance. ( I know KL can't be a measure, because the KL(P||Q) is not equal with KL(Q||P), but assume that I fix this problem.)

I read this question about how to calculate multivariate Gaussians KL divergence and I use the formula: $$ \begin{aligned} KL = \frac{1}{2}\left[\log\frac{|\Sigma_2|}{|\Sigma_1|} - d + \text{tr} \{ \Sigma_2^{-1}\Sigma_1 \} + (\mu_2 - \mu_1)^T \Sigma_2^{-1}(\mu_2 - \mu_1)\right]. \end{aligned} $$ I also implemented the matlab code as below.

ans1=log(norm(CovarianceMatrix(:,:,NrSNumberindx(i)))/norm(CovarianceMatrix(:,:,NrS)))-(size(A,2))+trace(pinv(CovarianceMatrix(:,:,NrSNumberindx(i)))*CovarianceMatrix(:,:,NrS));
ans1=ans1+(CenterTem-CenterNeuron(NrS,:))*CovarianceMatrix(:,:,NrSNumberindx(i))*(CenterTem-CenterNeuron(NrS,:))';
ans1=ans1*0.5;

I can't find any implementation problem in my code. But it gives me a negative value for KL which we already know, KL is always positive.

I trace the code. At the first of training, covariance matrix of data is equal to zero so I add the epsilon to each element of the covariance matrix. But still trace of these values are really small and if the dimensions of data are huge then the value of (size(A,2)) is huge too.

So the trace is a small value, the log is equal to zero (because each value of the covariance matrix is equal to epsilon) and value of (size(A,2)) is large and negative, then the result of KL become negative which doesn't have any meaning!!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

ans1=log(norm(CovarianceMatrix(:,:,NrSNumberindx(i)))/norm(CovarianceMatrix(:,:,NrS)))

The formula says $ \ln { | \Sigma_2 | \over | \Sigma_1 | } $ to mean the determinant, not the norm of the matrices.

ans1=ans1+(CenterTem-CenterNeuron(NrS,:))*CovarianceMatrix(:,:,NrSNumberindx(i))*(CenterTem-CenterNeuron(NrS,:))';

The formula says $(\mu_2 - \mu_1)^\intercal \Sigma_2^{-1} (\mu_2 - \mu_1)$, but you seem to have forgotten to invert the covariance matrix.

Does it work once you fix the errors above?

Improve this answer
$\endgroup$
6
  • $\begingroup$ your right about the problems.thanks. but can you help me more? when I use det the value of the log part is equal to NAN because all covariance matrix elements are equal and determinant of these value is equal to zero. so we have (zero/zero) which is NAN $\endgroup$
    – mkafiyan
    May 29, 2019 at 7:44
  • $\begingroup$ If the covariance matrix is all zeroes, it means that there is no variance in the data at all. (If the data was orthogonal, you would at least get non-zero diagonal values!) Constant data is, indeed, not multivariate normal, so trying to use it as such leads to errors. Why do your neurons have constant data? Shouldn't it mean that such neurons have no effect on the result? $\endgroup$
    – aitap
    May 29, 2019 at 9:58
  • $\begingroup$ No it's not like that. the network is self-struct. it means it can add new neuron or delete neuron by itself. data come to network one-by-one. center and covariance matrix of each neuron which is more close to data are changing. I just initialize covariance matrix with zero. it changes with data $\endgroup$
    – mkafiyan
    May 29, 2019 at 13:44
  • $\begingroup$ I am not an expert on RBF networks, but perhaps initializing the covariance matrix with a unity matrix instead of zero matrix may help? $\endgroup$
    – aitap
    May 30, 2019 at 15:05
  • $\begingroup$ I changed it to a unity matrix, but it has an effect on accuracy, not in a good way. $\endgroup$
    – mkafiyan
    Jun 3, 2019 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.