I am quite a novice to statistics and am currently fighting my way through the "Neural Networks and Machine Learning" Book by Haykin. (p.516-518) In the discussion about independent component analysis in chapter 10:

Starting from

$\bf Y = WX$

X = observable, W demixing matrix, Y = demixer output

as an input output relationship and individual components are assumed to be independent. And a probability density function is defined parameterised by a demixing matrix W:

$p_Y=(\bf{y}, \bf{W})$

and the corresponding factorial distribution as:

$\tilde{p_Y}(Y) = \prod_{i=1}^m \tilde p_{Y_i}(y_i) $

Then it says in the book that the factorial distribution is not parameterized for "obvious reasons". (also am I right in thinking that the factorial distribution exists since we assumed statistical independence and hence $P(a \cap b) = P(a)P(b)$)

Now I am not sure if I understand the whole ICA right, but I think that this obvious reason is that the factorial distribution is supposed to represent the "real" output without any noise etc, so that is why it is assumed to not be parameterized by W?

Which brings me to the next question, I have a feeling for what and why $p_Y$ is parmeterized by W but I don't actually understand it. I have an idea of what I am looking at if I read about a pdf parameterized with variance and mean etc, because then I can imagine a gaussian, but I am not sure how to think of it in this case.

Last step is to minemize the kullback-leibler divergence (KLD) between the two functions with respect to W.

Again my intuition here is that KLD is a measure for how difference the two distributions are and since we want to option the original output they should be as similar as possible.

I really have a feeling that I am jumping through the dark here and basing my understanding on very wobbly grounds, unfortunately I have no-one to ask whether I am heading the right direction and would appreciate it if someone here could be able to verify or point out errors in my thinking and understanding.

  • $\begingroup$ What book are you talking about? I can't find that one by Hinton $\endgroup$ – Jakub Bartczuk Oct 11 '17 at 14:28
  • 1
    $\begingroup$ Oh dear my mistake, I was watching a lecture by Hinton so I that was in the back of my mind. Its not Hinton its Haykin! Really sorry about that. I will edit it and fix it. $\endgroup$ – SandraK Oct 11 '17 at 14:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.