0
$\begingroup$

During my studies, I stumbled upon the following exercise:

We have the following joint probability distribution: $$p(x,y) = p(x) p(y|x)$$ $$p(x) = \mathcal{N}(0,1), p(y \mid x) = \frac{1}{2} \delta(y -x) + \frac{1}{2} \delta(y+x)$$ where $\delta(\cdot)$ is the Direc delta function. The exercise then asks to find the principal components of $p(x,y)$. It is hinted that this is equivalent to finding the parameters $\theta \in [0, 2 \pi [$ that maximize the variance of the projected data: $z(\theta) = x \cos(\theta) + y \sin(\theta)$, since in linear component analysis for two-dimensional probability distributions the set of possible directions to look for in $\mathbb{R}^2$ is given by: $\{ \begin{pmatrix} \cos(\theta) \\ \sin(\theta) \end{pmatrix}, 0 \leq \theta \leq 2\pi \}$.

Usually I would take the Lagrangian and the derive the maximum, but I don't know how the Langrangian would look like in this case. How would I go about solving this?

$\endgroup$
2
  • $\begingroup$ For any $\theta$, variance of $z$ is $1$ $\endgroup$ – gunes May 7 '19 at 13:45
  • $\begingroup$ Thanks! Could you elaborate on why that is the case? $\endgroup$ – MLStudent May 7 '19 at 13:48
0
$\begingroup$

Since $E[z(\theta)]=0$, we have $$\operatorname{var}(z(\theta))=E[z(\theta)^2]=E[\cos(\theta)^2x^2+\sin(\theta)^2y^2+2\cos\theta\sin\theta xy]$$ Here, by definition, $E[x^2]=E[y^2]=1$. So, the expression reduces to: $$\operatorname{var}(z(\theta))=1+\sin(2\theta) E[xy]$$ We also have $E[xy]=E[E[xy|x]]=E[xE[y|x]]=0$, which means $\operatorname{var}(z)=1$. This is an interesting dataset, the samples follow $y=x$ and $y=-x$ lines but we can't find these axes using this method. More importantly, $x$ and $y$ turns out to be uncorrelated.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.