# Principal component analysis in two dimensions

During my studies, I stumbled upon the following exercise:

We have the following joint probability distribution: $$p(x,y) = p(x) p(y|x)$$ $$p(x) = \mathcal{N}(0,1), p(y \mid x) = \frac{1}{2} \delta(y -x) + \frac{1}{2} \delta(y+x)$$ where $$\delta(\cdot)$$ is the Direc delta function. The exercise then asks to find the principal components of $$p(x,y)$$. It is hinted that this is equivalent to finding the parameters $$\theta \in [0, 2 \pi [$$ that maximize the variance of the projected data: $$z(\theta) = x \cos(\theta) + y \sin(\theta)$$, since in linear component analysis for two-dimensional probability distributions the set of possible directions to look for in $$\mathbb{R}^2$$ is given by: $$\{ \begin{pmatrix} \cos(\theta) \\ \sin(\theta) \end{pmatrix}, 0 \leq \theta \leq 2\pi \}$$.

Usually I would take the Lagrangian and the derive the maximum, but I don't know how the Langrangian would look like in this case. How would I go about solving this?

• For any $\theta$, variance of $z$ is $1$ – gunes May 7 '19 at 13:45
• Thanks! Could you elaborate on why that is the case? – MLStudent May 7 '19 at 13:48

Since $$E[z(\theta)]=0$$, we have $$\operatorname{var}(z(\theta))=E[z(\theta)^2]=E[\cos(\theta)^2x^2+\sin(\theta)^2y^2+2\cos\theta\sin\theta xy]$$ Here, by definition, $$E[x^2]=E[y^2]=1$$. So, the expression reduces to: $$\operatorname{var}(z(\theta))=1+\sin(2\theta) E[xy]$$ We also have $$E[xy]=E[E[xy|x]]=E[xE[y|x]]=0$$, which means $$\operatorname{var}(z)=1$$. This is an interesting dataset, the samples follow $$y=x$$ and $$y=-x$$ lines but we can't find these axes using this method. More importantly, $$x$$ and $$y$$ turns out to be uncorrelated.