I'm trying to get the intuition of how PCA works. So far I understood that:

1. I start from the input matrix $X = [X_{1},...,X_{p}]$ where each $X_{i}$ is composed by $n$ elements that are the $n$ observations for those features ($X_{i}$, in fact) and therefore $X$ is a ($n$ x $p$) matrix.
2. In order to transform my starting problem into a lower-dimensional one I must define a transformation and thus transformation vectors like:

$w_{(k)} = (w_{1},...,w_{p})_{(k)}$ that are $p$-dimensional vectors.

3. I must compute the 1st PCA vector by means of:

$w_{1} = argmax _{||w||=1} {1 \over m} \sum_{i=1}^m [(x_{i}w^2)]$.

4. The other PCA vectors, in general, will be computed as:

$w_{k} = argmax _{||w||=1} {1 \over m} \sum_{i=1}^m [(x_{i}-\sum_{j=1}^{k-1}x_{i}w_{j}w_{j}^T)w]^2$

(in brief: for the k-th PCA vector I have to subtract all other components to the data input matrix in order to pick the feature with the higher variance).

My questions now are:

1. are those $w_{(k)}$ vectors composed by just all zeros except for that element that corresponds to the feature I want to consider? I mean: since I know that $w_{(k)}$ are unit vectors (lenght one) then they have to be composed by all zeros except for one component that will be 1. Is this 1 used to pick from the associated $x_{i}$ the feature I want to consider?
2. Are $w_{(k)}$ vector of dimension $(1$ x $p)$?
• Tangential comment: For intuition, we can mention the big picture idea that although each point $X_i$ belongs to $\mathbb R^n$, and $n$ is very large, it often happens that all the points $X_i$ secretly belong to a low dimensional affine subspace. Finding a low dimensional affine subspace that contains the points $X_i$ (as closely as possible) is simply a linear algebra problem (and it's similar to the problem of finding an optimal low rank approximation of a given matrix, which is solved by the SVD). – littleO May 23 '16 at 9:36

1. are those $w_{(k)}$ vectors composed by just all zeros except for that element that corresponds to the feature I want to consider? I mean: since I know that $w_{(k)}$ are unit vectors (lenght one) then they have to be composed by all zeros except for one component that will be 1. Is this 1 used to pick from the associated $x_i$ the feature I want to consider?

For a vector $v = [v_1, ..., v_p]$, being a unit vector means that $v$ has length 1, i.e. $\|v\| = \sqrt{\sum_{i=1}^{p} v_i^2} = 1$. It doesn't mean that $v$ contains a single 1 and all other elements 0 (although that is one example of a unit vector).

You can think of the weight vectors $w_{(k)}$ as being the directions of maximal variance in your data space. For example, imagine you had an Gaussian/elliptical cloud of data points in a 2d space, centered at the origin. The first principal component would point in the direction along which the ellipse is longest. The second principal component would point in the orthogonal direction. The weight vectors will always be orthogonal to each other. So, the only way your weight vectors would contain a single one and the remaining elements zero is if the directions of maximal variance are parallel to the existing data axes.

You can think of PCA as a way to find the directions of maximal variance in the data, rotate the data s.t. these directions are aligned with the axes, then discard the dimensions along which the variance is small.

1. Are $w_{(k)}$ vector of dimension $(1 \times p)$?

In your setup (where columns of the data matrix correspond to features/dimensions), each $w_i$ will be a column vector, size $(p \times 1)$

A couple other points:

1. You're describing an iterative way to compute PCA. This is good for getting an intuition for what PCA is doing. In practice, you'd generally perform the computation another way. For example, you can solve for all weight vectors simultaneously as the eigenvectors of the covariance matrix of the data. This can be faster and more numerically stable than the iterative method you're describing, depending on the situation. There are many other ways to do it too.

2. Probably a typo: In your step (3), the square should be on the outside of the parentheses (you want the square of the projection of $x_i$ onto $w$, not the projection of $x_i$ onto $w^2$)

• Thank you for the answer. However, about your second point on the end, I'm looking at en.wikipedia.org/wiki/Principal_component_analysis and the square is on the outside of the parentheses actually. It was out there also in the case of 1st PCA vector I found above. – MatteoS93 May 23 '16 at 8:20