Confused about PCA transformation vectors I'm trying to get the intuition of how PCA works. So far I understood that:


*

*I start from the input matrix $X = [X_{1},...,X_{p}]$  where each $X_{i}$ is composed by $n$ elements that are the $n$ observations for those features ($X_{i}$, in fact) and therefore $X$ is a ($n$ x $p$) matrix.

*In order to transform my starting problem into a lower-dimensional one I must define a transformation and thus transformation vectors like:
$w_{(k)} = (w_{1},...,w_{p})_{(k)}$ 
that are $p$-dimensional vectors.

*I must compute the 1st PCA vector by means of:
$w_{1} = argmax _{||w||=1} {1 \over m} \sum_{i=1}^m [(x_{i}w^2)]$.

*The other PCA vectors, in general, will be computed as:
$w_{k} = argmax _{||w||=1} {1 \over m} \sum_{i=1}^m [(x_{i}-\sum_{j=1}^{k-1}x_{i}w_{j}w_{j}^T)w]^2$ 
(in brief: for the k-th PCA vector I have to subtract all other components to the data input matrix in order to pick the feature with the higher variance).
My questions now are: 


*

*are those $w_{(k)}$ vectors composed by just all zeros except for that element that corresponds to the feature I want to consider? I mean: since I know that $w_{(k)}$ are unit vectors (lenght one) then they have to be composed by all zeros except for one component that will be 1. Is this 1 used to pick from the associated $x_{i}$ the feature I want to consider?

*Are $w_{(k)}$ vector of dimension $(1$ x $p)$?

 A: 
  
*
  
*are those $w_{(k)}$ vectors composed by just all zeros except for that element that corresponds to the feature I want to consider? I mean:
  since I know that $w_{(k)}$ are unit vectors (lenght one) then they have to
  be composed by all zeros except for one component that will be 1. Is
  this 1 used to pick from the associated $x_i$ the feature I want to
  consider?
  

For a vector $v = [v_1, ..., v_p]$, being a unit vector means that $v$ has length 1, i.e. $\|v\| = \sqrt{\sum_{i=1}^{p} v_i^2} = 1$. It doesn't mean that $v$ contains a single 1 and all other elements 0 (although that is one example of a unit vector).
You can think of the weight vectors $w_{(k)}$ as being the directions of maximal variance in your data space. For example, imagine you had an Gaussian/elliptical cloud of data points in a 2d space, centered at the origin. The first principal component would point in the direction along which the ellipse is longest. The second principal component would point in the orthogonal direction. The weight vectors will always be orthogonal to each other. So, the only way your weight vectors would contain a single one and the remaining elements zero is if the directions of maximal variance are parallel to the existing data axes.
You can think of PCA as a way to find the directions of maximal variance in the data, rotate the data s.t. these directions are aligned with the axes, then discard the dimensions along which the variance is small.


  
*Are $w_{(k)}$ vector of dimension $(1 \times p)$?
  

In your setup (where columns of the data matrix correspond to features/dimensions), each $w_i$ will be a column vector, size $(p \times 1)$
A couple other points:


*

*You're describing an iterative way to compute PCA. This is good for getting an intuition for what PCA is doing. In practice, you'd generally perform the computation another way. For example, you can solve for all weight vectors simultaneously as the eigenvectors of the covariance matrix of the data. This can be faster and more numerically stable than the iterative method you're describing, depending on the situation. There are many other ways to do it too.

*Probably a typo: In your step (3), the square should be on the outside of the parentheses (you want the square of the projection of $x_i$ onto $w$, not the projection of $x_i$ onto $w^2$)
