I am learning PCA, and while this seems like an obvious question, I can't seem to understand some of the ideas behind which eigenvectors to use when performing orthogonal regression with PCA. My code is below, and a diagram.
I am using a parametric model of M=[X.^2;X;Y] to estimate a quadratic with parameters [2;0.7;2]. I am performing eigenvalue decomposition on that. In doing so, I am finding out the directions of variance in the data, and the scale of such variance. PCA aims to find which features best describes the data?
So when using the below code, I find that estimating with the first two eigenvectors (directions) (V(:,1:2)') provides a good fit of the function (shown by the green circles in the diagram). However, I cannot get a parameter estimate ([2;0.7;2]) from this data (maybe you can, but I couldn't figure out how). So instead I looked at the final eigenvector (V(:,3)), and this gives me the parameters of my function (by taking V(1:2,3)/V(3,3)). Applying this results in an estimation as shown by the red crosses.
As you can see from the diagram, both visual fits are exact. My question is, how can I have estimated this function in two separate ways? I thought that if I used an eigenvector that did not have the smallest eigenvalue, then I would be finding a line in some other direction, not the main direction?
Any help anyone can give me would be appreciated, I think I understand how to apply PCA, just unsure as to this directionality thing.
% PCA on Quadratic using parametric model % Setup data X=linspace(-1,1,100); Y=2*X.^2+0.7*X+2; M=[X.^2;X;Y]'; % Normalize around centroid meanM=mean(M); MAdjust = M-repmat(meanM,100,1); % Perform eigenvalue decomposition [V D]= eigs(cov(MAdjust)); % Estimate function using V(:,1:2) FinalData1 = V(:,1:2)'*MAdjust'; MHat1=repmat(meanM,100,1)+(pinv( V(:,1:2))'*FinalData1)'; % Plot starting data, and prediction using V(:,1:2) figure;hold on; scatter3(M(:,1),M(:,2),M(:,3),15,'k'); scatter3(MHat1(:,1),MHat1(:,2),MHat1(:,3),50, 'g','d'); % Estimation function using V(:,3) c = meanM*(V(:,3)/V(3,3)); param = -V(1:2,3)/V(3,3) yhat = [X.^2;X;X.^0]'*[param ; c]; % Plot the estimated function scatter3(X.^2,X,yhat,50, 'r','+'); xlabel('X^2');ylabel('X');zlabel('Y');