I am currently working on a presentation on PCA as part of a seminar programme and one slide is about giving an interpretation on the first constructed Principal Component on an arbitrary two-dimensional dataset. In one of my reference books called "An Introduction to Statistical Learning with Applications" by Gareth James, Daniela Witten, Trevor Hastie, Robert from 2015, the authors give an alternative interpretation on p. 379 regarding the first PC (or PCs all together in a generalized form) as being
"the line in the p-dimensional space that is closest to the n observations (using average squared Euclidean distance as a measure of closeness)".
I assumed that this would mean that adding the first PC line to a two-dimensional scatterplot of my data, which was obtained from PCA in R on my scaled and centered original dataset, would be equivalent to adding the regression line from a simple linear regression on the scaled and centered dataset (= best linear approximation), but values/results in my example speak against it. In numbers, when I performed PCA with prcomp(x) in base R on my data, it yielded the loading vector $\phi_1 = (-0.7071068, -0.7071068)$ for the first PC, but a simple linear regression yielded a slope parameter of $\hat{\beta}_1 = 0.7843085$.
I would therefore have to ask whether there is something wrong with my approach/thinking, does this interpretation regarding PC only approximately hold true or did I mess up in coding? My code example:
ex.df <- structure(list(x = c(70.7, 82.5, 36.5, 97.2, 52.7, 56, 61.3,
59.6, 32.5, 33.6, 70.7, 31.3, 67, 47.6, 98.5, 64.4, 78.7, 71.4,
88.7, 83.4, 44.7, 44, 56.4, 69, 30.8, 54.5, 67.3, 56.7, 40.7,
36.7, 57.8, 84.7, 35.9), y = c(16, 23.2, 12.2, 18.2, 16.2, 13.5,
14.8, 17.3, 8, 11.7, 21.8, 10, 21.9, 13, 19.1, 14.7, 20.2, 22.1,
20.5, 16.7, 9.4, 12.2, 9.4, 18.4, 5.8, 15.8, 13, 16, 12.5, 8.1,
16.4, 14.9, 10.5)), row.names = c(NA, -33L), class = c("tbl_df",
"tbl", "data.frame"))
ex.df <- as.data.frame(scale(ex.df, center = T, scale = T))
apply(ex.df, 2, mean) # check for ~0
apply(ex.df, 2, var) # check for 1
ex.pca <- prcomp(ex.df)
ex.pca # first loading = (−0.7071068,−0.7071068)
coef(lm(y ~ x, data = ex.df))
Edit: Apparently, when performing PCA on a scaled dataset in the two-dimensional case, the "loading" will only reflect the eigenvectors as was described here: Identical loadings in a PCA, which makes the example useless. Nonetheless, it is still an interesting question in my opinion
