I would like to implement closed form of PPCA (Bishop, Tipping, 1999, Appendix A). In this paper they calculate $W$ in formula (15):


where $K_q$ is a matrix from eigenvalues $\lambda_i$ of covariance matrix as defined in formula (5).

On one hand I read here, that it is possible to calculate eigenvalues with

$\lambda_i = \frac{s_i^2}{n-1}$

Is it true, that principal values depend on number of samples $n$?

On other hand I found python implementation of closed PPCA on github, function __fit_ml, where it is written

    mu = np.mean(self.y, 1)[:, np.newaxis]
    [u, s, v] = np.linalg.svd(self.y - mu)
        ss = s[:self.q]
    ss = np.sqrt(np.maximum(0, ss**2 - self.prior_sigma))
    w = u[:, :self.q].dot(np.diag(ss))

where author is apparently calculating with

$\lambda_i = s_i^2$

which is completely different.


Here $s_i$ is diagonal element of singular values matrix.


I have tried numeric example

X = np.float64(np.array([
    [-1, -1],
    [-2, -1],
    [-3, -2],
    [1, 1],
    [2, 1],
    [3, 2]]))

n_samples = np.shape(X)[0]

X -= np.mean(X, axis=0)

pca = PCA(n_components=2)

cov_matrix = np.dot(X.T, X) / n_samples

singularvalues = pca.singular_values_

for i in range(len(pca.components_)):
    eigenvector = pca.components_[i]
    eigenvalue = np.dot(eigenvector.T, np.dot(cov_matrix, eigenvector))
    singularvalue = singularvalues[i]
    eigenvalue2 = singularvalue**2 / n_samples
    print("%f\t%f;\t%f" % (eigenvalue, singularvalue, eigenvalue2))


6.616286    6.300612;   6.616286
0.050381    0.549804;   0.050381

and found that answer depends on covariance matrix normalization.


In the other hand it is written in Python's SVD documentation:

The rows of v are the eigenvectors of a.H a. The columns of u are the eigenvectors of a a.H. For row i in v and column i in u, the corresponding eigenvalue is s[i]**2.


Ah, sure, a.H a is "scatter matrix", i.e. covariance matrix w/o normalization.

  • $\begingroup$ In your reference at stats.stackexchange.com/questions/134282 you have misunderstood what $S$ is: it is not a covariance matrix. It is the singular part of the model matrix. $\endgroup$ – whuber Sep 5 '17 at 20:44
  • $\begingroup$ I was using notation from the paper. I see that S means different things in these 2 sources. $\endgroup$ – Dims Sep 5 '17 at 21:30
  • 1
    $\begingroup$ Covariance matrix has $1/n$ or $1/(n-1)$ factor right in the definition, see formula 5 in the linked paper. SVD does not. So clearly the formula has to be $\lambda = s^2/n$ or $s^2/(n-1)$ depending on whether one is using Bessel's correction. I don't know what is going on in the __fit_ml code, but $\lambda=s^2$ cannot possibly be correct. $\endgroup$ – amoeba says Reinstate Monica Sep 6 '17 at 7:33
  • $\begingroup$ @amoeba what if $S$ was defined without $1/N$? $\endgroup$ – Dims Sep 6 '17 at 8:25
  • 1
    $\begingroup$ OK, eigenvalues of the scatter matrix are of course $s^2$. (I added this to my previous comment.) $\endgroup$ – amoeba says Reinstate Monica Sep 6 '17 at 8:30