1
$\begingroup$

The question comes from a paragraph in page 171 of "Pattern Recognition and Machine Learning" by Christopher M. Bishop:

enter image description here

Here $\mathbf\Phi$ is the design matrix for a data set of $N$ samples $$\mathbf\Phi=\left( {\begin{array}{*{20}{c}} \phi_0(\mathbf x_1)&\phi_1(\mathbf x_1)&\cdots&\phi_{M-1}(\mathbf x_1)\\ \phi_0(\mathbf x_2)&\phi_1(\mathbf x_2)&\cdots&\phi_{M-1}(\mathbf x_2)\\ \vdots&\vdots&\ddots&\vdots\\ \phi_0(\mathbf x_N)&\phi_1(\mathbf x_N)&\cdots&\phi_{M-1}(\mathbf x_N) \end{array}} \right)= \left( {\begin{array}{*{20}{c}} \mathbf\phi^T(\mathbf x_1)\\ \mathbf\phi^T(\mathbf x_2)\\ \vdots\\ \mathbf\phi^T(\mathbf x_N) \end{array}} \right)$$ where $\mathbf\phi=(\phi_0,\ldots,\phi_{M-1})^T$ is a vector of basis functions (it should be boldfaced but I don't know how to make it, even though I tried \mathbf or \bf).

So $$\mathbf\Phi^T\mathbf\Phi=\bigl(\mathbf\phi(\mathbf x_1),\mathbf\phi(\mathbf x_2),\ldots,\mathbf\phi(\mathbf x_N)\bigr) \left( {\begin{array}{*{20}{c}} \mathbf\phi^T(\mathbf x_1)\\ \mathbf\phi^T(\mathbf x_2)\\ \vdots\\ \mathbf\phi^T(\mathbf x_N) \end{array}} \right) =\sum\limits_{n=1}^N \mathbf\phi(\mathbf x_n)\mathbf\phi^T(\mathbf x_n).\tag{1}$$

In my understanding, the sum in the right side of (1) is the "implicit sum" mentioned in the paragraph. Each term $\mathbf\phi(\mathbf x_n)\mathbf\phi^T(\mathbf x_n)$ in the sum is a positive definite $M\times M$ matrix, so its eigenvalue is positive. If eigenvalues of these terms are addable, the resulting eigenvalues $\lambda_i$ of $\mathbf\Phi^T\mathbf\Phi$ is indeed "increasing with the size of the data set." However, it seems to me that the eigenvalues of two matrices are addable only if these matrices have the same eigenvectors. To see it, assuming $\mathbf A_1\mathbf u_1=\lambda_1\mathbf u_1$ and $\mathbf A_2\mathbf u_2=\lambda_2\mathbf u_2$, we have $(\mathbf A_1+\mathbf A_2)\mathbf u=(\lambda_1+\lambda_2)\mathbf u$ only if $\mathbf u_1=\mathbf u_2=\mathbf u$. But I cannot find any common eigenvector amongst all matrix terms in the right side of (1). So, the question is: why do eigenvalues $\lambda_i$ of $\mathbf\Phi^T\mathbf\Phi$ increase with the size $N$ of data set? Thank you.

$\endgroup$
1
  • $\begingroup$ The sum of eigenvalues equals the trace of a matrix. $$\sum_{k=1}^m \lambda_k = \sum_{k=1}^m \sum\limits_{n=1}^N \mathbf\phi_k(\mathbf x_n)\mathbf\phi_k^T(\mathbf x_n)$$ If you increase the size of $N$ then the diagonal elements will increase in size (I assume that these terms $ \mathbf\phi_k(\mathbf x_n)\mathbf\phi_k^T(\mathbf x_n)$ are positive; that part of your question I don't follow) $\endgroup$ Commented Oct 20, 2022 at 9:32

1 Answer 1

3
$\begingroup$

ignoring means, Gram=$\Phi^T\Phi$ is N * the empirical covariance matrix of $\phi$. So he is really just saying that as the empirical covariance matrix stabilises with N, the eigenvalues of Gram grow with N - it's a statistical argument not a mathematical one

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.