Let $\Phi = \begin{bmatrix} 1 & \varphi_1(x_1) & \varphi_2(x_1) & \dots & \varphi_{M-1}(x_1) \\ 1 & \varphi_1(x_2) & \varphi_2(x_2) & \dots & \varphi_{M-1}(x_2) \\ \vdots \\ 1 & \varphi_1(x_N) & \varphi_2(x_N) & \dots & \varphi_{M-1}(x_N) \end{bmatrix}=\begin{bmatrix}\phi(x_1)^T\\ \vdots\\ \phi(x_N)^T\end{bmatrix}$

where $\varphi_m(x)$ is the $m^{th}$ basis function for regression

$y = 1 + \sum_{m=1}^{M-1}w_m\varphi_m(x)$

How can I prove that


where $k$ is the equivalent kernel for Bayesian linear regression. $w_m$ are the regression coefficients whose MLE solutions are already included in the equivalent kernel.


Okay. Looks like I have done a lousy job describing my question. My apologies. My question is really simple and it does not need more context than what I have already described. All I want to prove is given a simple design matrix of say for example a ${M-1}^{th}$ order polynomial regression $\Phi=$

$\begin{bmatrix} 1 & x_1 & x_1^2 & \dots & x_1^{M-1} \\ 1 & x_2 & x_2^2 & \dots & x_2^{M-1} \\ \vdots \\ 1 & x_N & x_N^2 & \dots & x_N^{M-1} \end{bmatrix}$

which can be written in a more compact form

$\Phi=\begin{bmatrix}\phi(x_1)^T\\ \vdots\\ \phi(x_N)^T\end{bmatrix}$

where $\phi(x_n)^T$ is the $n^{th}$ row of $\Phi$, $\phi(x_n)^T=[1,x_n,...,x_n^{M-1}]$. So each row of $\Phi$ is a mapping of an input x to an M-dimensional feature space which is spanned by the columns of $\Phi$. Also given a new $x^*$ with its corresponding mapping to the M-dimensional space, $\phi(x^*)^T=[1,x^*,...,{x^*}^{M-1}]$,prove that the following matrix operation yields a scalar 1.


  • 1
    $\begingroup$ If it is a probability density, that might be an axiom. $\endgroup$ – EngrStudent - Reinstate Monica Nov 5 '17 at 1:09
  • $\begingroup$ You need to give much more context than that. $\endgroup$ – Yair Daon Nov 5 '17 at 2:23
  • $\begingroup$ @YairDaon Sorry about that. I have added more detail $\endgroup$ – Karlsson Yu Nov 5 '17 at 5:10
  • $\begingroup$ What do you mean by "$k$"? What are the "$\phi_n$"? $\endgroup$ – whuber Nov 5 '17 at 17:33
  • $\begingroup$ @whuber $k$ is the equivalent kernel, $\phi(x)$ is a column vector of the basis functions $\endgroup$ – Karlsson Yu Nov 5 '17 at 23:36

Never mind, I think I proved it. Btw, the proof for Eq.3.64 provided by Bishop in the Machine learning and pattern recognition might not be correct.

$\sum_{n=1}^Nk(x,x^n)=\sum_{n=1}^N\phi(x)^T(\Phi^T\Phi)^{-1}\phi(x_n)$ can be rewritten as

$\sum_{n=1}^N\phi(x)^T(\Phi^T\Phi)^{-1}\phi(x_n)\times 1=\phi(x)^T(\Phi^T\Phi)^{-1}\Phi^T1_N$

let $\Phi^T1_N = a_1$. In fact $a_1$ is the first column of $A=\Phi^T\Phi$

Using the fact that $A^{-1}A=I=[e_1,...e_M]$, we have $A^{-1}a_1=e_1$

Therefore, $\sum_{n=1}^Nk(x,x_n)=\phi(x)^T(\Phi^T\Phi)^{-1}\Phi^T1_N=\phi(x)^Te_1=1$, according to the definition of $\Phi$, $\phi(x)^T=[1,\varphi_1(x),...,\varphi_{M-1}(x)]$

This proof only holds when there is always a bias term of 1 added to the Bayesian linear regression function

  • $\begingroup$ I think your proof might be wrong in several ways but it is so hard to tell with (still) so little context. $\endgroup$ – Yair Daon Nov 6 '17 at 2:57
  • 1
    $\begingroup$ This is all getting a bit of a mess. Why not start again with a new question with all the context and link it back to here? $\endgroup$ – mdewey Nov 6 '17 at 9:58
  • $\begingroup$ @YairDaon I have added more explanation of the problem $\endgroup$ – Karlsson Yu Nov 6 '17 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.