Linear Kernel in Baysian Linear Regression I came up with http://mlg.eng.cam.ac.uk/duvenaud/cookbook/index.html and it is actually very useful. 
At some point it says 

If you use just a linear kernel in a GP, you're simply doing Bayesian
  linear regression, and good news! You can do this in time $O(N)$ instead
  of $O(N^3)$

Why is that?
 A: The equations for linear regression can be performed in primal form (e.g. the regular normal equations), where there is one parameter per input variable, or dual form (e.g. kernel ridge regression with a linear kernel) where there is one parameter per training example.  This means you can choose which form to use depending on which is more efficient, if $N >> d$ then use the normal equation, which is $O(N)$, if $d >> N$ then use kernel ridge regression with a linear kernel, where $d$ is the number of attributes.  Bayesian linear regression is to GP with a linear covariance function what linear regression is to kernel ridge regression with a linear kernel.
If you use a quadratic kernel, you can achieve the same result by replacing it with a linear kernel and adding additional attributes that are the pairwise products of all of the existing attributes.  You can then use the same trick to get O(N) complexity.  However this becomes impractical if d is too large as this causes a very rapid increase in the number of product terms.
