I asked Why is the regularization penality equal to $\langle c , Kc \rangle_{R^n}$ when using the kernel trick in Tikhonov regularization? and got good answers, however, there is still one detail that does not quite make sense to me. Why is the norm when using Tikhonov regularization equal to:
$$ \| w \|^2_{\mathbb R^d} = x^T x$$
and not equal to:
$$ \langle c, X X^T c \rangle_{\mathbb R^d} = c X X^T c?$$
My intuition tells me that this should be a trivial question, embarrassingly I am unable to resolve it.
Usually in Tikhonov regularization we have the norm of a funciton be $\| f \|^2_{\mathcal H} = \langle f, f \rangle_{\mathcal H}$. Using the feature map $\Phi$ and the kernel trick $w = \Phi^T c$ one can show:
$$ \| f \|^2_{\mathcal H} = \langle f,f \rangle_{\mathcal H} = \langle w,w \rangle = \langle \phi^Tc,\Phi^Tc \rangle = \langle c,\Phi \Phi^Tc \rangle = \langle c,Kc \rangle .$$
In the case of linear functions we have:
$$ f(x) = \langle w, x \rangle = \sum^D_{d=1} w_i x_i .$$
If that is the case, then why don't we have as the norm in standard Tikhhonov regularization:
$$ \| f \|^2_{\mathcal H} = \langle f,f \rangle_{\mathcal H} = \langle w,w \rangle = \langle X^Tc,X^Tc \rangle = \langle c,X X^Tc \rangle? $$
Wouldn't that be a more consistent view of Tikhonov regularization?
