I know the Nadaraya-Watson kernel regression. What is new to me is the Kernel ridge regression from scitkit-learn's KernelRidge with kernel='rbf'. It mentions that the kernel trick is used, which is not done in the Nadaraya-Watson kernel regression, so it seems to me they are different concepts. Am I right, or are they the same afterall?


Yeah, you are right. You practically replace the square matrix $X^TX$ with a Kernel $K$ when you estimate your coefficients.

| cite | improve this answer | |
  • 1
    $\begingroup$ Can you elaborate? Kernel-Trick based methods are considered to be non-parametric, are they not? $\endgroup$ – Make42 Nov 15 '17 at 9:41
  • $\begingroup$ I dropped it since it seems to the distinction seems to be confusing. Parametric regression models, I would argue, establish a clear dependency of Y on X given by the coefficients (dimension of coefficients is finite). This is also what kernelized ridge regression does, since your $beta$ captures this. Non-parametric models do not establish this dependency in the same way. Reading your comment I am not sure if you still can say that KRR is parametric, because the dependency Y on X is not so clear anymore due to the kernel. $\endgroup$ – Simon Dirmeier Nov 15 '17 at 10:00
  • $\begingroup$ With "beta", do you mean the factor before the regularization sum (attributing for the "ridge" in the name)? I would say this is a model-hyperparameter, but not a "standard" model-parameter (which a coefficient would be). Or do you mean a kernel-parameter, like a kernel scale parameter? Btw, my professor considers methods non-parametric if there is an arbitrary number of parameters. $\endgroup$ – Make42 Nov 15 '17 at 12:20
  • $\begingroup$ No, with beta I mean the coefficients you are estimating. The estimator for beta for classical ridge is $beta = (X^T X + alpha I)^{-1} X^T y$ which you can also solve like this: $beta = X^T (XX^T + alpha I)^{-1} y $. Beta is p-dimensional, as the number of covariates. In kernel ridge regression you replace $X X^T$ with a kernel. So, I would argue that this is still a parametric approach, if we take the definition of a parametric model being one that has coefficients with of finite dimensionality (p). However, since you operate in some high-dimensional feature space now, I might be wrong. $\endgroup$ – Simon Dirmeier Nov 15 '17 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.