I know the Nadaraya-Watson kernel regression. What is new to me is the Kernel ridge regression from scitkit-learn's KernelRidge with kernel='rbf'. It mentions that the kernel trick is used, which is not done in the Nadaraya-Watson kernel regression, so it seems to me they are different concepts. Am I right, or are they the same afterall?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
6
Yeah, you are right. You practically replace the square matrix $X^TX$ with a Kernel $K$ when you estimate your coefficients.
-
3$\begingroup$ Can you elaborate? Kernel-Trick based methods are considered to be non-parametric, are they not? $\endgroup$– Make42Commented Nov 15, 2017 at 9:41
-
$\begingroup$ I dropped it since it seems to the distinction seems to be confusing. Parametric regression models, I would argue, establish a clear dependency of Y on X given by the coefficients (dimension of coefficients is finite). This is also what kernelized ridge regression does, since your $beta$ captures this. Non-parametric models do not establish this dependency in the same way. Reading your comment I am not sure if you still can say that KRR is parametric, because the dependency Y on X is not so clear anymore due to the kernel. $\endgroup$– SimonCommented Nov 15, 2017 at 10:00
-
$\begingroup$ With "beta", do you mean the factor before the regularization sum (attributing for the "ridge" in the name)? I would say this is a model-hyperparameter, but not a "standard" model-parameter (which a coefficient would be). Or do you mean a kernel-parameter, like a kernel scale parameter? Btw, my professor considers methods non-parametric if there is an arbitrary number of parameters. $\endgroup$– Make42Commented Nov 15, 2017 at 12:20
-
$\begingroup$ No, with beta I mean the coefficients you are estimating. The estimator for beta for classical ridge is $beta = (X^T X + alpha I)^{-1} X^T y$ which you can also solve like this: $beta = X^T (XX^T + alpha I)^{-1} y $. Beta is p-dimensional, as the number of covariates. In kernel ridge regression you replace $X X^T$ with a kernel. So, I would argue that this is still a parametric approach, if we take the definition of a parametric model being one that has coefficients with of finite dimensionality (p). However, since you operate in some high-dimensional feature space now, I might be wrong. $\endgroup$– SimonCommented Nov 15, 2017 at 14:55
-
$\begingroup$ Made an edit, but can't see it anywhere in my profile. Might still be hanging. $\endgroup$ Commented Apr 7, 2022 at 12:01
$\begingroup$
$\endgroup$
4
No, they are not the same as algorithms, though you might be able to find pairs of kernel where they give the same answer (model) in terms of predictors.
Kernel Regression (simplest form) is a density estimator with mean prediction: $$ \begin{align*} \mu_{\text{kernel-regression}} &= \sum_i w_i y_i, \quad w_i = \frac{K(X^*, X_i)}{\sum_j K(X^*, X_j)} \\ \end{align*} $$ while Kernel Ridge Regression is a regression (least-squares type inversion) with mean prediction: $$ \mu_{\text{kernel-ridge-regression}} = \sum_i w_i y_i, \quad w_i = \sum_j K(X^*, X_j) K_{ji}^{-1}(X, X) $$
And as noted before, the "kernel" part of kernel ridge regression is that you practically replace the square matrix $X^TX$ with a Kernel $K$ when you estimate your coefficients.
I have no idea why "kernel regression" means "kernel density esimation" ... if someone knows the history, or knows of complete results on mapping between the two methods please update this answer.
This is related: Is Kernel Regression similar to Gaussian Process Regression?
You can see discussiona about his in Kevin Murphy's (newer) book as well as the older Rasmussen book.
-
1$\begingroup$ Kernel Regression is a (supervised!) regression, kernel density estimation (KDE) is (unsupervised!) density estimation. They mean very different things. KDE does not have any target variable. $\endgroup$– Make42Commented Apr 8, 2022 at 10:43
-
$\begingroup$ @Make42 While it is true that KDE typically means estimate $P(X)$, when people say kernel regression, the usually mean estimating $P(Y|X)$. The aswer shows the two methods. The names are terrible but for historical reasons it is what we are stuck with. $\endgroup$ Commented Jul 14, 2022 at 9:01
-
$\begingroup$ Not to my knowledge, which is the following: Estimating the full $P(Y|X)$ is called "conditional density estimation". Estimating the mean of the conditional density $P(Y|X)$ might be what Kernel Regression does, but in fact, that is what all regression does - by definition. Historically, the term "regression" is a short form of "regression toward the mean". $\endgroup$– Make42Commented Jul 14, 2022 at 12:52
-
$\begingroup$ @Make42 Yes density estimation includes conditional density estimation which includes kernel regression. See en.wikipedia.org/wiki/Kernel_regression for example. $\endgroup$ Commented Jul 14, 2022 at 13:16