1
$\begingroup$

A quick perplexity popped up in my mind while reading about non-parametric linear regression.

In linear regression, we model our response $\textbf{y} \sim \mathcal{N}(X\beta, \sigma^2I)$ so basically we try to estimate a linear function of the form

$$f_\beta(\textbf{x}_i) =\textbf{x}_{i,1}\beta_1, \dots, \textbf{x}_{i,p}\beta _p$$

while in non-parametric regression we allow more possibilities for the structure of $f$ and the response is modeled as

$$\textbf{y} \sim \mathcal{N}(f(x), \sigma^2I)$$

with $f$ respecting some smoothness assumptions.

What it's not too clear to me is what is the main difference between kernel linear regression and non-parametric one. It is well known that the word linear in linear regression refers to the parameters, so one typically applies a non-linear feature transformation $\phi : \mathbb{R}^p \rightarrow \mathbb{R}^d$ to the features and then searches for some hyperplane fitting the data (brought in higher dimension by the map $\phi$).

$\endgroup$
3
  • $\begingroup$ Your characterizations of linear regression and non-parametric regression are not what one would expect: they both appear to be (extremely) limited ones. Thus, reviewing some of our posts about non-parametric models might resolve this question for you. $\endgroup$
    – whuber
    Oct 4, 2021 at 12:22
  • $\begingroup$ Aren’t they exactly what is meant for linear and non parametric regression? Could you provide more details please? $\endgroup$ Oct 4, 2021 at 12:35
  • $\begingroup$ "Non-parametric" refers to distributional assumptions. Your characterization would correspond either to semi-parametric regression (if a wide set of possible $f$ is contemplated) or a parametric regression (if $f$ is finitely parameterized). Your characterization of linear regression is unusually restrictive in specifying Normal responses. $\endgroup$
    – whuber
    Oct 4, 2021 at 12:38

1 Answer 1

3
$\begingroup$

A parametric model has fixed number of parameters, in case of non-parametric model, the number of parameters grows with the size of the data. What follows, with a parametric model we need to make stronger assumptions about the distribution of the data, while in case of the non-parametric model, it is "learned from the data" to greater degree, but the practical differences may be blurry in some cases. That is why models such as Gaussian processes are considered as non-parametric, no matter that they make distributional assumptions and have parameters.

Kernel regression is one of the non-parametric regression models, so it cannot differ from non-parametric models. It is a model that uses kernels to approximate the expected value of the distribution of the data. Other non-parametric models may use different ways of achieving this, for example in case of $k$-NN regression the predicted mean would be just an average of the $k$ closest neighbors of the datapoint.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.