1
$\begingroup$

Let us consider the linear regression model in finite dimensions given by $Y = X \beta + \epsilon$ where $Y \in \mathbb{R}^n, X \in \mathbb{R}^{n \times m}, \beta \in \mathbb{R}^m$, and $ \epsilon \in \mathbb{R}^n$ is the Gaussian noise. I know that to compute the loss function, using the $\ell^2$ or $\ell^p$ error for finite-dimensional spaces is used to measure the misfit.

I am wondering if other norms from functional analysis can be used for linear regression such as the sobolev norms or negative sobolev norms adapted to the finite-dimensional setting.

Is there any literature on this topic? Would it be too overkill to use other types of norms instead of the $\ell^2$ norm for the misfit?

Comments appreciated!

$\endgroup$
9
  • $\begingroup$ This question strikes me as needing some additional information or assumptions. The Sobolev norms involve derivatives. What do you hope the analog of a derivative would be in a finite dimensional space? $\endgroup$
    – whuber
    Dec 14, 2020 at 17:49
  • $\begingroup$ At the top of my head, the derivatives can be discretized using finite differences. Does this help? I'm primarily seeking references and intuition. $\endgroup$
    – secondrate
    Dec 14, 2020 at 19:55
  • $\begingroup$ Such discretization would appear to introduce nothing new. You seem to be in pursuit of a question rather than having any particular question to ask. $\endgroup$
    – whuber
    Dec 14, 2020 at 20:18
  • $\begingroup$ This question came about because I was curious how various norms on the misfit are sensitive to the noise $\epsilon$ for linear regression. Do you have any references on this? I have not encountered this while reading elements of statistical learning. $\endgroup$
    – secondrate
    Dec 14, 2020 at 20:24
  • $\begingroup$ Perhaps the mention of Sobolev norms has distracted from the main point of using norms that aren't $l^p$ norms (perhaps even metrics that don't come from norms). $\endgroup$
    – Dave
    Dec 14, 2020 at 20:26

1 Answer 1

0
$\begingroup$

Minimizing a norm other than $\ell^2$ is, in some sense, just some other extremum estimator. The OLS estimator is an extremum estimator because the estimated parameters are the points giving the minimum of $\vert\vert y - \hat y\vert\vert_2$. If you want to have the objective function of some other extremum estimator to be some other norm $\vert\vert y - \hat y\vert\vert_{\text{other}}$, go for it.

$$ \hat\beta_{\text{ols}} = \underset{\hat\beta}{\arg\min}\{ \vert\vert y - X\beta \vert\vert_2 \}\\ \hat\beta_{\text{other}} = \underset{\hat\beta}{\arg\min}\{ \vert\vert y - X\beta \vert\vert_{\text{other}} \} $$

For instance, minimizing the $\ell^1$ norm leads to at the median. Depending on the situation, this can give a better estimate of the mean than minimizing the $\ell^2$ norm gives. Consequently, more than just $\ell^2$ minimization is useful. Getting away from $\ell^p$ norms, minimizing a weighted norm corresponds to weighted least squares, and a similar idea should apply for generalized least squares, so it is not just $\ell^p$ norms whose minimizations find use in statistics.

Regarding Sobolev norms in particular, I do not see a way for that to make sense, since a Sobolev norm involves derivatives of the function in the function space.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.