# linear regression using other norms

Let us consider the linear regression model in finite dimensions given by $$Y = X \beta + \epsilon$$ where $$Y \in \mathbb{R}^n, X \in \mathbb{R}^{n \times m}, \beta \in \mathbb{R}^m$$, and $$\epsilon \in \mathbb{R}^n$$ is the Gaussian noise. I know that to compute the loss function, using the $$\ell^2$$ or $$\ell^p$$ error for finite-dimensional spaces is used to measure the misfit.

I am wondering if other norms from functional analysis can be used for linear regression such as the sobolev norms or negative sobolev norms adapted to the finite-dimensional setting.

Is there any literature on this topic? Would it be too overkill to use other types of norms instead of the $$\ell^2$$ norm for the misfit?

• This question strikes me as needing some additional information or assumptions. The Sobolev norms involve derivatives. What do you hope the analog of a derivative would be in a finite dimensional space?
– whuber
Dec 14, 2020 at 17:49
• At the top of my head, the derivatives can be discretized using finite differences. Does this help? I'm primarily seeking references and intuition. Dec 14, 2020 at 19:55
• Such discretization would appear to introduce nothing new. You seem to be in pursuit of a question rather than having any particular question to ask.
– whuber
Dec 14, 2020 at 20:18
• This question came about because I was curious how various norms on the misfit are sensitive to the noise $\epsilon$ for linear regression. Do you have any references on this? I have not encountered this while reading elements of statistical learning. Dec 14, 2020 at 20:24
• Perhaps the mention of Sobolev norms has distracted from the main point of using norms that aren't $l^p$ norms (perhaps even metrics that don't come from norms).
– Dave
Dec 14, 2020 at 20:26

Minimizing a norm other than $$\ell^2$$ is, in some sense, just some other extremum estimator. The OLS estimator is an extremum estimator because the estimated parameters are the points giving the minimum of $$\vert\vert y - \hat y\vert\vert_2$$. If you want to have the objective function of some other extremum estimator to be some other norm $$\vert\vert y - \hat y\vert\vert_{\text{other}}$$, go for it.
$$\hat\beta_{\text{ols}} = \underset{\hat\beta}{\arg\min}\{ \vert\vert y - X\beta \vert\vert_2 \}\\ \hat\beta_{\text{other}} = \underset{\hat\beta}{\arg\min}\{ \vert\vert y - X\beta \vert\vert_{\text{other}} \}$$
For instance, minimizing the $$\ell^1$$ norm leads to at the median. Depending on the situation, this can give a better estimate of the mean than minimizing the $$\ell^2$$ norm gives. Consequently, more than just $$\ell^2$$ minimization is useful. Getting away from $$\ell^p$$ norms, minimizing a weighted norm corresponds to weighted least squares, and a similar idea should apply for generalized least squares, so it is not just $$\ell^p$$ norms whose minimizations find use in statistics.