How to compute confidence and prediction intervals for (nonlinear) regression using non squared "loss function"

Question

Does somebody know (or can point me to a reference) how can I compute a confidence and prediction intervals for (posibly nonlinear) regression using non squared "loss functions"?

Let me add some details to clarify my question. Let's say I am interested in finding the parameters $\beta$ to fit $y = f(x,\beta)$. I'm mostly interested in nonlinear regression but also the solution for the linear case would interest me.

By confidence interval I mean an interval $[\beta_1, \beta_2]$ for $\beta$.

By prediction interval I mean an interval $[y_1, y_2]$ for $y(x^*)$. $y$ is computed for a given $x^*$.

By loss function I mean a measure of lack of fit. Normally one tries to minimize this. Thes most commonly used loss function $L$ is the sum of squares ($L=\Sigma (y_i - f(x_i,\beta))^2 $)

I know the typical formulas in regression books to computed those intervals in the case that the loss functionis a sum of squares. But what happens if I choose a different loss function $L$ like sum of absolute error, or sum of absolute percentage error and so on? Do the same formulas apply or not? Does it matter if the loss function is symmetric?

I read a little bit about bootstrapping, but I am thinking more in theoretical results instead of numerical.

Many thanks in advance!

Answer 1

In order to calculate a parametric confidence intervall you need distributional assumptions. Simply using the L2 loss does not allow you to calculate confidence/prediction intervals the "usual way". You need normally distributed errors in order for these intervals to be legitimate. In that case the L2-optimizer is also the maximum likelihood estimate. So in that sense there is a connection between distributional assumptions and loss functions (for every distribution you can use the negative log likelihood as a loss function, but not for every loss function there is a correponding distribution)

If you would use the absolute loss, this would correspond the assumption of laplace distributed errors, i.e.

$y= \theta^Tx+\epsilon$ where $\epsilon$ has pdf $\frac{1}{2b}e^{-\frac{|x-\mu|}{b}}$. So you could run your regression and then estimate the parameters of the laplace distribution from your residuals and then get the quantiles in order to get your prediction intervals (for your predictions).

Additional notes to clarify questions in the comment:

The loss function you use determines - together with the distribution of the underlying data-generating process - the distribution of your estimator, which in turn determines your confidence and prediction interval. Therefore the loss function is also relevant to the construction of your confidence/ prediction interval. This might become clearer when looking at the mathematical definition of the confidence interval:

Consider the simplest case of the linear model where $y_i= \theta_0+ \epsilon_i$ and let $\hat{\theta}$ be our estimator and assume for the sake of argument, that we know the distribution of $\epsilon\sim N(0,\sigma^2)$. Then we want $c$, such that $P_{\theta_{true}}(\theta_{true}\in [\hat{\theta}-c, \hat{\theta}+c])= P_{\theta_{true}}(\hat{\theta}\in [\theta_{true}-c, \theta_{true}+c])=95\%$ so you see from this formulation, that the definition of the confidence intervall depends on the distribution of $\hat{\theta}$ which is defined by the distribution of $\epsilon_i$ and the loss function that is minimized.

Consider the following example: Let`s say you use an extremely dumb loss function (think of anything really stupid) that leads to extremely bad predictions. Then you cannot calculate confidence intervals the usual way, even though the errors in the data-generating process are normally distributed. You still need the appropriate estimator, i.e. loss function or if you insist on your estimator than you would have to adjust the construction of your confidence interval.