Learning a confidence interval is there any idea to solve this problem?

Question

I was with my friend trying to figure a generic confidence interval for any real model $f$ (already trained) so we'd like to get:

$CI(x)=(L(x,\mathbf{\Theta)}),U(x,\mathbf{Z}))$

Where $L$ and $U$ are the lower and upper confidence limit to $f$ at the observation $x_i \in \Re^p$. That's we're looking for a confidence interval for the response of any $x$ in feature space.

Then we proposed ourselves to minimize $L$ at $\mathbf{\Theta},\mathbf{Z}$, where $L$ is:

$L=\displaystyle\sum_{i=1}^{N}(f(x_i)-L(x_i,\mathbf{\Theta)})^2+\sum_{i=1}^{N}(f(x_i)-U(x_i,\mathbf{Z}))^2+(1-\alpha-P^{*}(\{L(x_i,\mathbf{\Theta})\leq y_i \leq U(x_i,\mathbf{Z})\}_{i=1}^{N}))^2$

Where $P^{*}(.)$ is the sample-proportion of $(.)$ smoothed by a differentiable kernel density.

Of course we assumed $L,U$ to be non parametric models such as neural networks, decision trees, gradient boosting and so on.

We all know that there are so many packages to fit those non parametric models but note that this problem is not a simple regression fitting so here's my question:

Could I fit $L,U$ knowing how to fit $U$ and $L$? That's knowing how to minimize $\displaystyle\sum_{i=1}^{N}(f(x_i)-L(x_i,\mathbf{\Theta)})^2$ and $\sum_{i=1}^{N}(f(x_i)-U(x_i,\mathbf{Z}))^2$? if so how can I do this? if not what kind of optimization theory would be interesting to try solve this?

Answer 1

In your objective function $L$ I see a $y_i$. Are you instead interested in a prediction interval for a future observation?

A confidence interval is a set of hypotheses for a parameter such that the observed result is within a $100(1-\alpha)\%$ margin of error, i.e. those hypotheses that when tested are not significant at level $\alpha$.

Your function $L$ reminds me of a Lagrange multiplier. I think the intuition is in the right direction. If you are interested in inference on, say, the population mean my mind immediately goes to inverting an approximate Wald test. You could use your non-parametric methods or the empirical sandwich estimator to estimate the population variance and ultimately the standard error. This would be applicable if you are interested in some other population quantity other than the mean, e.g. a population percentile, and can propose a non-parametric estimator. You might incorporate a link function such as $g\{\cdot\}=\text{log}\{\cdot\}$ to improve the normal approximation of the sampling distribution of the estimator. Even if the population distribution is clearly non-normal, the sampling distribution of a parameter estimator is often well approximated by a normal distribution.