I was with my friend trying to figure a generic confidence interval for any real model $f$ (already trained) so we'd like to get:


Where $L$ and $U$ are the lower and upper confidence limit to $f$ at the observation $x_i \in \Re^p$. That's we're looking for a confidence interval for the response of any $x$ in feature space.

Then we proposed ourselves to minimize $L$ at $\mathbf{\Theta},\mathbf{Z}$, where $L$ is:

$L=\displaystyle\sum_{i=1}^{N}(f(x_i)-L(x_i,\mathbf{\Theta)})^2+\sum_{i=1}^{N}(f(x_i)-U(x_i,\mathbf{Z}))^2+(1-\alpha-P^{*}(\{L(x_i,\mathbf{\Theta})\leq y_i \leq U(x_i,\mathbf{Z})\}_{i=1}^{N}))^2$

Where $P^{*}(.)$ is the sample-proportion of $(.)$ smoothed by a differentiable kernel density.

Of course we assumed $L,U$ to be non parametric models such as neural networks, decision trees, gradient boosting and so on.

We all know that there are so many packages to fit those non parametric models but note that this problem is not a simple regression fitting so here's my question:

Could I fit $L,U$ knowing how to fit $U$ and $L$? That's knowing how to minimize $\displaystyle\sum_{i=1}^{N}(f(x_i)-L(x_i,\mathbf{\Theta)})^2$ and $\sum_{i=1}^{N}(f(x_i)-U(x_i,\mathbf{Z}))^2$? if so how can I do this? if not what kind of optimization theory would be interesting to try solve this?


1 Answer 1


In your objective function $L$ I see a $y_i$. Are you instead interested in a prediction interval for a future observation?

A confidence interval is a set of hypotheses for a parameter such that the observed result is within a $100(1-\alpha)\%$ margin of error, i.e. those hypotheses that when tested are not significant at level $\alpha$.

Your function $L$ reminds me of a Lagrange multiplier. I think the intuition is in the right direction. If you are interested in inference on, say, the population mean my mind immediately goes to inverting an approximate Wald test. You could use your non-parametric methods or the empirical sandwich estimator to estimate the population variance and ultimately the standard error. This would be applicable if you are interested in some other population quantity other than the mean, e.g. a population percentile, and can propose a non-parametric estimator. You might incorporate a link function such as $g\{\cdot\}=\text{log}\{\cdot\}$ to improve the normal approximation of the sampling distribution of the estimator. Even if the population distribution is clearly non-normal, the sampling distribution of a parameter estimator is often well approximated by a normal distribution.

  • $\begingroup$ Yeah a predictive confidence interval do you think I be better updating this? $\endgroup$ Dec 8, 2021 at 3:06
  • $\begingroup$ I think wald statistics would not be useful I think I would have to have so many observations to a single observation $x_i$ but the question is about how to solve the given problem (minimize $L$). Regardless, I'd think about booststrap but the computional spending would be a lot. I was figuring about just remove the two first terms in $L$ they dont seem be useful. $\endgroup$ Dec 8, 2021 at 5:20
  • $\begingroup$ Yes, I suggest replacing the word "confidence" with "prediction" in your question. Here is a thread on prediction intervals. If the prediction target is drawn from an approximately normal distribution you can create a Wald-type prediction interval as well. If you are predicting a subject-level observation from a non-normal distribution the link above discusses a paper with simple and effective methods. There are many methods for prediction and the intuition to your approach may be effective. $\endgroup$ Dec 8, 2021 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.