I am building a predictor for $y = f(x)$ using training samples ${(x_i, y_i)}$ (assume) drawn i.i.d from some distribution $p(x,y)$, by optimising the empirical L2-loss:

$f(x) = argmin_f \; \sum_i ||f(x_i)-y_i||_2^2$.

(Assume $f$ is suitably parameterised, say linear regression or neural networks, etc.)

It's known that the minimiser $f = E[y|x]$.

Now, I'd like to get a confidence estimate of my prediction $f(x)$, using say the variance of the prediction. So, I thought of this:

  1. Generate a new dataset ${(x_i,y_i^2)}$.
  2. Find $g(x) = argmin_g \; \sum_i ||g(x_i) - y_i^2||_2^2$.
  3. We know that the minimiser $g = E[y^2|x]$.
  4. Compute the conditional variance estimate as ${\rm var}(y|x) = g(x)-f(x)^2$ and use this as an estimate of the uncertainty in the prediction $f(x)$.

Is the above a theoretically sound way of estimating the confidence in the prediction?


2 Answers 2


Yes, estimating $E[y^2|x]$ and $E[y|x]$ separately and subtracting $E[y^2|x] - (E[y|x])^2$ is one way. But it is a biased estimator. One reason is that we did not get an unbiased estimate of $(E[y|x])^2$ but, hopefully, for $E[y|x]$. Also, it can be negative while conditional variance should be non-negative. This is described in Eq. 1.2 in http://stats.lse.ac.uk/q.yao/qyao.links/paper/bka98.pdf. See the discussion in Section 3.1. The paper proposes a kernel regression based estimator.


why do you square the observations $y_i$ ?

The Laplace approximation gives you an idea of the precision: the inverse curvature (where by 'curvature' I mean the set of eigenvalues of the Hessian) of the paraboloid approximating the log-posterior is an estimate of the uncertainty associated with the posterior.

However this approximation fails for multimodal distributions and other non-Gaussian phenomena (i.e. for which the central limit theorem does not apply).

  • $\begingroup$ I am not directly computing the posterior $p(y|x)$; rather, I am computing the first moment of the posterior (conditional mean), as it's easier to learn. $\endgroup$
    – Vimal
    Jan 24, 2015 at 19:20
  • $\begingroup$ I am deriving a new dataset $(x_i, z_i)$, where $z=y_i^2$, to learn $E[z|x]$ (i.e., $E[y^2|x]$). $\endgroup$
    – Vimal
    Jan 24, 2015 at 19:21
  • $\begingroup$ The conditional mean is the location parameter vector of the Laplace approximation above (which is just the truncated Taylor series of the log-posterior). $\endgroup$
    – ocramz
    Jan 24, 2015 at 19:24
  • $\begingroup$ I mean, at the end of the day you need to perform the marginalization over the parameters somehow: either by approximating with an "easier" parametric distribution (variational approach) or by sampling (slow) or quadrature (specialized) $\endgroup$
    – ocramz
    Jan 24, 2015 at 19:29
  • $\begingroup$ and mapping the observations onto a nonlinear manifold introduces another set of considerations (e.g. inner products and therefore distances between $z$'s need to take into account the metric transformation) $\endgroup$
    – ocramz
    Jan 24, 2015 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.