# Estimating conditional variance y|x

I am building a predictor for $y = f(x)$ using training samples ${(x_i, y_i)}$ (assume) drawn i.i.d from some distribution $p(x,y)$, by optimising the empirical L2-loss:

$f(x) = argmin_f \; \sum_i ||f(x_i)-y_i||_2^2$.

(Assume $f$ is suitably parameterised, say linear regression or neural networks, etc.)

It's known that the minimiser $f = E[y|x]$.

Now, I'd like to get a confidence estimate of my prediction $f(x)$, using say the variance of the prediction. So, I thought of this:

1. Generate a new dataset ${(x_i,y_i^2)}$.
2. Find $g(x) = argmin_g \; \sum_i ||g(x_i) - y_i^2||_2^2$.
3. We know that the minimiser $g = E[y^2|x]$.
4. Compute the conditional variance estimate as ${\rm var}(y|x) = g(x)-f(x)^2$ and use this as an estimate of the uncertainty in the prediction $f(x)$.

Is the above a theoretically sound way of estimating the confidence in the prediction?

Yes, estimating $$E[y^2|x]$$ and $$E[y|x]$$ separately and subtracting $$E[y^2|x] - (E[y|x])^2$$ is one way. But it is a biased estimator. One reason is that we did not get an unbiased estimate of $$(E[y|x])^2$$ but, hopefully, for $$E[y|x]$$. Also, it can be negative while conditional variance should be non-negative. This is described in Eq. 1.2 in http://stats.lse.ac.uk/q.yao/qyao.links/paper/bka98.pdf. See the discussion in Section 3.1. The paper proposes a kernel regression based estimator.

why do you square the observations $y_i$ ?

The Laplace approximation gives you an idea of the precision: the inverse curvature (where by 'curvature' I mean the set of eigenvalues of the Hessian) of the paraboloid approximating the log-posterior is an estimate of the uncertainty associated with the posterior.

However this approximation fails for multimodal distributions and other non-Gaussian phenomena (i.e. for which the central limit theorem does not apply).

• I am not directly computing the posterior $p(y|x)$; rather, I am computing the first moment of the posterior (conditional mean), as it's easier to learn. Jan 24, 2015 at 19:20
• I am deriving a new dataset $(x_i, z_i)$, where $z=y_i^2$, to learn $E[z|x]$ (i.e., $E[y^2|x]$). Jan 24, 2015 at 19:21
• The conditional mean is the location parameter vector of the Laplace approximation above (which is just the truncated Taylor series of the log-posterior). Jan 24, 2015 at 19:24
• I mean, at the end of the day you need to perform the marginalization over the parameters somehow: either by approximating with an "easier" parametric distribution (variational approach) or by sampling (slow) or quadrature (specialized) Jan 24, 2015 at 19:29
• and mapping the observations onto a nonlinear manifold introduces another set of considerations (e.g. inner products and therefore distances between $z$'s need to take into account the metric transformation) Jan 24, 2015 at 19:31