# PRML - Bishop: Order of Addition - Variance calculation

My question is related to section 3.2: Bias-Variance decomposition. My doubt is specific to formula for variance (3.47, below), on pg 150 - 151.

Background: General formula for variance is:

\begin{align*} variance = E_{\mathcal D} [\{y(\mathbf x;\mathcal D) - E_{\mathcal D}[y(\mathbf x; \mathcal D)]\}^2] \end{align*} \tag{3.40}

where:

1. $$y(\mathbf x;\mathcal D)$$: output based on prediction model for dataset $$\mathcal D \in L \, (one\,of\,100\, datasets)$$ $$\implies$$ there'll be 100 outputs, 1 for each of 100 models.
2. $$E_{\mathcal D}[y(\mathbf x; \mathcal D)]$$: Average of prediction for respective dataset $$\mathcal D$$ $$\implies$$ there'll be 100 mean values, 1 per model.
3. \begin{align*} \{y(\mathbf x;\mathcal D) - E_{\mathcal D} [y(\mathbf x; \mathcal D)]\}^2 \end{align*} : squared-difference w.r.t mean for $$\mathcal D$$ $$\implies$$ similarly 100 squared errors.
4. $$E_{\mathcal D} [.]:$$ averaged squared error, across all datasets $$\mathcal D$$, as calculated above

pls. note: $$E$$ stands for Expectation (average, in probability).

There're $$\mathbf {L=100}$$ datasets, each containing $$\mathbf {N=25}$$ data points.

Step 1-Modelling:- On the basis of analysis of n data points of a dataset l, we model a prediction formula $$y^{(l)}(.)$$ that outputs value - for an input x, based on model specific to that dataset.

Step 2 - Prediction (Calculate mean of predictions across models) : for any point x, take the mean of predictions $$\bar y$$, across models: \begin{align*} \bar y = \frac {1}{L} \sum_{l=1}^L y^{(l)}(x) \end{align*}

And then, the author goes on to calculate variance as:

\begin{align*} variance = \frac {1}{N} \sum_{n=1}^N \frac {1}{L} \sum_{l=1}^L \{y^{(l)}(x_n) - \bar y(x_n)\}^2 \end{align*} \tag{ 3.47}

Question:

Even though addition is commutative, my doubt is at conceptual level related to ordering of summation:

Shouldn't we first choose a dataset l from L (outer loop/ sum), and then find the variance - average of mean-squared error - over all data points, n $$\in$$ N - of that dataset (inner loop/ sum); i.e. shouldn't the equation be:

\begin{align*} \frac {1}{L} \sum_{l=1}^L \frac {1}{N} \sum_{n=1}^N \{y^{(l)}(x_n) - \bar y(x_n)\}^2 \end{align*}

I would appreciate if you can guide me on this.

The above-mentioned book is freely available on Microsoft research portal.

P.S.: Shifted his question from Mathematics to Cross Validation - as suggested by fellow learner.

The two formulas give exactly the same answer. However, the one in the book might sometimes be easier to implement, as $$\bar y(x_n)$$ is constant during the inner loop, so you only need to store or compute $$\bar y(x_n)$$ for the current $$n$$. With your version, the inner loop needs all the values of $$\bar y(x_n)$$
• I guess I got your explanation. Would appreciate if you can confirm/ flag the understanding: Variance calculation: i)Take first of $\mathbf N = 25$ points (outer loop) -> ii)calculate avg of predictions $\bar y$ across datasets' ($\mathbf L$ = 100) models -> iii): calculate the MSE of (model's prediction - $\bar y$) i.e. 100 squared terms (for each of these 100 sum, $\bar y$ remains constant.) Then take next point, move to step 2 - calculated avg prediction - and so on. Is my understanding right? Else, point out the gap(s). Regards Mar 15, 2022 at 7:37
• ..and so, implementation steps would be $\sum_{n=1}^N (\cdot) > \bar y(\cdot) > \sum_{l=1}^L (\cdot)$ i.e. take a point > calculate 100 models' predication mean > calculate MSE. Is my understanding correct. Would appreciate your input. Regards. Mar 15, 2022 at 10:53