1
$\begingroup$

My question is related to section 3.2: Bias-Variance decomposition. My doubt is specific to formula for variance (3.47, below), on pg 150 - 151.

Background: General formula for variance is:

$\begin{align*} variance = E_{\mathcal D} [\{y(\mathbf x;\mathcal D) - E_{\mathcal D}[y(\mathbf x; \mathcal D)]\}^2] \end{align*} \tag{3.40}$

where:

  1. $y(\mathbf x;\mathcal D)$: output based on prediction model for dataset $\mathcal D \in L \, (one\,of\,100\, datasets)$ $\implies $ there'll be 100 outputs, 1 for each of 100 models.
  2. $E_{\mathcal D}[y(\mathbf x; \mathcal D)]$: Average of prediction for respective dataset $\mathcal D$ $\implies $ there'll be 100 mean values, 1 per model.
  3. $\begin{align*} \{y(\mathbf x;\mathcal D) - E_{\mathcal D} [y(\mathbf x; \mathcal D)]\}^2 \end{align*}$ : squared-difference w.r.t mean for $\mathcal D$ $\implies $ similarly 100 squared errors.
  4. $E_{\mathcal D} [.]:$ averaged squared error, across all datasets $\mathcal D$, as calculated above

pls. note: $E$ stands for Expectation (average, in probability).

There're $\mathbf {L=100}$ datasets, each containing $\mathbf {N=25} $ data points.

Step 1-Modelling:- On the basis of analysis of n data points of a dataset l, we model a prediction formula $y^{(l)}(.)$ that outputs value - for an input x, based on model specific to that dataset.

Step 2 - Prediction (Calculate mean of predictions across models) : for any point x, take the mean of predictions $\bar y$, across models: $\begin{align*} \bar y = \frac {1}{L} \sum_{l=1}^L y^{(l)}(x) \end{align*}$

And then, the author goes on to calculate variance as:

$\begin{align*} variance = \frac {1}{N} \sum_{n=1}^N \frac {1}{L} \sum_{l=1}^L \{y^{(l)}(x_n) - \bar y(x_n)\}^2 \end{align*} \tag{ 3.47}$

Question:

Even though addition is commutative, my doubt is at conceptual level related to ordering of summation:

Shouldn't we first choose a dataset l from L (outer loop/ sum), and then find the variance - average of mean-squared error - over all data points, n $\in$ N - of that dataset (inner loop/ sum); i.e. shouldn't the equation be:

$\begin{align*} \frac {1}{L} \sum_{l=1}^L \frac {1}{N} \sum_{n=1}^N \{y^{(l)}(x_n) - \bar y(x_n)\}^2 \end{align*}$

I would appreciate if you can guide me on this.

The above-mentioned book is freely available on Microsoft research portal.

P.S.: Shifted his question from Mathematics to Cross Validation - as suggested by fellow learner.

$\endgroup$

1 Answer 1

1
$\begingroup$

The two formulas give exactly the same answer. However, the one in the book might sometimes be easier to implement, as $\bar y(x_n)$ is constant during the inner loop, so you only need to store or compute $\bar y(x_n)$ for the current $n$. With your version, the inner loop needs all the values of $\bar y(x_n)$

$\endgroup$
2
  • $\begingroup$ I guess I got your explanation. Would appreciate if you can confirm/ flag the understanding: Variance calculation: i)Take first of $\mathbf N = 25$ points (outer loop) -> ii)calculate avg of predictions $\bar y$ across datasets' ($\mathbf L$ = 100) models -> iii): calculate the MSE of (model's prediction - $\bar y$) i.e. 100 squared terms (for each of these 100 sum, $\bar y$ remains constant.) Then take next point, move to step 2 - calculated avg prediction - and so on. Is my understanding right? Else, point out the gap(s). Regards $\endgroup$ Mar 15, 2022 at 7:37
  • $\begingroup$ ..and so, implementation steps would be $\sum_{n=1}^N (\cdot) > \bar y(\cdot) > \sum_{l=1}^L (\cdot)$ i.e. take a point > calculate 100 models' predication mean > calculate MSE. Is my understanding correct. Would appreciate your input. Regards. $\endgroup$ Mar 15, 2022 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.