On pg. 34 of Introduction to Statistical Learning: $\newcommand{\Var}{{\rm Var}}$

Though the mathematical proof is beyond the scope of this book, it is possible to show that the expected test MSE, for a given value $x_0$, can always be decomposed into the sum of three fundamental quantities: the variance of $\hat{f}(x_0)$, the squared bias of $\hat{f}(x_0)$ and the variance of the error terms $\varepsilon$. That is,

$$ E\left(y_0 - \hat{f}(x_0)\right)^2 = \Var\big(\hat{f}(x_0)\big) + \Big[{\rm Bias}\big(\hat{f}(x_0)\big)\Big]^2 + \Var(\varepsilon) $$

[...]Variance refers to the amount by which $\hat{f}$ would change if we estimated it using a different training data set.

Question: Since $\Var\big(\hat{f}(x_0)\big)$ seems to denote the variance of functions, what does this mean formally?

That is, I am familiar with the concept of the variance of a random variable $X$, but what about the variance of a set of functions? Can this be thought of as just the variance of another random variable whose values take the form of functions?

  • 5
    $\begingroup$ Given that every time $\hat f$ appears in a formula it has been applied to a "given value" $x_0$, the variance applies to the number $\hat{f}(x_0)$, not to $\hat{f}$ itself. Since that number presumably has been developed from data that are modeled with random variables, it is also a (real-valued) random variable. The usual concept of variance applies. $\endgroup$ – whuber Apr 21 '16 at 19:19
  • 2
    $\begingroup$ I see. So $\hat{f}$ is changing (varying across different training data sets), but we still look at the variance of the $\hat{f}(x_0)$ themselves. $\endgroup$ – George Apr 21 '16 at 19:21
  • $\begingroup$ Who is the author of this textbook? I have been wanting to learn the subject myself and would greatly appreciate your reference recommendation. $\endgroup$ – Chill2Macht Apr 21 '16 at 21:58
  • 3
    $\begingroup$ @WilliamKrinsman This is the book: www-bcf.usc.edu/~gareth/ISL $\endgroup$ – Matthew Drury Apr 21 '16 at 22:13

Your correspondence with @whuber is correct.

A learning algorithm $\mathcal{A}$ can be viewed as a higher level function, mapping training sets to functions.

$$ \mathcal{A} : \mathcal{T} \rightarrow \{f \mid f: X \rightarrow \mathbb{R} \} $$

where $\mathcal{T}$ is the space of possible training sets. This can be a bit hairy conceptually, but basically each individual training set results, after using the model training algorithm, in a speicific function $f$ which can be used to make predictions given a data point $x$.

If we view the space of training sets as a probability space, so that there is some distribution of possible training data sets, then the model training algorithm becomes a function valued random variable, and we can think of statistical concepts. In particular, if we fix a specific data point $x_0$, then we get the numeric valued random variable

$$ \mathcal{A}_{x_0}(T) = \mathcal{A}(T)(x_0) $$

I.e., first train the algorithm on $T$, and then evaluate the resulting model at $x_0$. This is just a plain old, but rather ingeniously constructed, random variable on a probability space, so we can talk about its variance. This is the variance in your formula from ISL.


A visual interpretation using repeated kfolds

To give a visual / intuitive interpretation to @Matthew Drury's answer consider the following toy example.

  • Data is generated from noisy sine curve: "True $f(x) \ +$ noise"
  • The data is split between training and testing samples (75% - 25%)
  • A linear (polynomial) model is fitted to the training data: $\hat f(x)$
  • The process is repeated many times using the same data (i.e. splitting training - testing randomly using Sklearm repeated kfold)
  • This generates many different models, from which we compute the mean and the variance at each point $x=x_i$ as well as over all points.

See below for the resulting graphs for a polynomial model of degree 2 and degree 6. At first sight, it seems that the higher polynomial (in red) has greater variance.

enter image description here

Arguing that the red graph has greater variance - experimentally

Let $\hat f_g$ and $\hat f_r$ correspond to the green and red graphs respectively and $\hat f^{(i)}$ be one instance of the graphs, in light green and light red. Let $n$ be the number of points along the $x$ axis and $m$ be the number of graphs (i.e. the number of simulations). Here we have $n = 400$ and $m = 200$

I see three main scenarios

  1. The variance of the predicted values at one specific point $x = x_0$ is greater i.e. $ Var \ \left[ \{\hat f^{(1)}_r(x_0), ..., \hat f^{(m)}_r(x_0)\} \right] > Var \ \left[ \{\hat f^{(1)}_g(x_0),...,\hat f^{(i)}_g(x_0)\} \right]$
  2. The variance in $(1)$ is greater for all points $\{ x_1,...,x_{400} \}$ in the range $(0,1)$
  3. The variance is greater on average (i.e. may be smaller for some points)

In the case of this toy example, all three scenarios hold true over the range $(0,1)$ which justifies the argument that the higher order polynomial fit (in red) has higher variance than the lower order polynomial (in green).

An open ended conclusion

What should be argued when the above three scenarios do not all hold. For example, what if the variance of the red predictions is greater on average, but not for all points.

Details of the labels

Consider point $x_0 = 0.5$

  • The error bar is the range between min and max of $\hat f(x_0)$
  • The variance is computed at $x_0$
  • True $f(x)$ is the dotted blue line
  • $\begingroup$ I like this idea of illustrating a concept using pictures. I wonder about two aspects of your post, though, and hope you might be able to address them. First, could you more explicitly explain how these plots show the "variance of a function"? Second, it's not at all clear that the red plot exhibits "greater variance" or even that the two plots are amenable to such a simplistic comparison. Consider the vertical spread of red values above $x=0.95,$ for instance, and compare that to the spread of the green values at the same point: the red ones look a little less spread than the green ones. $\endgroup$ – whuber Jul 4 '18 at 11:39
  • $\begingroup$ My point is not whether it's possible to read your plots with high precision: it's that the meaning of comparing two such plots as if one could be considered of "higher" or "lower" variance than the other is questionable, given the possibility that for some ranges of $x$ the variances of the predictions will be higher in one plot and for other ranges of $x$ the variances will be lower. $\endgroup$ – whuber Jul 4 '18 at 14:10
  • $\begingroup$ Yes I agree - I have edited the post to reflect your comments $\endgroup$ – Xavier Bourret Sicotte Jul 4 '18 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.