What is meant by the variance of *functions* in *Introduction to Statistical Learning*?

Question

On pg. 34 of Introduction to Statistical Learning: $\newcommand{\Var}{{\rm Var}}$

Though the mathematical proof is beyond the scope of this book, it is possible to show that the expected test MSE, for a given value $x_0$, can always be decomposed into the sum of three fundamental quantities: the variance of $\hat{f}(x_0)$, the squared bias of $\hat{f}(x_0)$ and the variance of the error terms $\varepsilon$. That is,

$$ E\left(y_0 - \hat{f}(x_0)\right)^2 = \Var\big(\hat{f}(x_0)\big) + \Big[{\rm Bias}\big(\hat{f}(x_0)\big)\Big]^2 + \Var(\varepsilon) $$

[...]Variance refers to the amount by which $\hat{f}$ would change if we estimated it using a different training data set.

Question: Since $\Var\big(\hat{f}(x_0)\big)$ seems to denote the variance of functions, what does this mean formally?

That is, I am familiar with the concept of the variance of a random variable $X$, but what about the variance of a set of functions? Can this be thought of as just the variance of another random variable whose values take the form of functions?

Answer 1

Your correspondence with @whuber is correct.

A learning algorithm $\mathcal{A}$ can be viewed as a higher level function, mapping training sets to functions.

$$ \mathcal{A} : \mathcal{T} \rightarrow \{f \mid f: X \rightarrow \mathbb{R} \} $$

where $\mathcal{T}$ is the space of possible training sets. This can be a bit hairy conceptually, but basically each individual training set results, after using the model training algorithm, in a speicific function $f$ which can be used to make predictions given a data point $x$.

If we view the space of training sets as a probability space, so that there is some distribution of possible training data sets, then the model training algorithm becomes a function valued random variable, and we can think of statistical concepts. In particular, if we fix a specific data point $x_0$, then we get the numeric valued random variable

$$ \mathcal{A}_{x_0}(T) = \mathcal{A}(T)(x_0) $$

I.e., first train the algorithm on $T$, and then evaluate the resulting model at $x_0$. This is just a plain old, but rather ingeniously constructed, random variable on a probability space, so we can talk about its variance. This is the variance in your formula from ISL.

Answer 2

A visual interpretation using repeated kfolds

To give a visual / intuitive interpretation to @Matthew Drury's answer consider the following toy example.

• Data is generated from noisy sine curve: "True $f(x) \ +$ noise"
• The data is split between training and testing samples (75% - 25%)
• A linear (polynomial) model is fitted to the training data: $\hat f(x)$
• The process is repeated many times using the same data (i.e. splitting training - testing randomly using Sklearm repeated kfold)
• This generates many different models, from which we compute the mean and the variance at each point $x=x_i$ as well as over all points.

See below for the resulting graphs for a polynomial model of degree 2 and degree 6. At first sight, it seems that the higher polynomial (in red) has greater variance.

Arguing that the red graph has greater variance - experimentally

Let $\hat f_g$ and $\hat f_r$ correspond to the green and red graphs respectively and $\hat f^{(i)}$ be one instance of the graphs, in light green and light red. Let $n$ be the number of points along the $x$ axis and $m$ be the number of graphs (i.e. the number of simulations). Here we have $n = 400$ and $m = 200$

I see three main scenarios

1. The variance of the predicted values at one specific point $x = x_0$ is greater i.e. $ Var \ \left[ \{\hat f^{(1)}_r(x_0), ..., \hat f^{(m)}_r(x_0)\} \right] > Var \ \left[ \{\hat f^{(1)}_g(x_0),...,\hat f^{(i)}_g(x_0)\} \right]$
2. The variance in $(1)$ is greater for all points $\{ x_1,...,x_{400} \}$ in the range $(0,1)$
3. The variance is greater on average (i.e. may be smaller for some points)

In the case of this toy example, all three scenarios hold true over the range $(0,1)$ which justifies the argument that the higher order polynomial fit (in red) has higher variance than the lower order polynomial (in green).

An open ended conclusion

What should be argued when the above three scenarios do not all hold. For example, what if the variance of the red predictions is greater on average, but not for all points.

Details of the labels

Consider point $x_0 = 0.5$

• The error bar is the range between min and max of $\hat f(x_0)$
• The variance is computed at $x_0$
• True $f(x)$ is the dotted blue line