# How is True risk equal to the expected value of the empirical risk?

We know that the empirical risk is : $$L_s = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i)$$

where, $$n$$ = number of samples,$$l(f(x),z)$$ is a loss function, $$S = (z_1,...,z_n)$$ are the provided samples to test\train on, $$f(x_i)$$ is the output of a learning algorithm and $$z_i \in Z$$ is $$(x_i, y_i)$$

Some current literature like this (page 3 section 3.2), indicate that the true risk to be the expected value with respect to joint distribution of $$n$$ samples. $$L_\mu = E_{Z^n}[L_s]$$ $$_{(1)}$$

I do not understand how can the true risk be just the expected value of the empirical risk

This is how far I got :

We know that true risk is defined as : $$L_u = \int l(f(x),z) dP(x,y)$$

Now to write it in a summation form I can assume $$n$$ to be arbitrarily large, $$L_u = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i).P(x_i,y_i)$$ $$_{(2)}$$

Now have no idea about the next step to reconcile equation $$(2)$$ and $$(1)$$

You don't need to assume $$n$$ to be arbitrarily large. It's just linearity of expectation:$$\DeclareMathOperator{\E}{\mathbb E}$$ \begin{align} \E[\text{empirical risk}] &= \E\left[ \frac1n \sum_{i=1}^n l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \E\left[ l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \int l(f(x_i), z_i) \,\mathrm{d}P(x_i, z_i) \\&= \frac1n \sum_{i=1}^n \text{true risk} \\&= \text{true risk} .\end{align}
• The empirical risk of a particular model $f$ is an unbiased estimator (assuming iid sampled) of the true risk of that particular estimator, yes. But keep in mind that this will be broken if you've selected $f$ based on those same samples, e.g. by picking the empirical risk minimizer. – Dougal Nov 14 '18 at 19:46
• Using the same samples to train is overfitting. The easiest example to think about might be taking $f$ to be a (1-)nearest-neighbor model, so that if trained on $(x_i, z_i)$, then $f(x_i) = z_i$ exactly for presumably zero loss. Then using the same training set as testing set would give you zero empirical risk, while clearly the true risk (or the empirical risk on a different set) will be nonzero. – Dougal Nov 14 '18 at 20:02