We know that the empirical risk is : $L_s = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i)$

where, $n$ = number of samples,$l(f(x),z)$ is a loss function, $S = (z_1,...,z_n)$ are the provided samples to test\train on, $f(x_i)$ is the output of a learning algorithm and $z_i \in Z $ is $(x_i, y_i)$

Some current literature like this (page 3 section 3.2), indicate that the true risk to be the expected value with respect to joint distribution of $n$ samples. $L_\mu = E_{Z^n}[L_s]$ $_{(1)}$

I do not understand how can the true risk be just the expected value of the empirical risk

This is how far I got :

We know that true risk is defined as : $L_u = \int l(f(x),z) dP(x,y) $

Now to write it in a summation form I can assume $n$ to be arbitrarily large, $L_u = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i).P(x_i,y_i)$ $_{(2)}$

Now have no idea about the next step to reconcile equation $(2)$ and $(1)$


1 Answer 1


You don't need to assume $n$ to be arbitrarily large. It's just linearity of expectation:$\DeclareMathOperator{\E}{\mathbb E}$ \begin{align} \E[\text{empirical risk}] &= \E\left[ \frac1n \sum_{i=1}^n l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \E\left[ l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \int l(f(x_i), z_i) \,\mathrm{d}P(x_i, z_i) \\&= \frac1n \sum_{i=1}^n \text{true risk} \\&= \text{true risk} .\end{align}

  • $\begingroup$ Just to interpret what this means, will the empirical risk always be an unbiased estimator of the true risk ? This seems a little too specific $\endgroup$ Nov 14, 2018 at 18:09
  • $\begingroup$ The empirical risk of a particular model $f$ is an unbiased estimator (assuming iid sampled) of the true risk of that particular estimator, yes. But keep in mind that this will be broken if you've selected $f$ based on those same samples, e.g. by picking the empirical risk minimizer. $\endgroup$
    – Danica
    Nov 14, 2018 at 19:46
  • $\begingroup$ Also if the samples were not iid it would also be a biased estimator probably? Moreover, I did not quite get why using the same samples used for training, not yield the same result? $\endgroup$ Nov 14, 2018 at 20:00
  • 1
    $\begingroup$ Using the same samples to train is overfitting. The easiest example to think about might be taking $f$ to be a (1-)nearest-neighbor model, so that if trained on $(x_i, z_i)$, then $f(x_i) = z_i$ exactly for presumably zero loss. Then using the same training set as testing set would give you zero empirical risk, while clearly the true risk (or the empirical risk on a different set) will be nonzero. $\endgroup$
    – Danica
    Nov 14, 2018 at 20:02
  • 1
    $\begingroup$ If the samples aren't iid, then it in general won't be an unbiased estimator, but it depends on the particular kind of dependence. $\endgroup$
    – Danica
    Nov 14, 2018 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.