We know that the empirical risk is : $L_s = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i)$

where, $n$ = number of samples,$l(f(x),z)$ is a loss function, $S = (z_1,...,z_n)$ are the provided samples to test\train on, $f(x_i)$ is the output of a learning algorithm and $z_i \in Z $ is $(x_i, y_i)$

Some current literature like this (page 3 section 3.2), indicate that the true risk to be the expected value with respect to joint distribution of $n$ samples. $L_\mu = E_{Z^n}[L_s]$ $_{(1)}$

I do not understand how can the true risk be just the expected value of the empirical risk

This is how far I got :

We know that true risk is defined as : $L_u = \int l(f(x),z) dP(x,y) $

Now to write it in a summation form I can assume $n$ to be arbitrarily large, $L_u = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),z_i).P(x_i,y_i)$ $_{(2)}$

Now have no idea about the next step to reconcile equation $(2)$ and $(1)$


You don't need to assume $n$ to be arbitrarily large. It's just linearity of expectation:$\DeclareMathOperator{\E}{\mathbb E}$ \begin{align} \E[\text{empirical risk}] &= \E\left[ \frac1n \sum_{i=1}^n l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \E\left[ l(f(x_i), z_i) \right] \\&= \frac1n \sum_{i=1}^n \int l(f(x_i), z_i) \,\mathrm{d}P(x_i, z_i) \\&= \frac1n \sum_{i=1}^n \text{true risk} \\&= \text{true risk} .\end{align}

| cite | improve this answer | |
  • $\begingroup$ Just to interpret what this means, will the empirical risk always be an unbiased estimator of the true risk ? This seems a little too specific $\endgroup$ – vortex_sparrow Nov 14 '18 at 18:09
  • $\begingroup$ The empirical risk of a particular model $f$ is an unbiased estimator (assuming iid sampled) of the true risk of that particular estimator, yes. But keep in mind that this will be broken if you've selected $f$ based on those same samples, e.g. by picking the empirical risk minimizer. $\endgroup$ – Dougal Nov 14 '18 at 19:46
  • $\begingroup$ Also if the samples were not iid it would also be a biased estimator probably? Moreover, I did not quite get why using the same samples used for training, not yield the same result? $\endgroup$ – vortex_sparrow Nov 14 '18 at 20:00
  • $\begingroup$ Using the same samples to train is overfitting. The easiest example to think about might be taking $f$ to be a (1-)nearest-neighbor model, so that if trained on $(x_i, z_i)$, then $f(x_i) = z_i$ exactly for presumably zero loss. Then using the same training set as testing set would give you zero empirical risk, while clearly the true risk (or the empirical risk on a different set) will be nonzero. $\endgroup$ – Dougal Nov 14 '18 at 20:02
  • $\begingroup$ If the samples aren't iid, then it in general won't be an unbiased estimator, but it depends on the particular kind of dependence. $\endgroup$ – Dougal Nov 14 '18 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.