From understanding machine learning:

How are the two expected values equal in the proof of the theorem in the red box below?

Notation: $D$ is a distribution on the set $Z$, $E_{z \text{ ~ }D}$ means the expected value with respect to $z$ having distribution $D$, $D^m$ is the distribution of choosing $m$ points $(z_1, \dots, z_m)$, $l(A(S), z)$ is some loss function with respect to some learning algorithm $A$ trained on $S$ and evaluated at $z$.

By their definitions:

$$\ \Bbb E_{S \text{ ~ } D^m, z' \text{ ~ } D}[l(A(S), z')] = \Bbb E_S[\Bbb E_{z'}[l(A(S), z')]] = \Bbb E_S [\sum_{z' \in Z}l(A(S), z')D(z')]$$

$$ \Bbb E_{S \text{ ~ } D^m, z' \text{ ~ } D}[l(A(S^{(i)}), z_i)] = \Bbb E_S[\Bbb E_{z'}[l(A(S^{(i)}), z_i)]] = \Bbb E_S [\sum_{z' \in Z}l(A(S^{(i)}), z_i)D(z')]$$

But for these to be equal we must have:

$$ \sum_{z' \in Z}l(A(S), z')D(z')= \sum_{z' \in Z}l(A(S^{(i)}), z_i)D(z')$$

I can't see how these are equal.

Anyone have any ideas?

enter image description here


closed as unclear what you're asking by Xi'an, Michael Chernick, mdewey, kjetil b halvorsen, usεr11852 Feb 2 at 21:19

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    $\begingroup$ Much of the notation is opaque, so it's unclear exactly what is being asserted, but it looks like the authors believe it is an immediate consequence of the iid assumption. Indeed, if these expectations depend only on the distributions of the $z_i$ and $z^\prime,$ then the result follows without any further ado. $\endgroup$ – whuber Feb 1 at 22:48
  • $\begingroup$ @whuber I have edited the question to include the notation for clarity, it may be more clear now. $\endgroup$ – Oliver G Feb 1 at 22:54
  • 1
    $\begingroup$ Please give a full reference to the book(?) $\endgroup$ – Juho Kokkala Feb 2 at 8:22