2
$\begingroup$

I am currently trying to read the "Elements of Statistical Learning", by Efron, Hastie, and Tibshirani, and already at the beginning there is a bit above my level in mathematics. I have 3 questions regarding the move from (2.9) to (2.10):

  1. What is the meaning of integrating with respect to Pr(dx,dy) instead of with respect to dx,dy by themselves?

  2. Since this is an indefinite integral, shouldn't there be a constant C or something afterwards?

  3. This is more about the intuition behind this: why is the expected value of the loss function f is the same as the area beneath the function (that is, the integral of f)?

enter image description here

Improve this question
$\endgroup$
7
  • $\begingroup$ en.wikipedia.org/wiki/Multiple_integral $\endgroup$
    – Stephan Kolassa
    Commented Nov 19, 2017 at 11:07
  • 2
    $\begingroup$ If you're finding ESA too tricky (and it can be quite dense), can I recommend maybe having a look at "An Introduction to Statistical Learning" by Robert Tibshirani and Trevor Hastie instead? $\endgroup$
    – MachineEpsilon
    Commented Nov 19, 2017 at 14:03
  • $\begingroup$ @Xi'an is it ok now? :) $\endgroup$
    – Moran Reznik
    Commented Nov 19, 2017 at 16:07
  • 1
    $\begingroup$ If these formulas are not clear to you, it might be useful to work through the book alongside a solid reference in elementary probability, such as Grimmett and Stirzaker. $\endgroup$
    – Paul
    Commented Nov 19, 2017 at 16:19
  • 2
    $\begingroup$ Agreed with @Paul. Probability with Martingales by Williams or Probability and Measure by Billingsley are two excellent references if you want to understand the probability theory underlying machine learning. $\endgroup$
    – David Kozak
    Commented Nov 19, 2017 at 19:20

1 Answer 1

Reset to default
5
$\begingroup$
  1. what is the meaning of integrating with respect to Pr(dx,dy) instead of with respect to dx,dy by themselves?

You are missing the notion that this is an expectation, that is, the average value of $(Y-f(X))^2$ under the joint distribution of $(X,Y)$. (Using Pr is clearly not the most inspired choice!)

  1. since this is an indefinite integral, shouldn't there be a +C or something afterwards?

This is a regular integral and not an anti-derivative. The authors did not put the domain of integration on the integral sign, as is common when this domain does not suffer from possible confusion.

  1. this is more about the intuition behind this: why is the expected value of the loss function f is the same as the area beneath the function (that is, the integral of f)?

First, $f(X)$ is not the loss function but the transform of the random variable $X$. The loss function is $(Y-f(X))^2$. Second, the integral of the loss function under the probability measure is its average or averaged value. This is the definition for expectations.

About this entire excerpt, one way to look at it is to see it as a probabilistic version of the Pythagorean theorem: the average square distance between $Y$ and $f(X)$ is the sum $$\mathbb{E}[(Y-f(X))^2]=\mathbb{E}[(Y-\mathbb{E}[Y|X])^2]+\mathbb{E}[(\mathbb{E}[Y|X]-f(X))^2]$$ since the terms $(Y-\mathbb{E}[Y|X])$ and $(\mathbb{E}[Y|X]-f(X))$ are orthogonal in this probabilistic sense.

Improve this answer
$\endgroup$
2
  • $\begingroup$ thank you. I would like to explain why in my mind the integral of the loss function ins't the averaged value, so you can tell me where im getting it wrong. As far as I know, integrals are about taking infinitely many sums. in this case, summing up the results of every combination of P(X ,Y) in the loss function. While, expected value is more or less the same except that after you summed up the values of the loss function you than devide by the number of times you generated the loss, somthing that isn't done in integrals. Sholudent this diffrence give other results in those exprestions? $\endgroup$
    – Moran Reznik
    Commented Nov 20, 2017 at 8:34
  • 1
    $\begingroup$ @MoranReznik you should look into the difference between a Riemann integral and a Lebesgue integral. Expectations are the latter. $\endgroup$
    – Kuku
    Commented Dec 11, 2023 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.