Currently I am reading ESL. I am struggling to unterstand the step 2.15 to 2.16 in Chapter 2.4.

enter image description here

The authors mention that the stationary function can be found by differentiation. The commentary by Weatherwax and Epstein develops this step in more detail.

enter image description here

Why is $\ \dfrac{d}{ d \beta} \int (y-x^T\beta)^2 Pr(dx, dy) = \int \dfrac{d}{ d \beta} (y-x^T\beta)^2 Pr(dx, dy)$, that is differentiating the integral partially is equivalent to differentiating the function inside the integral? Any links or explanations are much appreciated.


For starters, I want to make sure that I understand correctly that when you say "deriving," you mean "differentiating".

What you are asking is called "differentiating under the equal sign" and is covered in basic calculus classes. If you still have your calculus textbook, you might want to do a quick review before going on in machine learning. I don't mean that condescendingly, just a suggestion.

The method is called the Leibnitz Integration Rule. You can read about it here with examples, and an interesting anecdote from Richard Feynmann. Working through the steps:

Notice that $EPE$ is a function of $\beta$ only because the dependence on $x$ and $y$ is being integrated out.

$$ \begin{align} \frac{d EPE}{d \beta} & = \frac{d}{d \beta} \int (y - x^T\beta)^2Pr(dx,dy) \\ & = \int \frac{\partial}{\partial \beta} (y - x^T\beta)^2 Pr(dx,dy) ]] \\ & \text{using the chain rule...} \\ & = \int -2(y - x^T\beta)Pr(dx,dy) \\ & = -2 \int (y - x^T\beta)Pr(dx,dy) \end{align} $$

The partial derivative is needed inside the integral because we have not yet integrated out $x$ and $y$, so the function we are differentiating is still a function of those variables.

  • $\begingroup$ My question is: why is $\ \dfrac{d}{ d \beta} \int (y-x^T\beta)^2 Pr(dx, dy) = \int \dfrac{d}{ d \beta} (y-x^T\beta)^2 Pr(dx, dy)$. Or expressed as functions $\ \dfrac{d}{ d \beta} F(f(\beta)) = F(\dfrac{d}{ d \beta} f(\beta)) $ Edited question for more clarity. $\endgroup$ – Bor Is Aug 13 at 19:19
  • $\begingroup$ It's called the Leibnitz Integration Rule. I've edited my answer with this and corrected a mistake. $\endgroup$ – abalter Aug 13 at 19:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.