Skip to main content
Cross Validated

Return to Question

Became Hot Network Question
deleted 2 characters in body
Source Link
kjetil b halvorsen ♦
kjetil b halvorsen ♦
  • 82.8k
  • 32
  • 201
  • 663

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

added 1 character in body
Source Link
User1865345
User1865345
  • 10.3k
  • 12
  • 23
  • 40

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) log\left(\frac{P(X)}{Q(X)}\right)dx$$$$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $log\left(\frac{P(X)}{Q(X)}\right)$$\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ \log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) \log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $\log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?

Source Link
Soltius
Soltius
  • 1.4k
  • 10
  • 17

Notation confusion regarding Expectation in Kullback-Leibler divergence definition

KL-divergence is often defined as $$D_{KL}(P||Q) = E_{x\sim P} \left[ log\left(\frac{P(X)}{Q(X)}\right) \right] = \int_{-\infty}^{\infty} P(x) log\left(\frac{P(X)}{Q(X)}\right)dx$$

I don't quite understand the meaning of the notation of the second term and how you can get the third term from there. The fact that there is the subscript $x \sim P$ under the expectation seems to indicate that there are several random variables in the Expectation bracket and the subscript is here to clarify the variable with respect to which we're integrating. In such cases, the notation I'm used to is $E_X[h(X,Y)]$ (eg in this answer). But here, I'm confused by the fact that there seems to be a single random variable $X$ following two different distributions. So could this expression be rewritten with two separate random variables $X$ and $Y$ ?

Regarding the derivation of the third term, do you get it via the LOTUS, setting $g(X)$ as $log\left(\frac{P(X)}{Q(X)}\right)$ ? I've already seen functions of random variables, but I find it strange that in this case, the $g$ function would the probability distributions $P(X)$ and $Q(X)$ of the random variable(s)... All this makes me think that I'm misunderstanding a basic notation. Can someone walk me through the explanation of the meaning of the second term and the derivation of the third one ?