Skip to main content
added 207 characters in body
Source Link
AruniRC
  • 479
  • 1
  • 4
  • 14

I have answered my own question. It turned out to be a rather obvious application of Bayes Rule only after making a somewhat arbitrary assumption. My question was not very clear, mostly due to my own tenuous understanding at that time.

However, this result is used quite a lot in machine learning literature involving integrating out missing variables. I am including the proof in case others find it helpful when seeing the result.

$$ P(x, y|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x, y) - A({\boldsymbol \theta}) \right) $$

By Bayes Rule,

$$ P(y|x, \theta) = \frac{ P(x|y, \theta)}{ \int_{y^{'}} P(x|{y^{'}}, \theta) P(y^{'}|\theta)d{y^{'}}} = \frac{ P(x, y| \theta)}{ \int_{y^{'}} P(x,{y^{'}}| \theta) d{y^{'}}} = \frac{h(x) \exp (\eta (\theta) . T(x,y) - A(\theta))}{ \int_{y^{'}} h(x) \exp (\eta (\theta) . T(x,y^{'}) - A(\theta))dy{'}} $$

Assumed the $h(x)$ base reference measure to be a function only of $x$ so that we can cancel it from numerator and denominator in the last step above, getting

$$ \frac{\exp ( \eta(\theta).T(x,y))}{\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}} = \exp ( \eta(\theta).T(x,y) - \log(\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}) ) = \exp ( \eta(\theta).T(x,y) - A(\theta|x) ) $$

The assumptions made above seems to make sense. I have answered my own question - this answer was satisfactory to my goal - deriving the expression for conditional probability in exponential families. Hopefully this might help others.

$$ P(x, y|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x, y) - A({\boldsymbol \theta}) \right) $$

By Bayes Rule,

$$ P(y|x, \theta) = \frac{ P(x|y, \theta)}{ \int_{y^{'}} P(x|{y^{'}}, \theta) P(y^{'}|\theta)d{y^{'}}} = \frac{ P(x, y| \theta)}{ \int_{y^{'}} P(x,{y^{'}}| \theta) d{y^{'}}} = \frac{h(x) \exp (\eta (\theta) . T(x,y) - A(\theta))}{ \int_{y^{'}} h(x) \exp (\eta (\theta) . T(x,y^{'}) - A(\theta))dy{'}} $$

Assumed the $h(x)$ base reference measure to be a function only of $x$ so that we can cancel it from numerator and denominator in the last step above, getting

$$ \frac{\exp ( \eta(\theta).T(x,y))}{\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}} = \exp ( \eta(\theta).T(x,y) - \log(\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}) ) = \exp ( \eta(\theta).T(x,y) - A(\theta|x) ) $$

The assumptions made above seems to make sense. I have answered my own question - this answer was satisfactory to my goal - deriving the expression for conditional probability in exponential families. Hopefully this might help others.

I have answered my own question. It turned out to be a rather obvious application of Bayes Rule only after making a somewhat arbitrary assumption. My question was not very clear, mostly due to my own tenuous understanding at that time.

However, this result is used quite a lot in machine learning literature involving integrating out missing variables. I am including the proof in case others find it helpful when seeing the result.

$$ P(x, y|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x, y) - A({\boldsymbol \theta}) \right) $$

By Bayes Rule,

$$ P(y|x, \theta) = \frac{ P(x|y, \theta)}{ \int_{y^{'}} P(x|{y^{'}}, \theta) P(y^{'}|\theta)d{y^{'}}} = \frac{ P(x, y| \theta)}{ \int_{y^{'}} P(x,{y^{'}}| \theta) d{y^{'}}} = \frac{h(x) \exp (\eta (\theta) . T(x,y) - A(\theta))}{ \int_{y^{'}} h(x) \exp (\eta (\theta) . T(x,y^{'}) - A(\theta))dy{'}} $$

Assumed the $h(x)$ base reference measure to be a function only of $x$ so that we can cancel it from numerator and denominator in the last step above, getting

$$ \frac{\exp ( \eta(\theta).T(x,y))}{\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}} = \exp ( \eta(\theta).T(x,y) - \log(\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}) ) = \exp ( \eta(\theta).T(x,y) - A(\theta|x) ) $$

Source Link
AruniRC
  • 479
  • 1
  • 4
  • 14

$$ P(x, y|\boldsymbol \theta) = h(x) \exp\left(\eta({\boldsymbol \theta}) . T(x, y) - A({\boldsymbol \theta}) \right) $$

By Bayes Rule,

$$ P(y|x, \theta) = \frac{ P(x|y, \theta)}{ \int_{y^{'}} P(x|{y^{'}}, \theta) P(y^{'}|\theta)d{y^{'}}} = \frac{ P(x, y| \theta)}{ \int_{y^{'}} P(x,{y^{'}}| \theta) d{y^{'}}} = \frac{h(x) \exp (\eta (\theta) . T(x,y) - A(\theta))}{ \int_{y^{'}} h(x) \exp (\eta (\theta) . T(x,y^{'}) - A(\theta))dy{'}} $$

Assumed the $h(x)$ base reference measure to be a function only of $x$ so that we can cancel it from numerator and denominator in the last step above, getting

$$ \frac{\exp ( \eta(\theta).T(x,y))}{\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}} = \exp ( \eta(\theta).T(x,y) - \log(\int_{y^{'}} \exp ( \eta(\theta).T(x,y^{'}))dy^{'}) ) = \exp ( \eta(\theta).T(x,y) - A(\theta|x) ) $$

The assumptions made above seems to make sense. I have answered my own question - this answer was satisfactory to my goal - deriving the expression for conditional probability in exponential families. Hopefully this might help others.