3
$\begingroup$

Suppose $p(x) = \frac{\tilde{p}(x)}{Z}$ is a density function. I was wondering if the gradient of the log of the normalizing constant has a name and if we know any properties of it (e.g. maybe some properties similar to the score function) $$ \nabla_x \log Z = \nabla_x \log \int_{\mathcal{X}} \tilde{p}(x) dx $$

$\endgroup$
3
  • $\begingroup$ Can we say $$ \nabla_x \log Z = \frac{1}{Z}\nabla_x \int_{\mathcal{X}} \tilde{p}(x) dx = \frac{1}{Z}\int_{\mathcal{X}} \nabla_x \tilde{p}(x) dx $$ $\endgroup$ May 17 at 10:24
  • $\begingroup$ @Arya McCarthy. Your edited comment, referring to $\nabla_{x} \log Z$, that "it doesn't have a special name, but it's the expectation of the sufficient statistics for exponential family distributions," is erroneous. The result you state applies to the gradient of the log-partition function with respect to the parameter $\theta$, that is, $\nabla_{\theta} \log Z$. $\endgroup$
    – microhaus
    May 17 at 15:24
  • $\begingroup$ @Physics_Student. The gradient of the log-partition function is with respect to the parameter $\theta$, that is, $\nabla_{\theta} \log Z$. $\endgroup$
    – microhaus
    May 17 at 15:29
1
$\begingroup$

This answer assumes that you are interested in computing $\nabla_{\theta} \log Z$ rather than $\nabla_{x} \log Z$.

Without foreclosing the possibility that there may exist other results relevant to your question, a key result that arises is when $p(x)$ is in the exponential family.

In this case, $\log Z$ is referred to as the log-partition function or cumulant function. And the main result is that its gradient $\nabla_{\theta} \log Z$ is the expectation of the sufficient statistics. Furthermore, higher-order moments of the sufficient statistics can be attained by taking higher order derivatives of the log-partition function with respect to the parameters $\theta$.

Addressing the comment (which has now been edited) that:

[The gradient of the log of the normalizing constant] doesn't have a special name but it's the expectation of the sufficient statistics.

To prevent the risk of confusion, the result is that $\nabla_{\theta} \log Z$ is the expectation of the sufficient statistics, and not $\nabla_{x} \log Z$, which has no meaning I am aware of.

Furthermore, the result concerning $\nabla_{\theta} \log Z$ holds only when $p(x)$ is in the exponential family, and is not in general appropriate to assume when $p(x)$ is some arbitrary density function not in the exponential family.

In response to:

Can we say that $\nabla_{\theta} \log Z = \frac{1}{Z}\nabla_{\theta} \int_{\mathcal{X}} \tilde{p}(x) dx = \frac{1}{Z}\int_{\mathcal{X}} \nabla_{\theta} \tilde{p}(x) dx$?

Again, if $p(x)$ is in the exponential family, then interchanging the order of differentiation and integration is appropriate i.e. the operators can commute and the equality holds without further thought.

However, when $p(x)$ is not in the exponential family, and is some arbitrary density, then whether the derivative and integration operators are commutative depends on the range of integration and hence on the support of $X$. In the case where the range of integration is finite, you can appeal to Leibniz integral rule under certain conditions to justify this interchange.

In the case where $p(x)$ is an arbitrary density and the range of integration is infinite, the interchange of differentiation and integration operators are governed by Lebesgue's dominated convergence theorem. This theorem requires the machinery of measure theory and real analysis to rigorously state and to invoke. There are simpler corollaries to assess this without recourse to this machinery, and these are stated in section 2.4 of Statistical Inference by Casella and Berger (2004).

$\endgroup$
3
  • 1
    $\begingroup$ Thank you so much, this is exactly the kind of answer I was looking for! $\endgroup$ May 17 at 14:19
  • $\begingroup$ @Physics_Student. Just realised that I misread your comment and question - it doesn't make much sense to compute derivative with respect to $x$, because you've integrated out $x$. Rather the derivative is with respect to the parameter $\theta$, and $\nabla_x$ should be $\nabla_{\theta}$. Is that what you had in mind? $\endgroup$
    – microhaus
    May 17 at 15:02
  • $\begingroup$ @Physics_Student. I have now edited this to address the error, as it's a crucial one. If my reading of your question is not what you are interested in, consider rescinding acceptance. $\endgroup$
    – microhaus
    May 17 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.