So I believe the sampling distribution of the likelihood function is a basic idea in frequentist statistics. For example, the Fisher information $\text{Var}_x(\nabla_\theta \log P(x|\theta))$ which tell us how sensitive $\log P(x|\theta)$ is wrt the data $x$.

Is there an analogous way to view an objective/loss function of an inference procedure as as a random function wrt data? It seems weird that loss functions treat the data $x$ as fixed. If an inference model assumes an underlying distribution of $x$, it should be able to describe the distribution of different loss functions that arise due to difference in sampling $x$.

One example I am wondering about is the ELBO in variational Bayes.

  • 1
    $\begingroup$ I think there is an issue with your expressions for the score and Fisher information. You frame both quantities as supplying information on "how the log likelihood changes wrt to the distribution of data". And you state in symbols that both involve computing $\nabla_x$, the gradient w.r.t $x$ (which presumably means w.r.t to the data). $\endgroup$ – microhaus Mar 3 at 21:45
  • $\begingroup$ The score function and Fisher information both involve derivatives/gradients w.r.t to the parameters $\theta$, that is, it should be $\nabla_{\theta}$, because the likelihood and log-likelihood are functions of the parameters. $\endgroup$ – microhaus Mar 3 at 21:47
  • $\begingroup$ Does that alter the substance of the question you have posed? $\endgroup$ – microhaus Mar 3 at 21:48
  • $\begingroup$ I am also a little confused when you use the phrases "sampling distribution of the likelihood" and "sampling distribution of the objective function"; to the best of my elementary statistical knowledge, its conventional to consider the sampling distribution of an estimator or a statistic. Can you elaborate a little further on what you mean? $\endgroup$ – microhaus Mar 3 at 21:53
  • $\begingroup$ @microhaus ah yes, I got the score function/fisher info wrong, thanks. The quantity $\nabla_x P(x|\theta)$ is actually what I am interested in: something that relates the randomness of the data sample $x$ to the likelihood of some estimator $\theta$ with respect to $x$. A related idea is the robustness of an estimator. $\endgroup$ – 900edges Mar 4 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.