Skip to main content

Questions tagged [fisher-information]

The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Should one account for the known variance of fixed X when estimating its relationship with random Y?

In Aldrich (2005), and specifically in sections 10 and 11, the author describes the sufficient statistic for the parameter $\beta$ in the simple regression of random $Y$ on fixed $X$, with a bivariate ...
virtuolie's user avatar
  • 642
3 votes
2 answers
59 views

Confusion over Fisher-scoring algorithm

Given a probability model $f(X;\theta)$ and a set of i.i.d. observations $x_1,\ldots,x_n$ which we assume to be drawn from some true parameter $f(X; \theta_0)$, we can perform maximum-likelihood ...
shem's user avatar
  • 186
7 votes
3 answers
462 views

Derivative of the Score Function in Fisher Information

I'm studying Fisher Information and am trying to develop an intuitive understanding. Keep in mind I only have bachelor level mathematics background so I would appreciate an answer that is more ...
Ryan's user avatar
  • 73
0 votes
0 answers
13 views

Specific Question about Deriving the Fisher Information for a Complex Multivariate Normal Distribution [duplicate]

I am starting with the following form for the likelihood function for a complex multivariate normal distribution for data with dimension $d$ and mean $\boldsymbol \mu$: $$ p(\mathbf x|\boldsymbol \...
StackMonkey's user avatar
0 votes
0 answers
14 views

Connection between mean update in CMA-ES and gradient of expected fitness

I currently learn about black-box optimization and CMA-ES. Now, I try to understand some of the theoretical foundations of it. The update of the mean in classic CMA-ES is as follows: $$m \leftarrow m +...
HansDoe's user avatar
1 vote
1 answer
28 views

Expected value of derivative derivative squared log equivalent to regularity condition

I'm given the following information (*): $\mathbb E[(\frac{\partial}{\partial \theta} log f(X_1, ..., X_n; \theta))^2] = - \mathbb E[(\frac{\partial^2}{\partial^2 \theta} log f(X_1, ..., X_n; \theta))$...
JohnD's user avatar
  • 117
1 vote
1 answer
34 views

Why can we get better asymptotic global estimators even for IID random variables?

Let $X_1,...,X_N$ be IID random variables sampled from a parametrised distribution $p_\theta$, and suppose my goal is to retrieve $\theta$ from these samples. We know that the MLE provides an ...
glS's user avatar
  • 383
1 vote
0 answers
43 views

How does reparametrization of the Fisher information matrix change the variance expression for the sufficient statistics?

If I have an exponential family distribution of the form $$p_{\theta}(x) = e^{\theta^T\cdot t(x) - \psi(\theta)},$$ where $\theta$ is a vector of parameters, $t(x)$ is a vector of sufficient ...
absolutelyzeroEQ's user avatar
0 votes
0 answers
31 views

Bootstrap method with 2 Fisher matrices in order to do the cross-correlations between both

I have 2 Fisher matrices where each colum/row represents the information (in Fisher's sense) of astrophysical parameters. These parameters are in the same order for both matrices. Now, I would like to ...
foutou_10's user avatar
1 vote
1 answer
68 views

What am I doing wrong when finding the Fisher information of a binomial distribution with $n=2$?

I am trying to find the Fisher information of a binomial distribution where $n=2$ and $p=\theta$. I have the log-likelihood function as $$n\ln2 + \sum^{n}_{i=1}x_i\ln \theta + (2n-\sum^{n}_{i=1}x_i)(...
Peverel Shipley's user avatar
3 votes
0 answers
89 views

Can Fisher Information Matrix be calculated numerically through finite differentiation?

In GLMs (generalized linear models), the negative of the Fisher information matrix takes the form of a cross product between covariates $\mathbf{X}$ and a diagonal matrix: $$ \mathbf{X}^T \mathbf{D} \...
anymous.asker's user avatar
0 votes
0 answers
31 views

Fisher information in Laplace approximation

Let $X$ and $Y$ be continuous random variables with Probability Density Function (PDF) $f_X$ and $f_Y$, respectively. Upon observing $Y=y$, the log-posterior PDF is given by Bayes' rule in log form: $$...
W. Zhu's user avatar
  • 135
0 votes
0 answers
33 views

Looking for an intuitive explanation of D-Criterion for Optimal Design Problem

I know only a little about Fisher information and optimal experimental design, but I'm trying to better understand the subject. If I have an experiment composed of a single detector and my detector ...
David G.'s user avatar
  • 149
3 votes
1 answer
50 views

Cramér-Rao / Wolfowitz bound with nuisance parameter

Let $F$ be a distribution with two parameters, $\theta$ and $\phi$, whose values are non-random but unknown. Consider a sampling procedure in which $N$ samples $x_1, \ldots x_N$ are obtained from i.i....
Luis Mendo's user avatar
  • 1,099
0 votes
0 answers
60 views

Why is Fisher information the Precision of MLE rather than Covariance? [duplicate]

I'm confused because if FIM is $I(\theta)=Var_x(s(\theta|x))=Var_x({d \over d \theta} log(L(\theta|x)))$ (variance of the score) and the MLE estimates are $\theta^*=dt * \sum^\infty_{t=0}{d\over d\...
profPlum's user avatar
  • 401
1 vote
0 answers
42 views

Derive Cramer-Rao lower bound for $Var(\hat{\theta})$ given that $\mathbb{E}[\hat{\theta}U]=1$

I am trying to derive the Cramer-Rao lower bound for $Var(\hat{\theta})$ given that we already know $\mathbb{E}[U]=0$, $Var(U)=I(\theta)$ and $\mathbb{E}[\hat{\theta}U]=1$. I am struggling with using ...
Lucas's user avatar
  • 11
0 votes
0 answers
30 views

Fisher information or Bayesian Uncertainty for non-parametric distributions

This question sounds ridiculous, let me clarify motivation: Fisher information & Bayesian inference uncertainty seemed very cool to me because they can effectively tell you "how ...
profPlum's user avatar
  • 401
0 votes
0 answers
38 views

Fisher information for gamma-Poisson distribution

I have $Y$ that is a gamma-Poisson distribution with mean $\mu$ and $\kappa$ is the overdispersion. I'm trying to obtain fisher information but i don't know how to solve expected value of trigamma ...
ali's user avatar
  • 71
5 votes
1 answer
93 views

difference between GLM covariance matrix from MLE vs. IRLS for non-canonical link

Someone asked a question on Stack Overflow where they noted a difference between Minitab and R (glm) results for the variance-covariance matrix of the parameters, ...
Ben Bolker's user avatar
  • 44.5k
0 votes
0 answers
22 views

Is the score function in Langevin dynamics related to the informant (score)?

In Langevin dynamics and diffusion models we see the score $\nabla_{\mathbf{x}}\log p(\mathbf{x})$. Here the notation seems to suggest we're taking partial derivatives w.r.t the free $\mathbf{x}$ ...
flinty's user avatar
  • 151
6 votes
4 answers
593 views

What is the information in an exact p-value?

Consider the following two statistical principles: 1) an exact test's $p$-value gives the exact frequency with which the observed random sample appears by chance, i.e., under a true null hypothesis; ...
virtuolie's user avatar
  • 642
0 votes
1 answer
140 views

How (or can) you formulate the Fisher information matrix in terms of a loss function, specifically cross-entropy loss?

I recently saw the following formulation of the Fisher information matrix in a paper on Transformer pruning: $$ \mathcal{I} := \frac{1}{|D|} \sum_{(x,y) \in D} \left( \frac{\partial \mathcal{L}(x,y;1)}...
premed's user avatar
  • 1
4 votes
0 answers
75 views

Intuition/meaning of information geometry distances and geodesics?

In information geometry, we consider a manifold of probability distributions, together with the Fisher Information metric (given by the Fisher Information matrix). I have some intuition (see ...
dherrera's user avatar
  • 1,328
0 votes
0 answers
151 views

$3\sigma$ region of locally non-parabolic cost function

Consider you have an optimization problem where your are given: Dataset $(\mathbf{x}, \mathbf{y})$ $\mathbf{x}$ is perfectly known $\mathbf{y} = \mathbf{y}_{\text{true}} + \mathbf{n}$ with $n_i \...
Ron's user avatar
  • 121
1 vote
1 answer
29 views

Why does the score test work for values longer in the tail that have a small log-likelihood derivative?

The score test says that we take the derivative of the log-likelihood at $H_0$ and divide it by the fisher information at $H_0$. $U(\theta )={\frac {\partial \log L(\theta \mid x)}{\partial \theta }}.$...
Estimate the estimators's user avatar
0 votes
1 answer
65 views

Between steps for fisher information matrix element using Poisson regression?

I am currently working through some math related to my work, and trying to understand how the individual pieces of the following equations come together for the Fisher information matrix expression (...
JHS's user avatar
  • 3
3 votes
1 answer
2k views

Approximate the Fisher information matrix of a multivariate normal distribution

For $d\geq 2$, consider the d-dimensional multivariate normal distribution $\mathcal N(x|\mu,\Sigma)$ whose the log of density is given by $$ l(x;\mu,\Sigma)=-\frac{d}{2}\log(2\pi)-\frac{1}{2}\log|\...
KNN's user avatar
  • 153
0 votes
0 answers
118 views

How to compute the covariance matrix for a mixture model estimated by the EM algorithm

I am trying to compute the observed Fisher information matrix for a mixture model estimated by the EM algorithm. My original thought is to simply compute the second derivative of a mixture density. ...
Lydia2kkx's user avatar
0 votes
1 answer
42 views

Obtaining the geometry of $\varphi_{\theta}(x)$ where the mean and variance are $\theta$-dependent

Consider an (unnormalized) univariate distribution in the exponential family, which is in canonical form: $$\varphi_{\theta}(x)= \exp \bigg( \frac{\theta}{\log x} \bigg)$$ $x\in(0,1).$ $\varphi_{\...
geocalc33's user avatar
3 votes
1 answer
46 views

Why $\nabla_{\theta} \log p(y;\theta) = \frac{\nabla_{\theta}p(y;\theta)}{p(y;\theta)}$?

I'm solving a problem for cs 229, problem description in below when i check the answer it mentioned the given equation, but I don't undertand why is that. I want to know why, anyone give me some sort ...
Yiffany's user avatar
  • 185
4 votes
1 answer
297 views

Cramer-Rao lower bound for the variance of unbiased estimators of $\theta = \frac{\mu}{\sigma}$

Let $X_1, \cdots, X_n$ be a sample from the $N(\mu, \sigma^2)$ density, where $\mu, \sigma^2$ are unknown. I want to find a lower bound $L_n$ which is valid for all sample-sizes $n$ for the variance ...
Oscar24680's user avatar
1 vote
0 answers
134 views

What does it mean if the Fisher information matrix in block diagonal form for the parameter estimation?

If for a given problem the fisher information matrix is diagonal or block diagonal, from the estimation point of view, what does it mean? Does it mean, for 2 parameters in different blocks, the ...
Mark's user avatar
  • 277
1 vote
0 answers
31 views

Which Fisher information to use to obtain Cramer-Rao bound in expectation-maximization?

I have a rather limited understanding of statistical estimation theory so I apologize if my question is strange or trivial. Say I have an expectation-maximization-based algorithm for determining the ...
DangerousTim's user avatar
1 vote
1 answer
137 views

Fisher Information for $\bar{X}^2 - \frac{\sigma^2}{n}$ with $X_1, \dots, X_n$ normally distributed

I need to find the Fisher Information for $T = \bar{X}^2 - \frac{\sigma^2}{n}$ with $X_1, \dots, X_n$ normally distributed sample with unknow mean $\mu$ and know variance $\sigma^2$. For this I'm ...
Peter Languilla's user avatar
2 votes
1 answer
98 views

Confusion about asymptotic distribution of the MLE and of the MAP

It's well known that the MLE $\hat{\theta}$ maximizes $f(y\mid\theta)$ and under regularity conditions has asymptotic distribution $$N\left(\theta, \frac{I(\theta)}{J^2(\theta)} \right)$$ where $I(\...
ThighCrush's user avatar
2 votes
1 answer
942 views

scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)

I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
Jerem Lachkar's user avatar
2 votes
0 answers
41 views

Coefficient standard error for "GLM" not in exponential family

For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
David Wang's user avatar
2 votes
1 answer
2k views

Fisher information and Expected Information for Gamma Distribution

I would like some help with calculating the Fisher Information $I_o(\beta)$ and the expected information for a gamma distribution defined by \begin{align*} f_X(x) = \frac{\beta^\alpha x^{\alpha - ...
DanielMariam's user avatar
4 votes
1 answer
77 views

Where does the efficient locally unbiased estimator $\tilde\theta(m)=\theta_0+\frac{F^{-1}\nabla p_\theta(m)}{p_\theta(m)}$ come from?

In Equation (9), page 9 of (Demkowicz-Dobrzanski et al. 2020), the authors mention that, given a probability distribution $p_{\boldsymbol\theta}(m)$ with hidden parameter $\boldsymbol\theta$ and $m$ ...
glS's user avatar
  • 383
1 vote
1 answer
78 views

Profile likelihood vs quadratic log-likelihood approximation

I want to compare two alternative approaches for evaluating the uncertainty of the multi-dimensional MLE $\widehat \theta$ based on a log-likelihood function $l$: Compute a Fisher-information-based ...
G. Gare's user avatar
  • 83
0 votes
1 answer
64 views

conditional expectation substitution in Fisher information [duplicate]

The Fisher information is given by $$J(\theta) = -E\left[\frac{d^{2}\log p(x | \theta)}{d\theta^{2}} \bigg|~\theta\right]$$ To consider the Fisher information for a binomial parameter: Let $p(x | \...
Physkid's user avatar
  • 251
4 votes
1 answer
243 views

Using Jeffreys prior for Bernoulli distribution to find the prior of a transformation on p

The question goes like this: Use Jeffreys prior for Bernoulli distribution and find the prior for $\eta$ where: $$\eta(p) = \left(\frac{p}{1-p}\right) $$ So $\eta$ here is some kind of a ...
CORy's user avatar
  • 543
2 votes
0 answers
149 views

Problem with the Fisher information matrix in case of N measurements of two observables

Let consider two observables, $x$ and $y$. Suppose that $y$ depends on the independent variable $x$ through the model $m(x; \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ is a vector of model ...
Wil's user avatar
  • 21
2 votes
2 answers
81 views

The distribution of $Y|aY+b$

What is the distribution of $Y|aY+b$? I'm assuming that $a\neq 0$. A related question here discusses the distribution of $X|X$, which is degenerate. My guess is that the distribution of $Y|aY+b$ ...
Aaron Hendrickson's user avatar
1 vote
2 answers
547 views

Fisher information vs Posterior Covariance

I have a parameter $\theta$ and data $y = f(\theta) + \mathrm{noise}$. My goal is finding the best fit for $\theta$ and assess the uncertainty I have on this best fit. I see two competing approaches ...
G. Gare's user avatar
  • 83
1 vote
0 answers
62 views

Expectation of Fisher score not equal to 0 when parametrize Categorical distribution differently

Expectation of Fisher score should equal to zero. The prove can be found in many palces, such as wikipedia. But I tried a categorical distribution that is not parameterizatized minimally, the expected ...
Haotian Chen's user avatar
1 vote
0 answers
56 views

What is the difference between $\partial^2/\partial{\boldsymbol\theta}^2$ and $\partial^2/\partial{\boldsymbol\theta}\partial{\boldsymbol\theta}^T$?

I am reading the paper Efficient Computation of the Fisher Information Matrix in the EM Algorithm (Meng & Spall) and am unsure what the difference (if any) there is between the notation $\partial^...
epsilonz3ro's user avatar
3 votes
3 answers
1k views

Can the off-diagonal elements of Fisher information matrix be negative?

The concept of Fisher information is new to me and as I understand the diagonal elements of the Fisher information matrix (FIM) are proportional to mean square error (to be precise the inverse of FIM)....
User101's user avatar
  • 171
4 votes
2 answers
1k views

Observed Fisher information for the binomial: How is $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ calculated?

I am currently studying the textbook In All Likelihood by Yudi Pawitan. Example 2.10 of chapter 2.5 Maximum and curvature of likelihood says the following: Example 2.10: Based on $x$ from the ...
The Pointer's user avatar
  • 2,086
0 votes
0 answers
43 views

What is the role of information matrix in Likelihood estimation?

I couldn't grasp what it refers to exactly so I would like to understand how we use it: from MLE, Score Vector is: $$ S(\theta;y) = \frac{\partial l(\theta;y) }{\partial \theta} $$ $l$ comes from the ...
Tatanik501's user avatar

1
2 3 4 5
7