Questions tagged [fisher-information]
The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.
321
questions
0
votes
0
answers
21
views
Can probabilistic (e.g., Fisher, Shannon) and non-probabilistic (e.g., Hartley, Kolmogorov) information types be jointly useful?
Suppose you draw a random sample from a probability distribution, with the objective of gaining information about a parameter of that distribution. The inferential usefulness of the probabilistic ...
- 326
0
votes
0
answers
19
views
Confusion about asymptotic distribution of the MLE and of the MAP
It's well known that the MLE $\hat{\theta}$ maximizes $f(y|\theta)$ and under regularity conditions has asymptotic distribution
$$N\left(\theta, \frac{I(\theta)}{J^2(\theta)} \right)$$ Where $I(\theta)...
- 151
2
votes
1
answer
82
views
scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)
I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
- 240
2
votes
0
answers
16
views
Coefficient standard error for "GLM" not in exponential family
For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
- 87
2
votes
1
answer
81
views
Fisher information and Expected Information for Gamma Distribution
I would like some help with calculating the Fisher Information $I_o(\beta)$ and the expected information for a gamma distribution defined by
\begin{align*}
f_X(x) = \frac{\beta^\alpha x^{\alpha - ...
- 99
4
votes
1
answer
47
views
Where does the efficient locally unbiased estimator $\tilde\theta(m)=\theta_0+\frac{F^{-1}\nabla p_\theta(m)}{p_\theta(m)}$ come from?
In Equation (9), page 9 of (Demkowicz-Dobrzanski et al. 2020), the authors mention that, given a probability distribution $p_{\boldsymbol\theta}(m)$ with hidden parameter $\boldsymbol\theta$ and $m$ ...
- 329
1
vote
1
answer
55
views
Profile likelihood vs quadratic log-likelihood approximation
I want to compare two alternative approaches for evaluating the uncertainty of the multi-dimensional MLE $\widehat \theta$ based on a log-likelihood function $l$:
Compute a Fisher-information-based ...
- 73
0
votes
1
answer
32
views
conditional expectation substitution in Fisher information [duplicate]
The Fisher information is given by
$$J(\theta) = -E\left[\frac{d^{2}\log p(x | \theta)}{d\theta^{2}} \bigg|~\theta\right]$$
To consider the Fisher information for a binomial parameter:
Let $p(x | \...
- 251
4
votes
1
answer
55
views
Using Jeffreys prior for Bernoulli distribution to find the prior of a transformation on p
The question goes like this: Use Jeffreys prior for Bernoulli distribution and find the prior for $\eta$ where: $$\eta(p) = \left(\frac{p}{1-p}\right) $$
So $\eta$ here is some kind of a ...
- 423
0
votes
0
answers
46
views
What is the Fisher information matrix in the logit model? [duplicate]
Assume that $Y_1,\dots, Y_n$ follows a Binomial distribution with probability of $p(d_i)$. Assume that the pdf of $Y_i$:
$$
f(p_i,Y_i)=\binom{n_i}{y_i}p_i^{y_i}(1-p_i)^{n_i-y_i}
$$
Assume that a model ...
- 625
2
votes
0
answers
76
views
Problem with the Fisher information matrix in case of N measurements of two observables
Let consider two observables, $x$ and $y$. Suppose that $y$ depends on the independent variable $x$ through the model $m(x; \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ is a vector of model ...
- 21
1
vote
2
answers
61
views
The distribution of $Y|aY+b$
What is the distribution of $Y|aY+b$? I'm assuming that $a\neq 0$. A related question here discusses the distribution of $X|X$, which is degenerate. My guess is that the distribution of $Y|aY+b$ ...
1
vote
2
answers
168
views
Fisher information vs Posterior Covariance
I have a parameter $\theta$ and data $y = f(\theta) + \mathrm{noise}$. My goal is finding the best fit for $\theta$ and assess the uncertainty I have on this best fit.
I see two competing approaches ...
- 73
1
vote
0
answers
40
views
Expectation of Fisher score not equal to 0 when parametrize Categorical distribution differently
Expectation of Fisher score should equal to zero. The prove can be found in many palces, such as wikipedia.
But I tried a categorical distribution that is not parameterizatized minimally, the expected ...
- 763
0
votes
0
answers
21
views
How to define the relation between the Fisher metric and KL divergence in the following case
Let we have a point cloud data consists of exactly $n$ distinct points in $\mathbb{R^d}$ that each each point clod is of the form $X=\{x_1,...,x_n/ x_i\in \mathbb{R^d},x_{i}\neq x_{j},i\neq j\}$. The ...
- 133
1
vote
0
answers
49
views
What is the difference between $\partial^2/\partial{\boldsymbol\theta}^2$ and $\partial^2/\partial{\boldsymbol\theta}\partial{\boldsymbol\theta}^T$?
I am reading the paper Efficient Computation of the Fisher Information Matrix in the EM Algorithm (Meng & Spall) and am unsure what the difference (if any) there is between the notation $\partial^...
- 111
3
votes
3
answers
365
views
Can the off-diagonal elements of Fisher information matrix be negative?
The concept of Fisher information is new to me and as I understand the diagonal elements of the Fisher information matrix (FIM) are proportional to mean square error (to be precise the inverse of FIM)....
- 151
1
vote
2
answers
135
views
Observed Fisher information for the binomial: How is $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ calculated?
I am currently studying the textbook In All Likelihood by Yudi Pawitan. Example 2.10 of chapter 2.5 Maximum and curvature of likelihood says the following:
Example 2.10: Based on $x$ from the ...
- 1,414
0
votes
0
answers
31
views
What is the role of information matrix in Likelihood estimation?
I couldn't grasp what it refers to exactly so I would like to understand how we use it:
from MLE, Score Vector is:
$$
S(\theta;y) = \frac{\partial l(\theta;y) }{\partial \theta}
$$
$l$ comes from the ...
- 21
2
votes
0
answers
60
views
Lost in Fisher information notation
I believe I understand the gist of what Fisher Information is, but I want to be rigorous and I am confused by notation. Also I believe I am masquerading mistakes by ...
- 193
0
votes
1
answer
71
views
Asymptotic covariance matrix of an ML estimator and Fisher information
Let $(Y_i, X_i)_{1\leq i \leq n}$ be i.i.d. such that $Y_i = (Y_i^A, Y_i^B)' \in \mathbb{R}^2$ and $X_i = (X_i^A, X_i^B)' \in \mathbb{R}^2$. Suppose that
$$
Y_i^A = X_i^A \beta_A + \epsilon_i^A,
$$
$$
...
- 113
0
votes
1
answer
37
views
Question about Fisher information matrix on a statistical model
Consider a family $S$ of probability density functions on $X$ which is defined as $p:X\to\mathbb{R}$ such that $p(x\geq 0)$ and $\int_{X}p(x)dx=1$.Suppose each element of $S$ may be parameterized ...
- 133
0
votes
0
answers
16
views
Goodness-of-fit and confidence intervals
I am having trouble grasping the subtleties of determining confidence intervals on parameters from degrading the goodness-of-fit (ie shifting $\chi^2$ or $2\ln {\cal L}$ by 1).
For instance, around ...
- 378
1
vote
1
answer
189
views
Fisher information for the negative binomial distribution
I have the negative binomial distribution and want to find the fisher information: $I(\theta) = V[\ell'(\theta)]$
How do i calculate this?
I know that the derivative of the log-likelihood is: $\ell'(\...
- 95
1
vote
3
answers
231
views
Deriving the asymptotic distribution using delta method
I have the density function:
$P_Y(y) = \sqrt{\frac{1}{2\pi y^3}} \exp\left(-\frac{(y-\mu)^2}{2\mu^2y}\right)$
If we define $r := \mu^2$ what is its asymptotic distribution?
The right answer is $\sqrt{...
- 95
0
votes
0
answers
79
views
Critical Value for a 2-sided Wald's test
Consider the following set of hypotheses:
$H_0:\theta = 1$
$H_1:\theta ≠ 1$
AFAIK, the Wald's test converges in distribution as follows:
$\sqrt{nI(\hat{\theta}_ {MLE})}(\hat{\theta}_{MLE}-\theta)\...
- 107
0
votes
0
answers
48
views
Fiding the test statistic, using wald test
Given the random sample $X_1,...,X_n \sim N(\mu, \sigma^2)$, I want to perform a Wald test for:
$\mathrm{H}_\mathrm{0}: \mu = \mathrm{\mu}_\mathrm{0}$
$\mathrm{H}_\mathrm{1}: \mu \neq \mathrm{\mu}_\...
- 423
3
votes
2
answers
102
views
How to prove $\mathcal{I}_{1}(\eta) = \mathcal{I}_{1}(\theta)[h'(\eta)]^{2}$ where $\mathcal{I}_{1}$ is the Fisher information and $\theta = h(\eta)$?
I am trying to apply the following definition of Fisher Information:
\begin{align*}
\mathcal{I}_{1}(\theta) = \mathbb{E}_{\theta}\left[\left(\frac{\partial}{\partial\theta}\ln f(x_{1}|\theta)\right)^{...
- 173
0
votes
0
answers
21
views
Error in sign when deriving Fisher information matrix for a linear model
We have that $Y_i ~\sim N(\beta x_i,\sigma^2)$ for $i=1,2, \dots,n$, all independent where $x_i$ is a known covariate. We would like to derive the Fisher information matirx.
I know that the $(1,1)$ ...
user358331
0
votes
0
answers
217
views
Find the Fisher Information for geometric distribution
Given $X1,\dotsc,Xn \sim \mathcal{Geo}(p)$ , and I need to find the MLE and the CI for the MLE.
I found the MLE for this distribution, using the maximum likelihood function: $L(p;X) = (1-p)^(Xi-1) * p$...
- 423
0
votes
0
answers
90
views
Intuitively, how is fisher information different from inverse of the variance of the likelihood?
If I observe data $X$ then Fisher information is supposed to tell me how concretely I can say that the inferred (mle) value of parameter produced this data (I am not sure if it is accurate to ...
- 2,231
4
votes
1
answer
98
views
What do you think of this proof for Fisher information?
I want to prove This formula:
The score function is basically the derivative of the maximum likelihood's log, so to get the information I make another derivative of that:
$$ -E[∂/∂θ s(X;θ)] = -E[∂/∂θ ...
- 409
0
votes
0
answers
84
views
Fisher information for predictions
Assume I have a model (linear regression, neural network, etc) in the form $g(\theta)$ and I assume that my data is generated according to $f(x; g(\theta))$ (eg $f$ is the pdf of a normal Gaussian ...
- 429
1
vote
0
answers
42
views
Derivation of an over-parametrized Fisher information for categorical distribution, with $n\ge3$
Having found interesting this earlier discussion I wonder if there is any specific example of building a Fisher information matrix (FIM) in case $n\ge3$. I tried myself, but every time, I am getting ...
- 11
7
votes
4
answers
646
views
Why the variance of Maximum Likelihood Estimator(MLE) will be less than Cramer-Rao Lower Bound(CRLB)?
Consider this example. Suppose we have three events to happen with probability $p_1=p_2=\frac{1}{2}\sin ^2\theta ,p_3=\cos ^2\theta $ respectively. And we suppose the true value $\theta _0=\frac{\pi}{...
- 115
0
votes
0
answers
17
views
Fisher information reparametrization formula in non-bijective cases
The formula writes as this:
$$I(X) = \left(\frac{\partial Z}{\partial X}\right)^\top I(Z) \left(\frac{\partial Z}{\partial X}\right),$$
in particular, if $Z = AX$, we have
$$I(X) = A^\top I(Z) A.$$
...
- 101
0
votes
0
answers
21
views
Parameter estimation of a communication channel (conditional probability distribution)
I am trying to find literature in statistics that deals with the following problem.
Given a parameter-dependent communication channel, mathematically described as a conditional probability $p_{\vec{\...
- 11
1
vote
0
answers
64
views
What's the consequences when Fisher Information Matrix(FIM) is not invertible?
From the point of the unbiased estimator, there will still be a lower bound of covariance which will have nothing to do with FIM when it's singular, so I want to know are there some references about ...
- 115
0
votes
0
answers
27
views
Why we do not define the reciprocal variance of the Minimum Variance Unbiased Estimators as the FIsher information?
If I give you data on death rate of rats in China and ask you to estimate the
population of Cuba based on that, you'll surely say that the data contains no information about the quantity to be ...
user318514
0
votes
1
answer
75
views
Is Hessian of neural nets with NLL loss positive semi-definite?
I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite
$$
\begin{aligned}
F(\theta)
&= E_{x \sim p_\theta}...
0
votes
0
answers
81
views
How to get Fisher information matrix from Likelihood?
Since det $R(k) = (1 + \sum_i S/N_i) det N(k)$, only the ex-
ponential part of the density function will depend on the
delays. Let the signal delay vector D be defined as
The likelihood function for $...
- 51
1
vote
1
answer
59
views
Which form of Jeffrey's prior can be used for a three-parameter distribution?
Let X be a random variable which follows a distribution, say S with parameters a, b and c. Knowing that or Assuming that a, b and c are independent of one another, which one is reasonable to do?
a) Is ...
- 364
2
votes
0
answers
137
views
Natural gradients with Moore–Penrose inverse of the Fisher information matrix
I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
0
votes
0
answers
78
views
Estimating the Cramér–Rao bound
Given a random vector $\boldsymbol{X}=(X_1,X_2,...)$, which can be described by the sum of a multivariant Poisson distribution $\alpha P(\boldsymbol{\lambda})$ with a scaling factor $\alpha$ and ...
- 111
0
votes
0
answers
32
views
Fisher Information usage [duplicate]
regarding fisher information in wikipedia, it is mentioned that fisher information is used in optimsl design of experiments. so an example is needed to illustrate how fisher information is used in ...
0
votes
0
answers
62
views
How to calculate the Fisher Information Matrix in GARCH?
How to calculate the Fisher Information Matrix in GARCH?
I want to know how to calculate the standard errors in them and without the empirical fisher information, empirical likelihood, I have no idea ...
user318514
1
vote
1
answer
170
views
Demonstration and Interpretation between a Fisher matrix and its dual space which is covariance matrix
I have a simple (maybe not) issue about the interpretation of the link between Fisher information matrix and its inverse which is the covariance matrix.
How to formulate that a line of Covariance ...
user226073
0
votes
0
answers
128
views
Fisher matrix for a discrete distribution
Let $\mathbf{X} = \{X_1, \ldots, X_n\}$ be a sample of i.i.d. variables following a discrete distribution with parameters $\mathbf{p}^T = (p_1, p_2, p_3)$. How can I find the Fisher information matrix ...
- 405
6
votes
1
answer
224
views
Expected Fisher information isn't positive definite for truncated normal with heteroskedasticity
This question is about having a non-positive-definite expected Fisher information in a normal model in which observations have different dispersions.
Consider this simple normal model:
$$Y_i \sim N(\...
- 3,581
0
votes
0
answers
48
views
How to calculate a multivariable fisher information matrix
I do not understand what is the definition for
$$\mathcal I(\theta) = -E[H(\theta)]$$
where $H$ is the hessian of log-likelihood function
how should I calculate this if $\theta$ is a vector. What ...
- 1