Questions tagged [fisher-information]

The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.

Filter by
Sorted by
Tagged with
0 votes
0 answers
21 views

Can probabilistic (e.g., Fisher, Shannon) and non-probabilistic (e.g., Hartley, Kolmogorov) information types be jointly useful?

Suppose you draw a random sample from a probability distribution, with the objective of gaining information about a parameter of that distribution. The inferential usefulness of the probabilistic ...
virtuolie's user avatar
  • 326
0 votes
0 answers
19 views

Confusion about asymptotic distribution of the MLE and of the MAP

It's well known that the MLE $\hat{\theta}$ maximizes $f(y|\theta)$ and under regularity conditions has asymptotic distribution $$N\left(\theta, \frac{I(\theta)}{J^2(\theta)} \right)$$ Where $I(\theta)...
ThighCrush's user avatar
2 votes
1 answer
82 views

scipy minimize gives a hess_inv that is completely different from inv(statmodel.approx_hess)

I'm fitting a model with MLE using scipy.minimize (method BFGS). I want to have the hessian to compute its inverse and retrieve the standard error of each parameter....
Jerem Lachkar's user avatar
2 votes
0 answers
16 views

Coefficient standard error for "GLM" not in exponential family

For GLMs in the exponential family, we can obtain the standard errors for the regression coefficients as a function of the diagonal of the fisher information matrix. Does this still hold if the ...
David Wang's user avatar
2 votes
1 answer
81 views

Fisher information and Expected Information for Gamma Distribution

I would like some help with calculating the Fisher Information $I_o(\beta)$ and the expected information for a gamma distribution defined by \begin{align*} f_X(x) = \frac{\beta^\alpha x^{\alpha - ...
DanielMariam's user avatar
4 votes
1 answer
47 views

Where does the efficient locally unbiased estimator $\tilde\theta(m)=\theta_0+\frac{F^{-1}\nabla p_\theta(m)}{p_\theta(m)}$ come from?

In Equation (9), page 9 of (Demkowicz-Dobrzanski et al. 2020), the authors mention that, given a probability distribution $p_{\boldsymbol\theta}(m)$ with hidden parameter $\boldsymbol\theta$ and $m$ ...
glS's user avatar
  • 329
1 vote
1 answer
55 views

Profile likelihood vs quadratic log-likelihood approximation

I want to compare two alternative approaches for evaluating the uncertainty of the multi-dimensional MLE $\widehat \theta$ based on a log-likelihood function $l$: Compute a Fisher-information-based ...
G. Gare's user avatar
  • 73
0 votes
1 answer
32 views

conditional expectation substitution in Fisher information [duplicate]

The Fisher information is given by $$J(\theta) = -E\left[\frac{d^{2}\log p(x | \theta)}{d\theta^{2}} \bigg|~\theta\right]$$ To consider the Fisher information for a binomial parameter: Let $p(x | \...
Physkid's user avatar
  • 251
4 votes
1 answer
55 views

Using Jeffreys prior for Bernoulli distribution to find the prior of a transformation on p

The question goes like this: Use Jeffreys prior for Bernoulli distribution and find the prior for $\eta$ where: $$\eta(p) = \left(\frac{p}{1-p}\right) $$ So $\eta$ here is some kind of a ...
CORy's user avatar
  • 423
0 votes
0 answers
46 views

What is the Fisher information matrix in the logit model? [duplicate]

Assume that $Y_1,\dots, Y_n$ follows a Binomial distribution with probability of $p(d_i)$. Assume that the pdf of $Y_i$: $$ f(p_i,Y_i)=\binom{n_i}{y_i}p_i^{y_i}(1-p_i)^{n_i-y_i} $$ Assume that a model ...
Hermi's user avatar
  • 625
2 votes
0 answers
76 views

Problem with the Fisher information matrix in case of N measurements of two observables

Let consider two observables, $x$ and $y$. Suppose that $y$ depends on the independent variable $x$ through the model $m(x; \boldsymbol{\theta})$, where $\boldsymbol{\theta}$ is a vector of model ...
Wil's user avatar
  • 21
1 vote
2 answers
61 views

The distribution of $Y|aY+b$

What is the distribution of $Y|aY+b$? I'm assuming that $a\neq 0$. A related question here discusses the distribution of $X|X$, which is degenerate. My guess is that the distribution of $Y|aY+b$ ...
Aaron Hendrickson's user avatar
1 vote
2 answers
168 views

Fisher information vs Posterior Covariance

I have a parameter $\theta$ and data $y = f(\theta) + \mathrm{noise}$. My goal is finding the best fit for $\theta$ and assess the uncertainty I have on this best fit. I see two competing approaches ...
G. Gare's user avatar
  • 73
1 vote
0 answers
40 views

Expectation of Fisher score not equal to 0 when parametrize Categorical distribution differently

Expectation of Fisher score should equal to zero. The prove can be found in many palces, such as wikipedia. But I tried a categorical distribution that is not parameterizatized minimally, the expected ...
Haotian Chen's user avatar
0 votes
0 answers
21 views

How to define the relation between the Fisher metric and KL divergence in the following case

Let we have a point cloud data consists of exactly $n$ distinct points in $\mathbb{R^d}$ that each each point clod is of the form $X=\{x_1,...,x_n/ x_i\in \mathbb{R^d},x_{i}\neq x_{j},i\neq j\}$. The ...
Andyale's user avatar
  • 133
1 vote
0 answers
49 views

What is the difference between $\partial^2/\partial{\boldsymbol\theta}^2$ and $\partial^2/\partial{\boldsymbol\theta}\partial{\boldsymbol\theta}^T$?

I am reading the paper Efficient Computation of the Fisher Information Matrix in the EM Algorithm (Meng & Spall) and am unsure what the difference (if any) there is between the notation $\partial^...
epsilonz3ro's user avatar
3 votes
3 answers
365 views

Can the off-diagonal elements of Fisher information matrix be negative?

The concept of Fisher information is new to me and as I understand the diagonal elements of the Fisher information matrix (FIM) are proportional to mean square error (to be precise the inverse of FIM)....
User101's user avatar
  • 151
1 vote
2 answers
135 views

Observed Fisher information for the binomial: How is $I(\hat{\theta}) = \frac{n}{\hat{\theta}(1 - \hat{\theta})}$ calculated?

I am currently studying the textbook In All Likelihood by Yudi Pawitan. Example 2.10 of chapter 2.5 Maximum and curvature of likelihood says the following: Example 2.10: Based on $x$ from the ...
The Pointer's user avatar
  • 1,414
0 votes
0 answers
31 views

What is the role of information matrix in Likelihood estimation?

I couldn't grasp what it refers to exactly so I would like to understand how we use it: from MLE, Score Vector is: $$ S(\theta;y) = \frac{\partial l(\theta;y) }{\partial \theta} $$ $l$ comes from the ...
Tatanik501's user avatar
2 votes
0 answers
60 views

Lost in Fisher information notation

I believe I understand the gist of what Fisher Information is, but I want to be rigorous and I am confused by notation. Also I believe I am masquerading mistakes by ...
Fred Guth's user avatar
  • 193
0 votes
1 answer
71 views

Asymptotic covariance matrix of an ML estimator and Fisher information

Let $(Y_i, X_i)_{1\leq i \leq n}$ be i.i.d. such that $Y_i = (Y_i^A, Y_i^B)' \in \mathbb{R}^2$ and $X_i = (X_i^A, X_i^B)' \in \mathbb{R}^2$. Suppose that $$ Y_i^A = X_i^A \beta_A + \epsilon_i^A, $$ $$ ...
Sky's user avatar
  • 113
0 votes
1 answer
37 views

Question about Fisher information matrix on a statistical model

Consider a family $S$ of probability density functions on $X$ which is defined as $p:X\to\mathbb{R}$ such that $p(x\geq 0)$ and $\int_{X}p(x)dx=1$.Suppose each element of $S$ may be parameterized ...
Andyale's user avatar
  • 133
0 votes
0 answers
16 views

Goodness-of-fit and confidence intervals

I am having trouble grasping the subtleties of determining confidence intervals on parameters from degrading the goodness-of-fit (ie shifting $\chi^2$ or $2\ln {\cal L}$ by 1). For instance, around ...
Mister Mak's user avatar
1 vote
1 answer
189 views

Fisher information for the negative binomial distribution

I have the negative binomial distribution and want to find the fisher information: $I(\theta) = V[\ell'(\theta)]$ How do i calculate this? I know that the derivative of the log-likelihood is: $\ell'(\...
0xcc's user avatar
  • 95
1 vote
3 answers
231 views

Deriving the asymptotic distribution using delta method

I have the density function: $P_Y(y) = \sqrt{\frac{1}{2\pi y^3}} \exp\left(-\frac{(y-\mu)^2}{2\mu^2y}\right)$ If we define $r := \mu^2$ what is its asymptotic distribution? The right answer is $\sqrt{...
0xcc's user avatar
  • 95
0 votes
0 answers
79 views

Critical Value for a 2-sided Wald's test

Consider the following set of hypotheses: $H_0:\theta = 1$ $H_1:\theta ≠ 1$ AFAIK, the Wald's test converges in distribution as follows: $\sqrt{nI(\hat{\theta}_ {MLE})}(\hat{\theta}_{MLE}-\theta)\...
pecer10012's user avatar
0 votes
0 answers
48 views

Fiding the test statistic, using wald test

Given the random sample $X_1,...,X_n \sim N(\mu, \sigma^2)$, I want to perform a Wald test for: $\mathrm{H}_\mathrm{0}: \mu = \mathrm{\mu}_\mathrm{0}$ $\mathrm{H}_\mathrm{1}: \mu \neq \mathrm{\mu}_\...
CORy's user avatar
  • 423
3 votes
2 answers
102 views

How to prove $\mathcal{I}_{1}(\eta) = \mathcal{I}_{1}(\theta)[h'(\eta)]^{2}$ where $\mathcal{I}_{1}$ is the Fisher information and $\theta = h(\eta)$?

I am trying to apply the following definition of Fisher Information: \begin{align*} \mathcal{I}_{1}(\theta) = \mathbb{E}_{\theta}\left[\left(\frac{\partial}{\partial\theta}\ln f(x_{1}|\theta)\right)^{...
user1234's user avatar
  • 173
0 votes
0 answers
21 views

Error in sign when deriving Fisher information matrix for a linear model

We have that $Y_i ~\sim N(\beta x_i,\sigma^2)$ for $i=1,2, \dots,n$, all independent where $x_i$ is a known covariate. We would like to derive the Fisher information matirx. I know that the $(1,1)$ ...
user avatar
0 votes
0 answers
217 views

Find the Fisher Information for geometric distribution

Given $X1,\dotsc,Xn \sim \mathcal{Geo}(p)$ , and I need to find the MLE and the CI for the MLE. I found the MLE for this distribution, using the maximum likelihood function: $L(p;X) = (1-p)^(Xi-1) * p$...
CORy's user avatar
  • 423
0 votes
0 answers
90 views

Intuitively, how is fisher information different from inverse of the variance of the likelihood?

If I observe data $X$ then Fisher information is supposed to tell me how concretely I can say that the inferred (mle) value of parameter produced this data (I am not sure if it is accurate to ...
figs_and_nuts's user avatar
4 votes
1 answer
98 views

What do you think of this proof for Fisher information?

I want to prove This formula: The score function is basically the derivative of the maximum likelihood's log, so to get the information I make another derivative of that: $$ -E[∂/∂θ s(X;θ)] = -E[∂/∂θ ...
Programming Noob's user avatar
0 votes
0 answers
84 views

Fisher information for predictions

Assume I have a model (linear regression, neural network, etc) in the form $g(\theta)$ and I assume that my data is generated according to $f(x; g(\theta))$ (eg $f$ is the pdf of a normal Gaussian ...
Ant's user avatar
  • 429
1 vote
0 answers
42 views

Derivation of an over-parametrized Fisher information for categorical distribution, with $n\ge3$

Having found interesting this earlier discussion I wonder if there is any specific example of building a Fisher information matrix (FIM) in case $n\ge3$. I tried myself, but every time, I am getting ...
A5kar's user avatar
  • 11
7 votes
4 answers
646 views

Why the variance of Maximum Likelihood Estimator(MLE) will be less than Cramer-Rao Lower Bound(CRLB)?

Consider this example. Suppose we have three events to happen with probability $p_1=p_2=\frac{1}{2}\sin ^2\theta ,p_3=\cos ^2\theta $ respectively. And we suppose the true value $\theta _0=\frac{\pi}{...
narip's user avatar
  • 115
0 votes
0 answers
17 views

Fisher information reparametrization formula in non-bijective cases

The formula writes as this: $$I(X) = \left(\frac{\partial Z}{\partial X}\right)^\top I(Z) \left(\frac{\partial Z}{\partial X}\right),$$ in particular, if $Z = AX$, we have $$I(X) = A^\top I(Z) A.$$ ...
metricspace's user avatar
0 votes
0 answers
21 views

Parameter estimation of a communication channel (conditional probability distribution)

I am trying to find literature in statistics that deals with the following problem. Given a parameter-dependent communication channel, mathematically described as a conditional probability $p_{\vec{\...
justmyfault's user avatar
1 vote
0 answers
64 views

What's the consequences when Fisher Information Matrix(FIM) is not invertible?

From the point of the unbiased estimator, there will still be a lower bound of covariance which will have nothing to do with FIM when it's singular, so I want to know are there some references about ...
narip's user avatar
  • 115
0 votes
0 answers
27 views

Why we do not define the reciprocal variance of the Minimum Variance Unbiased Estimators as the FIsher information?

If I give you data on death rate of rats in China and ask you to estimate the population of Cuba based on that, you'll surely say that the data contains no information about the quantity to be ...
user avatar
0 votes
1 answer
75 views

Is Hessian of neural nets with NLL loss positive semi-definite?

I learned that expected Hessian of negative log likelihood is the same as Fisher information matrix, which is known to be positive semi-definite $$ \begin{aligned} F(\theta) &= E_{x \sim p_\theta}...
Hohyun Kim's user avatar
0 votes
0 answers
81 views

How to get Fisher information matrix from Likelihood?

Since det $R(k) = (1 + \sum_i S/N_i) det N(k)$, only the ex- ponential part of the density function will depend on the delays. Let the signal delay vector D be defined as The likelihood function for $...
Amartya's user avatar
  • 51
1 vote
1 answer
59 views

Which form of Jeffrey's prior can be used for a three-parameter distribution?

Let X be a random variable which follows a distribution, say S with parameters a, b and c. Knowing that or Assuming that a, b and c are independent of one another, which one is reasonable to do? a) Is ...
RRMT's user avatar
  • 364
2 votes
0 answers
137 views

Natural gradients with Moore–Penrose inverse of the Fisher information matrix

I'd like to show you my rough sketch for scaling up natural gradients to deep neural networks that appears to be easy to automate just like automatic differentiation. I think there must be a flaw ...
all feedback welcome's user avatar
0 votes
0 answers
78 views

Estimating the Cramér–Rao bound

Given a random vector $\boldsymbol{X}=(X_1,X_2,...)$, which can be described by the sum of a multivariant Poisson distribution $\alpha P(\boldsymbol{\lambda})$ with a scaling factor $\alpha$ and ...
user_na's user avatar
  • 111
0 votes
0 answers
32 views

Fisher Information usage [duplicate]

regarding fisher information in wikipedia, it is mentioned that fisher information is used in optimsl design of experiments. so an example is needed to illustrate how fisher information is used in ...
Khaleel Abdul Hameed's user avatar
0 votes
0 answers
62 views

How to calculate the Fisher Information Matrix in GARCH?

How to calculate the Fisher Information Matrix in GARCH? I want to know how to calculate the standard errors in them and without the empirical fisher information, empirical likelihood, I have no idea ...
user avatar
1 vote
1 answer
170 views

Demonstration and Interpretation between a Fisher matrix and its dual space which is covariance matrix

I have a simple (maybe not) issue about the interpretation of the link between Fisher information matrix and its inverse which is the covariance matrix. How to formulate that a line of Covariance ...
user avatar
0 votes
0 answers
128 views

Fisher matrix for a discrete distribution

Let $\mathbf{X} = \{X_1, \ldots, X_n\}$ be a sample of i.i.d. variables following a discrete distribution with parameters $\mathbf{p}^T = (p_1, p_2, p_3)$. How can I find the Fisher information matrix ...
GingerBadger's user avatar
6 votes
1 answer
224 views

Expected Fisher information isn't positive definite for truncated normal with heteroskedasticity

This question is about having a non-positive-definite expected Fisher information in a normal model in which observations have different dispersions. Consider this simple normal model: $$Y_i \sim N(\...
half-pass's user avatar
  • 3,581
0 votes
0 answers
48 views

How to calculate a multivariable fisher information matrix

I do not understand what is the definition for $$\mathcal I(\theta) = -E[H(\theta)]$$ where $H$ is the hessian of log-likelihood function how should I calculate this if $\theta$ is a vector. What ...
ddttdd's user avatar
  • 1

1
2 3 4 5
7