Questions tagged [fisher-information]

The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Fisher Information with respect to the Standard deviation of Normal distribution

Let $X\sim\mathcal{N}(0,\sigma^2)$ be given. I computed the Fisher Information to be $I(\sigma)=\frac{2}{\sigma^2}$. Note that the Fisher Information for the variance is given by $I(\sigma^2)=\frac{1}{...
3
votes
0answers
45 views

Is it true that Fisher information for a statistic and the sample are equal if and only if the statistic is sufficient?

According to https://en.wikipedia.org/wiki/Fisher_information#Sufficient_statistic we have if and only if, but according to https://projecteuclid.org/download/pdfview_1/euclid.imsc/1362751193 we don'...
1
vote
0answers
20 views

How can I find CRLB for an integer valued Maximum Likelihood Estimator?

Reference: How to find Maximum Likelihood estimates of an *integer* parameter? Extending the above question, how to find the Fisher information and hence CRLB for the integer-valued Estimator using ...
0
votes
0answers
11 views

Fisher Information Matrix for regression problem

In DeepMind's paper [Overcoming catastrophic forgetting in neural networks]( https://arxiv.org/abs/1612.00796), elastic weight consolidation with Fisher Information Matrix is used to tackle the ...
0
votes
0answers
5 views

Fisher Scoring for Constrained optimization

Suppose I am estimating a parameter vector using the usual Fisher Scoring updates; \begin{equation} \theta_{s+1} = \theta_{s} + \lambda\mathcal{I}(\theta_{s})^{-1}\frac{dl}{d\theta}\bigg|_{\theta=\...
2
votes
0answers
14 views

Signal-to-noise-ratio, Fisher information and and “estimability”

Given a parametric statistical model, is it common to study the quantity $$ Q_{\theta} = \theta^2 I_{\theta} \, ,$$ where $I_{\theta}$ is the Fisher information? (I focus on a single parameter for ...
1
vote
1answer
28 views

Fisher Information for general one parameter exponential family (inconsistent with Poisson distribution)

For one of my hw questions, I was asked to derive Fisher Information for one parameter exponential family. Here's my approach: $$L(\theta) = f(x\mid\theta) = e^{\theta T(x) - \eta(\theta)}h(x)$$ $$\...
0
votes
0answers
16 views

Sampling distribution of the Score statistics of a GLM MODEL

In the context of a GLM (with a distribution that belongs to the exponential family), we often compute the score statistics $$ U = \frac{\partial LogLike(\boldsymbol{\beta};\mathbf{y})}{\partial\...
0
votes
0answers
35 views

Fisher Information for Gamma

Question: Find the fisher's information for $\mathcal{G}(\alpha, \beta)$ ,$\beta$ known. Attempt: Since $\mathcal{G}(\alpha, \beta)$ ,$\beta$ known. $$f(x|\alpha) =\dfrac{x^{\alpha-1}e^{x/\beta}}...
0
votes
0answers
25 views

Fisher information for double exponential distribution [duplicate]

I am asked to find the Fisher information function for double exponential distribution with mean $\mu$ and variance 1. Here's my approach: $l(\mu) = -\frac{1}{2}e^{|x-\mu|}$, so $\log l(\mu) = ...
1
vote
0answers
34 views

Compute the information matries related to normal distribution

This is a problem that I have trouble with. Suppose that we have $X_{1}, \ldots, X_{m}$ are iid $N\left(\mu, \sigma^{2}\right), Y_{1}, \ldots, Y_{n}$ are iid $N\left(0, \sigma^{2}\right),$ the $X$...
0
votes
0answers
39 views

Why do we use $S^2$ while estimating the variance?

Sorry the title is a bit silly, but I currently confront a problem related to Fisher's information. Let $X_1, X_2, \cdots, X_n$ be of $N(\mu , \sigma^2 )$ distribution where $\mu$ is known, $U^2 := ...
0
votes
1answer
169 views

Fisher information from MLE in R?

Reworded the question: I have read "The Fisher information I(p) is this negative second derivative of the log-likelihood function, averaged over all possible X = {h, N–h}, when we assume some value ...
0
votes
0answers
45 views

Inverse Covariance Matrix of a Gaussian Distribution: Relationship of Precision Matrix and Information Matrix

In the book "Probabilistic Robotics" (Thrun et al.), chapter 3.5.1 states that The canonical parameterization of a multivariate Gaussian is given by a matrix $\Omega$ and a vector $\xi$. The ...
0
votes
1answer
74 views

Calculate the uncertainty of a MLE

I have minimized the negative LL of a Poisson distribution to get an MLE of three parameters using scipy.minimize w/ Nelder-Mead. I want to calculate the uncertainty of the MLE. From reading, I ...
0
votes
0answers
14 views

Does C-efficiency exist? If so, how is it defined?

When using an algorithm to construct an optimal experimental design one has a measurement of optimality to find an optimal design with this optimality-criterion. For any arbitrary chosen design a D- ...
0
votes
0answers
37 views

Asymptotic Mean Squared Error of Maximum Likelihood estimator

I want to show that $n$ times mean squared error for the maximum likelihood estimator converges to the inverse of Fisher information, where $n$ is the number of samples. But The standard proofs of ...
1
vote
1answer
24 views

Statistical comparison of (covariance) matrices

I am trying to test whether the covariance matrix for the maximum likelihood estimates for a gaussian general lienear model approaches the inverse Fisher information matrix (times 1/n , n being the ...
0
votes
0answers
15 views

Does Fisher Information quantify precision? [duplicate]

Looking at perspective from estimating the actual value from a set of data measured by the instrument. Does Fisher information just quantify the precision of the measurement? What does it say about ...
0
votes
0answers
28 views

What statistic to use in testing the variance of maximum likelihood estimators

(A physicist self-studying statistics here) I was previously confused about the meaning of the standard error in a maximum likelihood estimate. Certain stack exchange posts (linked below) have gone ...
0
votes
0answers
14 views

Efficient GPU (batch) implementation of Empirical Fisher information matrix?

I have seen many implementations. It seems to be a limitation of autograd itself that we can compute the gradient of loglikelihood only one sample at a time. The batch version has been used but in a ...
0
votes
0answers
12 views

Action of Fisher information in “Latent Variable Model Selection…”

I'm having trouble understanding the role of the Fisher information matrix in the assumptions of Chandrasekaran et al. 2012. In the paper, the authors define the Fisher information matrix (i.e., ...
0
votes
1answer
31 views

Sufficient Statistic and Unbiased Estimate in Exponential Family

I am reading this classic paper (Information and the Accuracy Attainable in the Estimation of Statistical Parameters) by CR Rao where he deals with sufficient statistics in exponential distributions ...
2
votes
2answers
268 views

Score function of poisson distribution

I have a stupid question I haven't figured out. So when counting the score for poisson distribution I get the log likelihood $$S(\mu ) = \frac{\partial \ell(\lambda )}{\partial \lambda } = \sum_1^n \...
3
votes
0answers
33 views

Fisher information matrix and gradients

I'm a math Ph.D. without formal training in statistics. Quite a few papers on normalization methods in deep learning mention the Fisher information matrix and how it's related to the Riemannian metric ...
2
votes
1answer
39 views

How do we deduce this fisher information relation?

Given a RS $X_{1},X_{2},\ldots,X_{n}$ whose distribution is well known (unless its parameters), how do we prove the following Fischer Information relationship \begin{align*} I_{F}(\theta) =\textbf{E}\...
0
votes
0answers
39 views

The role of Fisher information matrix in Fisher kernel

I read the original paper proposed the Fisher kernel. The Fisher kernel is defined as $K(X_i,X_j) \propto U_{X_i}I^{-1}U_{X_j}$, where $U_X$ is the Fisher schore and $I$ is the Fisher information ...
1
vote
1answer
134 views

If $X\sim\text{Beta}(\theta,1)$, obtain the confidence interval of $100(1-\alpha)\%$ based on the asymptotic distribution of the score function

Let $X_{1},X_{2},\ldots,X_{n}$ be a random sample whose distribution is given by $\text{Beta}(\theta,1)$. Obtain the approximate confidence interval of $100(1-\alpha)\%$ based on the asymptotic ...
1
vote
1answer
70 views

Given $X\sim\mathcal{N}(0,\sigma^{2})$, obtain the Fischer information of $\sigma$ and $\sigma^{2}$

Suppose the random variable $X\sim\mathcal{N}(0,\sigma^{2})$, where we do not know the value of the standard deviation $\sigma$. Then obtain the Fisher information $I_{F}(\sigma)$ through $X$. Suppose ...
1
vote
0answers
53 views

Fisher Information for a Gaussian Process

Suppose I fit a Gaussian process to data such that the posterior distribution over any output is also a Gaussian process, $\mathcal{G}\mathcal{P}(\mu(x),\sigma^2(x))$ where $x$ is some valid input. ...
0
votes
0answers
24 views

Fisher information in GLM where $Y_1, Y_2,… ,Y_n$ are independent but not identically distributed

When making inference about regression coefficients, our estimate of asymptotic covariance matrix of $\mathbf{\hat {\beta}}$ is $$\hat{cov}(\hat{\beta}) = I^{-1}(\hat{\beta})$$ where $I$ is Fisher ...
0
votes
0answers
9 views

Weighted D optimality

I am working with D optimality for nonlinear models. My model has 8 parameters and some are more important than others. Is there a way to give weights to each of the parameters? I am thinking of ...
0
votes
1answer
169 views

Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
2
votes
1answer
117 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
0
votes
0answers
7 views

Finding Fisher information [duplicate]

Let $X$ distribution belongs for the family $\mathcal{P}\{P_{\theta}, \theta \in \Theta \}$. We need to find Fisher information $I(\theta)$ according $n$ simple sample, when $P_{\theta}$ is $N(\mu,\...
0
votes
1answer
75 views

Cramér-Rao Lower Bound & Fisher information - error in textbook?

I'm currently reading the textbook "Statistics for Mathematician" from Victor Panaretos. On page 65, the author presents the following equation for the Cramér-Rao Lower Bound (Note: I set the ...
1
vote
1answer
218 views

How to compute variance/confidence intervals from Fisher information matrix. Mistake in this document?

https://www.stat.umn.edu/geyer/5931/mle/mle.pdf In this document (by the great Geyer nonetheless!) it calculates confidence intervals using the Fisher matrix: But the standard deviation is not the ...
2
votes
0answers
123 views

Singular Fisher Information and nuisance parameters

I am interested in a problem where the Fisher Information matrix is singular and I want to treat some of the parameters as nuisance parameters. If the FI matrix is singular, that means that some ...
1
vote
1answer
201 views

Confidence interval for the 95th percentile of the normal distribution

Let $X_1, .., X_n \sim Normal(\mu, \sigma^2)$. Let $\tau$ be the 95th percentile of this distribution. Thus, $P(X_i < \tau) = 0.95$. What is the $1 - \alpha$ confidence interval for $\tau$? I ...
1
vote
0answers
22 views

Alternatives to calculating the rank of the information matrix in determining if the model is identifiable

I have a known non-linear model $h \in \mathbf{R}^n$: $$ y = h(\theta) + \epsilon, $$ where $\theta\in \mathbf{R}^m$ is a parameter vector, and $\epsilon$ is a normal random variable with zero mean ...
1
vote
1answer
147 views

Different version of Wald test statistic formula

I came across two formulas for the Wald test statistic in a maximum likelihood framework: One has $(R\hat{\theta}-r)'(RI_n^{-1}R')^{-1}(R\hat{\theta}-r)$, where $I_n^{-1}$ is the inverse of the ...
3
votes
1answer
159 views

Statistics : Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ) ?

Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ) ? Could someone provide an intuitive explanation? I am having trouble understanding the concept.
1
vote
0answers
28 views

Inference for Maximum Likelihood Estimator Using Particle Filter

How does one compute standard errors for the MLE when using a particle filter approximation to the likelihood? I know that the estimator is asymptotically normal and that the variance-covariance ...
1
vote
1answer
168 views

understanding natural policy gradient

I'm reading this paper on Natural Policy Gradient https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf and have some questions regarding how it works. I'm coming at this from an ML ...
4
votes
1answer
372 views

Fisher information for MLE with constraint

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should ...
1
vote
1answer
101 views

Why does Fisher use covariance when only variance is needed?

With reference to the following image from here: (can not inline it due to unsupported format) https://wikimedia.org/api/rest_v1/media/math/render/svg/9af8aa035642689bb2004047416b069a15406447 If we ...
3
votes
0answers
70 views

Fisher information for models that are not one-to-one

When a parameterized data model and corresponding pdf are known, the Cramér-Rao lower bound provides an lower bound for the variance of an estimator of one of the parameters. That is, given the data ...
1
vote
0answers
139 views

Observed Fisher Information & Cramer-Rao bound

The Cramer-Rao bound is usually derived for the "expected" or "total" fisher-information. Is a similar result possible for the observed-fisher information?
1
vote
1answer
88 views

Is there a Fisher Information equivalent in MAP Empirical Bayes estimation?

Background The Fisher information for a linear Gaussian model is $\mathcal{I}_{\theta} = \frac{X X^T}{\sigma^2} $. This is used in optimal experiment design techniques, for example, maximisation of $|...
5
votes
1answer
205 views

Does a quadratic log-likehood mean the MLE is (approximately) normally distributed?

So, in the usual case, one can prove from the asymptotic normality of a maximum likelihood estimator that the corresponding log-likehood surface is quadratic near the MLE (e.g. in the proof of the ...