Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [fisher-information]

The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.

2
votes
1answer
34 views

How do we deduce this fisher information relation?

Given a RS $X_{1},X_{2},\ldots,X_{n}$ whose distribution is well known (unless its parameters), how do we prove the following Fischer Information relationship \begin{align*} I_{F}(\theta) =\textbf{E}\...
0
votes
0answers
18 views

The role of Fisher information matrix in Fisher kernel

I read the original paper proposed the Fisher kernel. The Fisher kernel is defined as $K(X_i,X_j) \propto U_{X_i}I^{-1}U_{X_j}$, where $U_X$ is the Fisher schore and $I$ is the Fisher information ...
1
vote
1answer
63 views

If $X\sim\text{Beta}(\theta,1)$, obtain the confidence interval of $100(1-\alpha)\%$ based on the asymptotic distribution of the score function

Let $X_{1},X_{2},\ldots,X_{n}$ be a random sample whose distribution is given by $\text{Beta}(\theta,1)$. Obtain the approximate confidence interval of $100(1-\alpha)\%$ based on the asymptotic ...
1
vote
1answer
62 views

Given $X\sim\mathcal{N}(0,\sigma^{2})$, obtain the Fischer information of $\sigma$ and $\sigma^{2}$

Suppose the random variable $X\sim\mathcal{N}(0,\sigma^{2})$, where we do not know the value of the standard deviation $\sigma$. Then obtain the Fisher information $I_{F}(\sigma)$ through $X$. Suppose ...
1
vote
0answers
30 views

Fisher Information for a Gaussian Process

Suppose I fit a Gaussian process to data such that the posterior distribution over any output is also a Gaussian process, $\mathcal{G}\mathcal{P}(\mu(x),\sigma^2(x))$ where $x$ is some valid input. ...
0
votes
0answers
19 views

Fisher information in GLM where $Y_1, Y_2,… ,Y_n$ are independent but not identically distributed

When making inference about regression coefficients, our estimate of asymptotic covariance matrix of $\mathbf{\hat {\beta}}$ is $$\hat{cov}(\hat{\beta}) = I^{-1}(\hat{\beta})$$ where $I$ is Fisher ...
0
votes
0answers
7 views

Weighted D optimality

I am working with D optimality for nonlinear models. My model has 8 parameters and some are more important than others. Is there a way to give weights to each of the parameters? I am thinking of ...
0
votes
1answer
44 views

Computing Empirical Fisher Information matrix for natural gradient

I would like to implement the natural gradient for reinforcement learning as described in the following paper: https://arxiv.org/pdf/1703.02660.pdf However, I do not know how to compute the empirical ...
1
vote
0answers
39 views

Observed information matrix with multivariate normal distribution

$$ \DeclareMathOperator\tr{tr} \DeclareMathOperator\vecOP{vec} \newcommand\di{\mathrm{d}} \newcommand\D{\mathrm{D}} \newcommand\Hess{\mathrm{H}} $$ I do not have much experience with matrix ...
0
votes
0answers
7 views

Finding Fisher information [duplicate]

Let $X$ distribution belongs for the family $\mathcal{P}\{P_{\theta}, \theta \in \Theta \}$. We need to find Fisher information $I(\theta)$ according $n$ simple sample, when $P_{\theta}$ is $N(\mu,\...
0
votes
1answer
57 views

Cramér-Rao Lower Bound & Fisher information - error in textbook?

I'm currently reading the textbook "Statistics for Mathematician" from Victor Panaretos. On page 65, the author presents the following equation for the Cramér-Rao Lower Bound (Note: I set the ...
1
vote
1answer
109 views

How to compute variance/confidence intervals from Fisher information matrix. Mistake in this document?

https://www.stat.umn.edu/geyer/5931/mle/mle.pdf In this document (by the great Geyer nonetheless!) it calculates confidence intervals using the Fisher matrix: But the standard deviation is not the ...
0
votes
0answers
15 views

Fisher information under different noise models

Directly lifted from Wikipedia: Fisher information (sometimes simply called information) is a way of measuring the amount of information that an observable random variable $X$ carries about an unknown ...
2
votes
0answers
67 views

Singular Fisher Information and nuisance parameters

I am interested in a problem where the Fisher Information matrix is singular and I want to treat some of the parameters as nuisance parameters. If the FI matrix is singular, that means that some ...
0
votes
0answers
6 views

Statistical meaning of $\mathbb E_P[score_\theta(Z)]^T\operatorname{Cov}_P[score_\theta(Z)]^{-1}\mathbb E_P[score_\theta(Z)]$

Consider a random vector $Z$ with distribution $P$ having mean $\mu$ and covarance matrix $\Sigma$. Question Statistically what is the meaning of the quadratic quantity $\mu^T\Sigma^{-1}\mu$ ? More ...
1
vote
1answer
128 views

Confidence interval for the 95th percentile of the normal distribution

Let $X_1, .., X_n \sim Normal(\mu, \sigma^2)$. Let $\tau$ be the 95th percentile of this distribution. Thus, $P(X_i < \tau) = 0.95$. What is the $1 - \alpha$ confidence interval for $\tau$? I ...
0
votes
0answers
17 views

Can negative of empirical second derivative of the log likelihood with respect to the parameters not be semi-positive definite?

This is the empirical Fischer Information. Also consider the outer product with itself of the first derivative of the log likelihood with respect to the parameters. This will always be semi-negative ...
1
vote
0answers
19 views

Alternatives to calculating the rank of the information matrix in determining if the model is identifiable

I have a known non-linear model $h \in \mathbf{R}^n$: $$ y = h(\theta) + \epsilon, $$ where $\theta\in \mathbf{R}^m$ is a parameter vector, and $\epsilon$ is a normal random variable with zero mean ...
1
vote
1answer
89 views

Different version of Wald test statistic formula

I came across two formulas for the Wald test statistic in a maximum likelihood framework: One has $(R\hat{\theta}-r)'(RI_n^{-1}R')^{-1}(R\hat{\theta}-r)$, where $I_n^{-1}$ is the inverse of the ...
3
votes
1answer
115 views

Statistics : Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ) ?

Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ) ? Could someone provide an intuitive explanation? I am having trouble understanding the concept.
1
vote
0answers
22 views

Inference for Maximum Likelihood Estimator Using Particle Filter

How does one compute standard errors for the MLE when using a particle filter approximation to the likelihood? I know that the estimator is asymptotically normal and that the variance-covariance ...
1
vote
1answer
128 views

understanding natural policy gradient

I'm reading this paper on Natural Policy Gradient https://papers.nips.cc/paper/2073-a-natural-policy-gradient.pdf and have some questions regarding how it works. I'm coming at this from an ML ...
0
votes
0answers
10 views

Fisher information matrix of the mean of a circularly symmetric complex Gaussian distribution

Does the FIM always exist for the mean vector of a complex Gaussian distribution? The log-likelihood function of a circularly symmetric complex Gaussian distribution for a $K\times1$ vector of ...
2
votes
1answer
240 views

Fisher information for MLE with constraint

Supposing I have a probability distribution $f(x|\vec\theta)$, where $x$ is a random variable and $\vec\theta$ is a vector of distribution parameters. I also know that parameters $\vec\theta$ should ...
1
vote
1answer
83 views

Why does Fisher use covariance when only variance is needed?

With reference to the following image from here: (can not inline it due to unsupported format) https://wikimedia.org/api/rest_v1/media/math/render/svg/9af8aa035642689bb2004047416b069a15406447 If we ...
3
votes
0answers
60 views

Fisher information for models that are not one-to-one

When a parameterized data model and corresponding pdf are known, the Cramér-Rao lower bound provides an lower bound for the variance of an estimator of one of the parameters. That is, given the data ...
1
vote
0answers
102 views

Observed Fisher Information & Cramer-Rao bound

The Cramer-Rao bound is usually derived for the "expected" or "total" fisher-information. Is a similar result possible for the observed-fisher information?
1
vote
1answer
53 views

Is there a Fisher Information equivalent in MAP Empirical Bayes estimation?

Background The Fisher information for a linear Gaussian model is $\mathcal{I}_{\theta} = \frac{X X^T}{\sigma^2} $. This is used in optimal experiment design techniques, for example, maximisation of $|...
5
votes
1answer
175 views

Does a quadratic log-likehood mean the MLE is (approximately) normally distributed?

So, in the usual case, one can prove from the asymptotic normality of a maximum likelihood estimator that the corresponding log-likehood surface is quadratic near the MLE (e.g. in the proof of the ...
3
votes
1answer
62 views

Is this reasoning about a chi squared distribution correct?

Consider for given mean $m\in \mathbb R$ the Gaussian product model $(\mathbb R^n, \mathcal{B}(\mathbb R^n),\mathcal{N}_{m,\theta}:\vartheta >0)$ where $\vartheta$ denotes the variance, and a ...
2
votes
1answer
61 views

Correct computation of Fisher Information

Let $\mathbf{X} = (X_1, \ldots, X_n)$ be an i.i.d sample from the parametric family of distributions $\mathcal{P} = \{P_\theta: \theta \in \Theta \}$ ($X_i \sim P_{\theta_0}$ are i.i.d. random ...
2
votes
1answer
219 views

Variance of estimator seemingly lower than CRLB?

While practicing for a mid-term, I came across a question where I was asked to investigate the variance of $\frac{(n+1)Y_{n}}{n}$ where $Y_{n}$ is the largest observation of a random sample of size $n$...
0
votes
1answer
26 views

Finding Fisher Matrix for Line Fitting

I am going through the "Fitting a Line" example from here. $f_1 = ax_1 + b$ and $f_2 = ax_2 + b$ are the models used to observed two data points in $R^2$. If $\sigma_1$ and $\sigma_2$ is the ...
1
vote
0answers
33 views

Fisher information of independent observations with transformation of parameters

Let $\boldsymbol\theta$ be a vector parameter, which can be estimated from two different time observation models (stochastic processes) $\mathbf{y}_1(t)$ and $\mathbf{y}_2(t)$. The Fisher information ...
4
votes
1answer
392 views

Fisher's Information for Laplace distribution

Say we have $f(x , \theta) = \frac{1}{2}e^{-|x-\theta|}$ Lets assume for simplicity, we only have 1 sample. We find that the log-likelihood for this distribution is: $$ l(\theta , x) = -log(2) + (\...
0
votes
1answer
130 views

Cramér–Rao bound to multiple parameters

I was reading Cramér–Rao bound to multiple parameters from Wikipedia page, but I could not follow this line in the article: Let $\displaystyle {\boldsymbol {T}}(X)$ be an estimator of any ...
2
votes
0answers
134 views

Are natural gradients ever computed in practice?

The method of natural gradient adaption has been proposed as an improvement on gradient descent (e.g. Amari, "Natural Gradient Works Efficiently in Learning", 1998.) In gradient descent the usual ...
4
votes
1answer
117 views

Does marginalization of some of the latent variables improve convergence in EM?

Given a likelihood to maximize $$ \log p(x | \theta) $$ Imagine that, in order to apply EM, we can augment the model with one or two latent variables. In that case, we can derive two lower bounds: $...
2
votes
1answer
222 views

How to compute confidence intervals for maximum likelihood from the Fisher information matrix?

Looking at the Fisher information matrix for a simple linear model, such as here, I do not understand how to use the matrix to compute confidence intervals. There are multiple examples on the internet ...
4
votes
0answers
121 views

Meaning of the Bayesian Cramer-Rao bound for a coin flip

Consider a coin $X\sim\operatorname{Bernoulli}(p)$. Its Fisher information is given by $J=\frac{1}{p(1-p)}$. Now suppose we are in a Bayesian setting and our prior on $p$ is $\pi=\operatorname{Beta}(...
2
votes
0answers
49 views

Fisher information for marginal distribution

Suppose I have a random variable $X$ with the following distribution: $$f(x|\theta)=\sum_{k=0}^{\infty}{p(k|\theta) \cdot \varphi_k(x)},$$ where $p(k|\theta)$ is some discrete probability ...
5
votes
1answer
1k views

Why is the Fisher information the inverse of the (asymptotic) covariance, and vice versa?

For the multinomial distribution, I had spent a lot of time and effort calculating the inverse of the Fisher information (for a single trial) using things like the Sherman-Morrison formula. But ...
0
votes
0answers
143 views

Information matrix for exponential distribution (using covariates)

(Reposting after changing some things and the title: standard error for glm: R and theory arent agreeing) I was trying to work out by hand the standard errors for an exponential glm (gamma distn with ...
2
votes
1answer
150 views

Calculation in the lower bound Cramer-Rao inequality

I read from http://math.arizona.edu/~jwatkins/N_unbiased.pdf We know the Fisher information is $$I(\theta)=E\bigg[\bigg(\frac{\partial \log f(X) }{\partial \theta}\bigg)^2\bigg]. $$ Cramer-Rao lower ...
1
vote
0answers
192 views

Estimators of the Fisher information, advantages, disadvantages?

I am reading a paper, which defines two estimators of the Fisher information. The first is the well-known negative hessian of the log-likelihood at the ML estimate, the second is $$ \sum_{t=1}^T \frac{...
2
votes
0answers
19 views

Is probability distribution known in parametric inference?

I have two questions about parametric inference. Assume we estimate parameter $\theta$ from a set of measurements. Is probability distribution $p(x|\theta)$ known from the begining? In other words, ...
1
vote
0answers
26 views

Fisher Information in LAD model for ML estimator

Considering an i.i.d. sample from a linear model $y_i=\alpha x_i+u_i$ (both $y$ and $x$ are centered with respect to their means) errors are homoscedastic and are distributed as: $$u\sim\frac{1}{\sqrt{...
3
votes
0answers
42 views

(When not assuming differentiability) what is the definition of Fisher information?

Here we assume (for simplicity) that the parameter $\theta$ is one-dimensional. When one has sufficient regularity and can push in partial derivatives to and pull out partial derivatives from the ...
1
vote
1answer
130 views

Standard error of the estimate in logistic regression

We usually get an estimate of $\beta$ in the logistic regression by finding the $MLE$ of the observed random samples of $X_1,X_2....,X_N$. Then we use Wald's test i.e. ${[\hat \beta / S.E.(\hat \beta)]...
5
votes
0answers
86 views

Maximum likelihood and degenerate Fisher information

I am wondering if there are some standard results to find rates of convergence of the MLE for different sub-parameters, when the Fisher information is degenerate. More precisely, suppose that I ...