Questions tagged [fisher-information]

The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.

Filter by
Sorted by
Tagged with
0 votes
0 answers
16 views

Demonstration and Interpretation between a Fisher matrix and its dual space which is covariance matrix

I have a simple (maybe not) issue about the interpretation of the link between Fisher information matrix and its inverse which is the covariance matrix. How to formulate that a line of Covariance ...
user avatar
  • 101
0 votes
0 answers
9 views

Find fisher information matrix for optimization estimator

I have that $$f(x)=\frac{1}{\sqrt{2 \pi}}e^{-\frac{1}{2}x^2}$$ I have the conditional distribution: $f_{\beta}(y|x)=\frac{1}{\sqrt{2 \pi}}e^{-\frac{1}{2}(y-\beta_0-\beta_1x-\beta_2x^2)^2}$ and we have ...
user avatar
  • 253
0 votes
0 answers
35 views

Fisher matrix for a discrete distribution

Let $\mathbf{X} = \{X_1, \ldots, X_n\}$ be a sample of i.i.d. variables following a discrete distribution with parameters $\mathbf{p}^T = (p_1, p_2, p_3)$. How can I find the Fisher information matrix ...
user avatar
6 votes
1 answer
135 views

Expected Fisher information isn't positive definite for truncated normal with heteroskedasticity

This question is about having a non-positive-definite expected Fisher information in a normal model in which observations have different dispersions. Consider this simple normal model: $$Y_i \sim N(\...
user avatar
  • 3,491
0 votes
0 answers
18 views

How to calculate a multivariable fisher information matrix

I do not understand what is the definition for $$\mathcal I(\theta) = -E[H(\theta)]$$ where $H$ is the hessian of log-likelihood function how should I calculate this if $\theta$ is a vector. What ...
user avatar
  • 1
0 votes
0 answers
20 views

Fisher Information of Weight in Mixture distribution

Let's assume $x$ follows a mixture of two arbitrary continuous probability distributions with probability density functions $p_1(x)$ and $p_2(x)$, respectively. The probability density function of $x$ ...
user avatar
  • 762
1 vote
1 answer
37 views

For the family, $f_\theta(x)=\frac{e^{-(x-\theta)}}{1+e^{-(x-\theta)})^2}$, compute the fisher information, is it an exponential family?

For the family, $f_\theta(x)=\frac{e^{-(x-\theta)}}{1+e^{-(x-\theta)})^2}$, compute the fisher information, is it an exponential family, $x\in \mathbb{R},\theta \in \mathbb{R}$? I computed the fisher ...
user avatar
1 vote
0 answers
35 views

Are there any theoretical guarantees about the log-likelihood's inverse Hessian when the observations are not i.i.d.?

Let $X=[X_1...X_n]$ be some random variables in which $X_i$ are not independent. For example, you may envision the observations came from some stochastic process. Let $\ell(\theta; X)$ be the log-...
user avatar
0 votes
0 answers
17 views

Fisher Information for a Cauchy-distributed Variable (MLE, Variance) [duplicate]

I think, I'm close but I'm still having issues with the following problem: I'm looking at a Cauchy distributed random variable, with: $$ f(x,\gamma,\theta)=\frac{1}{\gamma\pi}\frac{1}{1+(\frac{x-\...
user avatar
0 votes
0 answers
27 views

Connection between the Fisher information matrix and the Gaussian-weighted structure tensor

In image processing, if we call $I(x):\mathbb{Z}^2\mapsto \mathbb{R}$ to the function that gives the brightness value at a discrete image location $x=(u,v)^\top\in\mathbb{Z}^2$, then the structure ...
user avatar
  • 1,068
0 votes
1 answer
70 views

Does Fisher scoring exist as such?

I'm studying second order optimization methods in statistics, and I've run into a conceptual barrier that I was hoping someone can help me with. First for some notation: consider a sample $X_1,\dots,...
user avatar
0 votes
0 answers
31 views

Fisher Information and MLE

I know that the Fisher Information is defined as the variance of the score function: $$ I(\theta)=Var(\frac{d}{d\theta}\mathrm{log}L(x|\theta))=\int(\frac{d}{d\theta}\mathrm{log}f(x|\theta))^2p_\theta(...
user avatar
0 votes
0 answers
15 views

Analogous information matrix and divergence for the Bhattacharyya bound

In the case of Cramér-Rao lower bound (CRLB), the Fisher information matrix (FIM) is obtained from the K-L divergence (KLD), i.e. $D(p_\theta\|p_\theta') = \int p_\theta(x)\log\frac{p_\theta(x)}{p_{\...
user avatar
  • 153
1 vote
1 answer
115 views

Observed Fisher Information and confidence intervals

I'm trying to put confidence intervals on parameters fitted through MLE through the inversion of the observed Fisher information matrix. More specifically, I define the observed FIM as: $$ J_{n}(\hat{...
user avatar
0 votes
0 answers
23 views

Why do we divide by n when solving for the Cramer-Rao Lower Bound here?

"Let $X_1,...,X_n$ be iid Bernoulli(1,$p$), with $p$ unknown. Find the CRLB for the variances of unbiased estimators of $p$." With pdf $p^x(1-p)^{1-x}$, the derivative of the log function is ...
user avatar
  • 1
0 votes
0 answers
27 views

Fisher Information Matrix for generalized skew-t regression

How do I compute the entries of a Fisher information matrix for a regression model with generalized skew-$t$ (location $0$) errors? Consider the regression model $$\vec{y}=X\beta+\vec{\varepsilon},$$ ...
user avatar
1 vote
0 answers
20 views

Can the Fisher Information be Used to Evaluate the Quality of a Regression Model?

Can the Fisher Information be Used to Evaluate the Quality of a Regression Model? I have often heard of the Fisher Information described as "how much information about an unknown parameter we can ...
user avatar
  • 5,698
1 vote
0 answers
26 views

approximate fisher information for intractable likelihoods

Suppose I have a data set $X_1, \ldots, X_n$, and from that I compute a statistic $T(X_1, \ldots, X_n) := T$. I want to assess how reactive/sensitive this calculation is to changes in parameter values....
user avatar
  • 18.6k
0 votes
1 answer
31 views

Show that solving this equation is the same as m'th Fisher Scoring step

In my exercise we assume that $Y_i|X_i$ has distribution with density $f_i(y_i,\eta_i) $ for $i=1,...,n$ where $\eta_i=X_i^T$ is the linear predictor. The generalized linear model with an exponential ...
user avatar
  • 253
0 votes
0 answers
48 views

Score statistic and Fisher information

In my exercise we assume that $Y_i|X_i$ has distribution with density $f_i(y_i,\eta_i) $ for $i=1,...,n$ where $\eta_i=X_i^T$ is the linear predictor. The generalized linear model with an exponential ...
user avatar
  • 253
0 votes
0 answers
25 views

Evaluating the asymptotic distribution of a metric that's a function of both ML estimated parameters and data, generalizing the delta method

I have a particular problem with likelihood function $\mathcal{L}(\theta \mid X)$, in which I interested in the distribution of a metric $m(X) = f(X,\hat\theta)$ (even asymptotically, though knowing ...
user avatar
  • 3,779
0 votes
0 answers
62 views

Fisher Information Matrix for Non Canonical link function for Poisson regression

I'm trying to solve this problem and I don't know how to do it since is not using the canonical link function. When I was researching online I couldn't find an example when the canonical function is ...
user avatar
  • 45
0 votes
0 answers
34 views

Implications of expectation of score function being zero

I understand why mathematically, the expectation of the score function (evaluated at the true parameter) is zero. But what are the implications of this fact? Can you give one or more examples on where ...
user avatar
  • 186
0 votes
1 answer
43 views

What is the definition and upper bound on the variable "m" in the definition of the multivariate normal Fisher Information?

Multivariate normal distribution [edit] The FIM for a $N$-variate multivariate normal distribution, $X \sim N(\mu(\theta), \Sigma(\theta))$ has a special form. Let the $K$-dimensional vector of ...
user avatar
  • 224
0 votes
1 answer
120 views

Monte Carlo approximation to find expected value of gradient square

I need to to calculate this term: $ \mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right] $ Where $ S(Y,L,\theta) =\frac{\partial}{\partial\theta} l(Y,L,\theta) $ With $\theta$ = maximum ...
user avatar
2 votes
1 answer
120 views

Fisher Information matrix as covariance matrix from scores

I've found in a paper here. Observed Fisher information estimated in a way that does not convince me at all. They estimated it as the covariance matrix of the scores, but to me the formula used is ...
user avatar
1 vote
0 answers
38 views

Confusion about the Hessian approximation

There are two popular forms of the neural network Hessian approximation in the literature: $$ H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right) \left(\frac{\partial y}{\partial w_i}\right)^...
user avatar
1 vote
1 answer
141 views

How can nuisance parameters in Fisher matrix can deteriorate the useful constraints?

I have a Fisher matrix $F$ which has the matrix blocks form like this : $$ F=\begin{bmatrix} A & B\\ C & D \end{bmatrix} $$ The block $A$ is the most important block, in the sense the ...
user avatar
2 votes
1 answer
38 views

Which is correct $Var(\theta) \ge (F^{-1})_{ii} $ or $Var(\theta) \ge 1/(F_{ii}) $?

With $\textbf{F}(\theta)$ denoting the Fisher matrix, $\textbf{V}(\theta)$ the variance matrix, $\textbf{C}(\theta)$ the covariance matrix, and parameter vector $\theta = [\theta_1, \theta_2, \dots,...
user avatar
  • 121
2 votes
1 answer
46 views

Fisher information as the variance of the 1st derivative of the log-lh different from the expectation of the 2nd derivative of the log-lh

I have the following pdf: $f(x)=\theta \times x^{\theta-1} \mathbf1(0\le x\le 1)$ where $\mathbf 1$ is the indicator function. I am trying to calculate the Fisher information using the expectation of ...
user avatar
  • 23
11 votes
3 answers
3k views

Connection between Fisher information and variance of score function

The fisher information's connection with the negative expected hessian at $\theta_{MLE}$, provides insight in the following way: at the MLE, high curvature implies that an estimate of $\theta$ even ...
user avatar
0 votes
0 answers
38 views

discontinuity of Fisher information / confidence interval for Fisher information

From the question given at Fisher Information for frequency estimation under non-circular complex Gaussian noise when I consider bivariate normal distribution for $p(z|\omega,\phi)$, then I have found ...
user avatar
  • 111
1 vote
0 answers
34 views

Does the Fisher information matrix have to be convex when assessing an optimal criteria?

I've reached a local minimum for a proposed model using a set of experimental data (positive definite Hessian) and want to select an additional experiment that will reduce the parameter uncertainties (...
user avatar
3 votes
1 answer
65 views

Fisher information as negative log likelihood

The Fisher Information is defined as the covariance matrix, or $E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))^T]$. It can also be defined as $E_{y \sim P(y;\...
user avatar
1 vote
0 answers
67 views

Fisher information matrix for model based design of experiments

I'm trying to use the Fisher information matrix and D-optimal design for model-based experiment selection, but I'm not sure if I'm implementing or interpreting it correctly. I begin with the values of ...
user avatar
0 votes
0 answers
11 views

What can make fisher information matrix diagonal? [duplicate]

How can we intuitively interprete the diagonal Fisher information matrix? Can we say these parameters are independent?
user avatar
  • 205
4 votes
0 answers
137 views

Fisher Information for frequency estimation under non-circular complex Gaussian noise

Consider a non-circular complex Gaussian noise $v$, which is given as $v = v_r + iv_i$. Here, real and imaginary components are independent and are Normal random variables such as $v_r \sim \mathcal{N}...
user avatar
  • 111
8 votes
1 answer
436 views

Are all log-likelihood functions twice differentiable?

For maximum likelihood estimation we need to set the first derivative of the log-likelihood function equal to $\mathbf{0}$. The negative expected value of the Hessian matrix (second derivative) is ...
user avatar
  • 762
0 votes
0 answers
22 views

Should the expected observed fisher information matrix be positive definite in Fisher Scoring algorithm? [duplicate]

I want to use the Fisher Scoring Algorithm in Newton-Raphson method. I(theta_m) in above that calculated at each iteration, has to be positive definite? Is it possible this matrix be indefinite?
user avatar
6 votes
1 answer
283 views

Is it possible Fisher information matrix be indefinite?

I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function. At each iteration, I want to check that is the Hessian matrix negative definite or not and I ...
user avatar
1 vote
0 answers
53 views

Is it possible that Fisher information matrix be indefinite? [duplicate]

I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function. At each iteration, I want to check that is the Hessian matrix negative definite or not and I ...
user avatar
1 vote
0 answers
31 views

Fisher Information Matrix of Matrix Variate F Distribution

Let $\mathbf{X}$ follow a matrix variate F distribution with pdf $$ \begin{align} f\left(\mathbf{X} | \mathbf{\Sigma}, n, \nu\right) = \frac{\Gamma_p(\frac{n + \nu }{2})}{\Gamma_p(\frac{n}{2})\...
user avatar
  • 762
0 votes
0 answers
28 views

Is it possible to estimate the Hessian as the covariance of primal and cotangent?

Let's say we have a function $$f: \mathbb R^n \to \mathbb R.$$ Can we numerically approximate the Hessian $f''(x)$ as $$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$ where $$E(a) = x?$$
user avatar
  • 13.7k
1 vote
0 answers
38 views

Finding the score function

Suppose we have conditional distributional model: $$y|x\sim \mathcal{D}(g(x,\theta_1),h(x,\theta_2),\theta_3)$$ where $\mathcal{D}$ is known distribution, $(g(x,\theta_1)$ is conditional mean with ...
user avatar
3 votes
0 answers
189 views

Information Matrix for Conditional Likelihood

I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
user avatar
  • 1,103
0 votes
0 answers
72 views

Fisher information of an Ornstein-Uhlenbeck process

I would like to compute the Fisher information of an Ornstein-Uhlenbeck process $X_t = Y_t - \beta Z_t$ where $Y_t$ and $Z_t$ are two time-series. My log-likelihood function in this case is: $$\...
user avatar
  • 299
0 votes
0 answers
33 views

Covariance Matrix and it's dependence on the dataset size

I am new in the analysis of covariance matrix and plotting the corresponding ellipses. Here are my confusions in summary : If I have a dataset of size: N rows. I compute the covariance matrix, I get ...
user avatar
2 votes
1 answer
148 views

Cramér–Rao Lower Bound and UMVUE for $\frac1{\theta}$

Problem: Find the UMVUE of $\frac1\theta$ for a random sample from the population distribution with density $$f(x;\theta)=\theta x^{\theta-1}$$ and show that its variance reaches the Cramér–Rao lower ...
user avatar
4 votes
1 answer
206 views

Fisher information of $\rho$ in a symmetric normal $N_p(\mathbf 0,\Sigma)$ distribution

Suppose $\boldsymbol X=(X_1,\ldots,X_p)'\sim N_p(\mathbf0,\Sigma)$ where $\Sigma=(1-\rho)I_p+\rho\mathbf1\mathbf1'$ is positive definite. The objective is to obtain the asymptotic variance of the MLE ...
user avatar
  • 9,120
0 votes
0 answers
69 views

Does the score function have expectation of zero, or does the score function have expectation of zero when evaluated at the true parameter?

I am slightly confused about two versions of the idea that the score function has expected value of zero. I learned that the score function is essentially the function of the slope of log likelihood. ...
user avatar
  • 131

1
2 3 4 5 6