Questions tagged [fisher-information]
The Fisher information measures the curvature of the log-likelihood and can be used to assess the efficiency of estimators.
290
questions
0
votes
0
answers
16
views
Demonstration and Interpretation between a Fisher matrix and its dual space which is covariance matrix
I have a simple (maybe not) issue about the interpretation of the link between Fisher information matrix and its inverse which is the covariance matrix.
How to formulate that a line of Covariance ...
- 101
0
votes
0
answers
9
views
Find fisher information matrix for optimization estimator
I have that
$$f(x)=\frac{1}{\sqrt{2 \pi}}e^{-\frac{1}{2}x^2}$$
I have the conditional distribution:
$f_{\beta}(y|x)=\frac{1}{\sqrt{2 \pi}}e^{-\frac{1}{2}(y-\beta_0-\beta_1x-\beta_2x^2)^2}$
and we have ...
- 253
0
votes
0
answers
35
views
Fisher matrix for a discrete distribution
Let $\mathbf{X} = \{X_1, \ldots, X_n\}$ be a sample of i.i.d. variables following a discrete distribution with parameters $\mathbf{p}^T = (p_1, p_2, p_3)$. How can I find the Fisher information matrix ...
- 405
6
votes
1
answer
135
views
Expected Fisher information isn't positive definite for truncated normal with heteroskedasticity
This question is about having a non-positive-definite expected Fisher information in a normal model in which observations have different dispersions.
Consider this simple normal model:
$$Y_i \sim N(\...
- 3,491
0
votes
0
answers
18
views
How to calculate a multivariable fisher information matrix
I do not understand what is the definition for
$$\mathcal I(\theta) = -E[H(\theta)]$$
where $H$ is the hessian of log-likelihood function
how should I calculate this if $\theta$ is a vector. What ...
- 1
0
votes
0
answers
20
views
Fisher Information of Weight in Mixture distribution
Let's assume $x$ follows a mixture of two arbitrary continuous probability distributions with probability density functions $p_1(x)$ and $p_2(x)$, respectively. The probability density function of $x$ ...
- 762
1
vote
1
answer
37
views
For the family, $f_\theta(x)=\frac{e^{-(x-\theta)}}{1+e^{-(x-\theta)})^2}$, compute the fisher information, is it an exponential family?
For the family, $f_\theta(x)=\frac{e^{-(x-\theta)}}{1+e^{-(x-\theta)})^2}$, compute the fisher information, is it an exponential family, $x\in \mathbb{R},\theta \in \mathbb{R}$?
I computed the fisher ...
- 181
1
vote
0
answers
35
views
Are there any theoretical guarantees about the log-likelihood's inverse Hessian when the observations are not i.i.d.?
Let $X=[X_1...X_n]$ be some random variables in which $X_i$ are not independent. For example, you may envision the observations came from some stochastic process. Let $\ell(\theta; X)$ be the log-...
0
votes
0
answers
17
views
Fisher Information for a Cauchy-distributed Variable (MLE, Variance) [duplicate]
I think, I'm close but I'm still having issues with the following problem:
I'm looking at a Cauchy distributed random variable, with:
$$
f(x,\gamma,\theta)=\frac{1}{\gamma\pi}\frac{1}{1+(\frac{x-\...
0
votes
0
answers
27
views
Connection between the Fisher information matrix and the Gaussian-weighted structure tensor
In image processing, if we call $I(x):\mathbb{Z}^2\mapsto \mathbb{R}$ to the function that gives the brightness value at a discrete image location $x=(u,v)^\top\in\mathbb{Z}^2$, then the structure ...
- 1,068
0
votes
1
answer
70
views
Does Fisher scoring exist as such?
I'm studying second order optimization methods in statistics, and I've run into a conceptual barrier that I was hoping someone can help me with.
First for some notation: consider a sample $X_1,\dots,...
- 400
0
votes
0
answers
31
views
Fisher Information and MLE
I know that the Fisher Information is defined as the variance of the score function:
$$
I(\theta)=Var(\frac{d}{d\theta}\mathrm{log}L(x|\theta))=\int(\frac{d}{d\theta}\mathrm{log}f(x|\theta))^2p_\theta(...
0
votes
0
answers
15
views
Analogous information matrix and divergence for the Bhattacharyya bound
In the case of Cramér-Rao lower bound (CRLB), the Fisher information matrix (FIM) is obtained from the K-L divergence (KLD), i.e. $D(p_\theta\|p_\theta') = \int p_\theta(x)\log\frac{p_\theta(x)}{p_{\...
- 153
1
vote
1
answer
115
views
Observed Fisher Information and confidence intervals
I'm trying to put confidence intervals on parameters fitted through MLE through the inversion of the observed Fisher information matrix. More specifically, I define the observed FIM as:
$$
J_{n}(\hat{...
0
votes
0
answers
23
views
Why do we divide by n when solving for the Cramer-Rao Lower Bound here?
"Let $X_1,...,X_n$ be iid Bernoulli(1,$p$), with $p$ unknown. Find the CRLB for the variances of unbiased estimators of $p$."
With pdf $p^x(1-p)^{1-x}$, the derivative of the log function is ...
- 1
0
votes
0
answers
27
views
Fisher Information Matrix for generalized skew-t regression
How do I compute the entries of a Fisher information matrix for a regression model with generalized skew-$t$ (location $0$) errors?
Consider the regression model $$\vec{y}=X\beta+\vec{\varepsilon},$$
...
1
vote
0
answers
20
views
Can the Fisher Information be Used to Evaluate the Quality of a Regression Model?
Can the Fisher Information be Used to Evaluate the Quality of a Regression Model?
I have often heard of the Fisher Information described as "how much information about an unknown parameter we can ...
- 5,698
1
vote
0
answers
26
views
approximate fisher information for intractable likelihoods
Suppose I have a data set $X_1, \ldots, X_n$, and from that I compute a statistic $T(X_1, \ldots, X_n) := T$. I want to assess how reactive/sensitive this calculation is to changes in parameter values....
- 18.6k
0
votes
1
answer
31
views
Show that solving this equation is the same as m'th Fisher Scoring step
In my exercise we assume that $Y_i|X_i$ has distribution with density $f_i(y_i,\eta_i) $ for $i=1,...,n$ where $\eta_i=X_i^T$ is the linear predictor. The generalized linear model with an exponential ...
- 253
0
votes
0
answers
48
views
Score statistic and Fisher information
In my exercise we assume that $Y_i|X_i$ has distribution with density $f_i(y_i,\eta_i) $ for $i=1,...,n$ where $\eta_i=X_i^T$ is the linear predictor. The generalized linear model with an exponential ...
- 253
0
votes
0
answers
25
views
Evaluating the asymptotic distribution of a metric that's a function of both ML estimated parameters and data, generalizing the delta method
I have a particular problem with likelihood function $\mathcal{L}(\theta \mid X)$, in which I interested in the distribution of a metric $m(X) = f(X,\hat\theta)$ (even asymptotically, though knowing ...
- 3,779
0
votes
0
answers
62
views
Fisher Information Matrix for Non Canonical link function for Poisson regression
I'm trying to solve this problem and I don't know how to do it since is not using the canonical link function. When I was researching online I couldn't find an example when the canonical function is ...
- 45
0
votes
0
answers
34
views
Implications of expectation of score function being zero
I understand why mathematically, the expectation of the score function (evaluated at the true parameter) is zero. But what are the implications of this fact? Can you give one or more examples on where ...
- 186
0
votes
1
answer
43
views
What is the definition and upper bound on the variable "m" in the definition of the multivariate normal Fisher Information?
Multivariate normal distribution [edit] The FIM for a $N$-variate
multivariate normal distribution, $X \sim N(\mu(\theta),
\Sigma(\theta))$ has a special form. Let the $K$-dimensional vector of
...
- 224
0
votes
1
answer
120
views
Monte Carlo approximation to find expected value of gradient square
I need to to calculate this term:
$
\mathbb{E}\left[S(Y, L,\theta)S(Y,L,\theta)^\prime\right]
$
Where
$
S(Y,L,\theta) =\frac{\partial}{\partial\theta}
l(Y,L,\theta)
$
With $\theta$ = maximum ...
- 25
2
votes
1
answer
120
views
Fisher Information matrix as covariance matrix from scores
I've found in a paper here. Observed Fisher information estimated in a way that does not convince me at all.
They estimated it as the covariance matrix of the scores, but to me the formula used is ...
- 25
1
vote
0
answers
38
views
Confusion about the Hessian approximation
There are two popular forms of the neural network Hessian approximation in the literature:
$$
H \simeq \sum_i \left(\frac{\partial y}{\partial w_i}\right)
\left(\frac{\partial y}{\partial w_i}\right)^...
1
vote
1
answer
141
views
How can nuisance parameters in Fisher matrix can deteriorate the useful constraints?
I have a Fisher matrix $F$ which has the matrix blocks form like this :
$$
F=\begin{bmatrix}
A & B\\
C & D
\end{bmatrix}
$$
The block $A$ is the most important block, in the sense the ...
- 29
2
votes
1
answer
38
views
Which is correct $Var(\theta) \ge (F^{-1})_{ii} $ or $Var(\theta) \ge 1/(F_{ii}) $?
With $\textbf{F}(\theta)$ denoting the Fisher matrix, $\textbf{V}(\theta)$ the variance matrix, $\textbf{C}(\theta)$ the covariance matrix, and parameter vector $\theta = [\theta_1, \theta_2, \dots,...
- 121
2
votes
1
answer
46
views
Fisher information as the variance of the 1st derivative of the log-lh different from the expectation of the 2nd derivative of the log-lh
I have the following pdf: $f(x)=\theta \times x^{\theta-1} \mathbf1(0\le x\le 1)$ where $\mathbf 1$ is the indicator function.
I am trying to calculate the Fisher information using the expectation of ...
- 23
11
votes
3
answers
3k
views
Connection between Fisher information and variance of score function
The fisher information's connection with the negative expected hessian at $\theta_{MLE}$, provides insight in the following way: at the MLE, high curvature implies that an estimate of $\theta$ even ...
0
votes
0
answers
38
views
discontinuity of Fisher information / confidence interval for Fisher information
From the question given at Fisher Information for frequency estimation under non-circular complex Gaussian noise when I consider bivariate normal distribution for $p(z|\omega,\phi)$, then I have found ...
- 111
1
vote
0
answers
34
views
Does the Fisher information matrix have to be convex when assessing an optimal criteria?
I've reached a local minimum for a proposed model using a set of experimental data (positive definite Hessian) and want to select an additional experiment that will reduce the parameter uncertainties (...
3
votes
1
answer
65
views
Fisher information as negative log likelihood
The Fisher Information is defined as the covariance matrix, or $E_{y \sim P(y;\theta)}[ \nabla_{\theta} ln(p(y;\theta)) \nabla_{\theta} ln(p(y;\theta))^T]$.
It can also be defined as $E_{y \sim P(y;\...
- 73
1
vote
0
answers
67
views
Fisher information matrix for model based design of experiments
I'm trying to use the Fisher information matrix and D-optimal design for model-based experiment selection, but I'm not sure if I'm implementing or interpreting it correctly.
I begin with the values of ...
0
votes
0
answers
11
views
What can make fisher information matrix diagonal? [duplicate]
How can we intuitively interprete the diagonal Fisher information matrix? Can we say these parameters are independent?
- 205
4
votes
0
answers
137
views
Fisher Information for frequency estimation under non-circular complex Gaussian noise
Consider a non-circular complex Gaussian noise $v$, which is given as $v = v_r + iv_i$. Here, real and imaginary components are independent and are Normal random variables such as $v_r \sim \mathcal{N}...
- 111
8
votes
1
answer
436
views
Are all log-likelihood functions twice differentiable?
For maximum likelihood estimation we need to set the first derivative of the log-likelihood function equal to $\mathbf{0}$.
The negative expected value of the Hessian matrix (second derivative) is ...
- 762
0
votes
0
answers
22
views
Should the expected observed fisher information matrix be positive definite in Fisher Scoring algorithm? [duplicate]
I want to use the Fisher Scoring Algorithm in Newton-Raphson method.
I(theta_m) in above that calculated at each iteration, has to be positive definite? Is it possible this matrix be indefinite?
- 71
6
votes
1
answer
283
views
Is it possible Fisher information matrix be indefinite?
I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function.
At each iteration, I want to check that is the Hessian matrix negative definite or not and I ...
- 71
1
vote
0
answers
53
views
Is it possible that Fisher information matrix be indefinite? [duplicate]
I`m using the Newton-Raphson method for obtaining MLE for parameters for maximizing my objective function.
At each iteration, I want to check that is the Hessian matrix negative definite or not and I ...
- 71
1
vote
0
answers
31
views
Fisher Information Matrix of Matrix Variate F Distribution
Let $\mathbf{X}$ follow a matrix variate F distribution with pdf
$$
\begin{align}
f\left(\mathbf{X} | \mathbf{\Sigma}, n, \nu\right) = \frac{\Gamma_p(\frac{n + \nu }{2})}{\Gamma_p(\frac{n}{2})\...
- 762
0
votes
0
answers
28
views
Is it possible to estimate the Hessian as the covariance of primal and cotangent?
Let's say we have a function
$$f: \mathbb R^n \to \mathbb R.$$
Can we numerically approximate the Hessian $f''(x)$ as
$$\textrm{Var}(a)^{-1} \textrm{Cov}(a, f'(a))$$
where
$$E(a) = x?$$
- 13.7k
1
vote
0
answers
38
views
Finding the score function
Suppose we have conditional distributional model: $$y|x\sim \mathcal{D}(g(x,\theta_1),h(x,\theta_2),\theta_3)$$
where $\mathcal{D}$ is known distribution, $(g(x,\theta_1)$ is conditional mean with ...
- 233
3
votes
0
answers
189
views
Information Matrix for Conditional Likelihood
I am studying the MLE theory on my own and I am confused by the difference between the fisher information matrix for the full sample and for one observation, when it comes to conditional likelihood. ...
- 1,103
0
votes
0
answers
72
views
Fisher information of an Ornstein-Uhlenbeck process
I would like to compute the Fisher information of an Ornstein-Uhlenbeck process $X_t = Y_t - \beta Z_t$ where $Y_t$ and $Z_t$ are two time-series.
My log-likelihood function in this case is:
$$\...
- 299
0
votes
0
answers
33
views
Covariance Matrix and it's dependence on the dataset size
I am new in the analysis of covariance matrix and plotting the corresponding ellipses. Here are my confusions in summary :
If I have a dataset of size: N rows.
I compute the covariance matrix, I get ...
- 151
2
votes
1
answer
148
views
Cramér–Rao Lower Bound and UMVUE for $\frac1{\theta}$
Problem: Find the UMVUE of $\frac1\theta$ for a random sample from the population distribution with density $$f(x;\theta)=\theta x^{\theta-1}$$ and show that its variance reaches the Cramér–Rao lower ...
- 23
4
votes
1
answer
206
views
Fisher information of $\rho$ in a symmetric normal $N_p(\mathbf 0,\Sigma)$ distribution
Suppose $\boldsymbol X=(X_1,\ldots,X_p)'\sim N_p(\mathbf0,\Sigma)$ where $\Sigma=(1-\rho)I_p+\rho\mathbf1\mathbf1'$ is positive definite. The objective is to obtain the asymptotic variance of the MLE ...
- 9,120
0
votes
0
answers
69
views
Does the score function have expectation of zero, or does the score function have expectation of zero when evaluated at the true parameter?
I am slightly confused about two versions of the idea that the score function has expected value of zero.
I learned that the score function is essentially the function of the slope of log likelihood. ...
- 131