The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
1answer
32 views

Differential Entropy of Gaussian Process

I have $N$ datapoints that have $d$ features in a GP and their covariance matrix $K$ and I want to calculate the differential entropy of that GP. Is this formula right? $E(I)= \frac{1}{2} ...
0
votes
1answer
39 views

Is the information matrix equality valid for the Poisson distribution?

As far as I know it should, since the support of the Poisson is independent of its lambda parameter. The negative of the expected Hessian equals $\frac{n}{\hat{\lambda}}$, where $\hat{\lambda}$ is ...
5
votes
3answers
196 views

Does dimension reduction always lose some information?

Like the title says, does dimension reduction always lose some information? Consider for example PCA. If the data I have is very sparse, I would assume a "better encoding" could be found (is this ...
3
votes
1answer
108 views

The Fisher information matrix

I have read two definitions of the Fisher information matrix. Are they equivalent? (The formulas given below are slightly modified versions of those given in the indicated sources, so as to bring out ...
2
votes
2answers
160 views

Word entropy / frequency in human speech

I am wondering how to best approximate the information value of a word $x$ in general human speech. By information value I literally mean its entropy: $H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in ...
1
vote
0answers
52 views

Observed info matrix via Hessian

In some resources, I saw that the observed information matrix is the negative of the expected value of the Hessian matrix. However, in some other resources I saw that it is just the negative of the ...
9
votes
2answers
578 views

Observed information matrix is a consistent estimator of the expected information matrix?

I am trying to prove that the observed information matrix evaluated at the weakly consistent maximum likelihood estimator (MLE), is a weakly consistent estimator of the expected information matrix. ...
3
votes
1answer
130 views

Item information in IRT

According to item information curves, item information for a 2PL IRT model is $I(\theta)=a^2_i p_i(\theta) q_i(\theta)$ To determine $p_i(\theta)$ and $q_i(\theta)$, do you just use the observed ...
4
votes
1answer
1k views

Information gain as a feature selection for 3-class classification problem

I am facing a sentiment analysis task where I am using Naive Bayes to classify documents as Positive, Negative or Neutral. I have thought of using Information Gain as my filter for feature selection. ...
24
votes
6answers
1k views

Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to ...
1
vote
0answers
550 views

What can be going wrong when Maximum Likelihood standard errors are high?

In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of ...
22
votes
8answers
4k views

Measuring entropy/ information/ patterns of a 2d binary matrix

I want to measure the entropy/ information density/ pattern-likeness of a two-dimensional binary matrix. Let me show some pictures for clarification: This display should have a rather high entropy: ...
3
votes
1answer
283 views

How to interpret the divergence of Fisher information expectation?

Consider translated Weibull distribution with probability density function: $$ f(x ; k, \lambda, \theta) = \frac{k}{\lambda} \left( \frac{x-\theta}{\lambda} \right)^{k-1} \exp\left( - ...
2
votes
0answers
44 views

How to interpret the divergence of Fisher information expectation [duplicate]

Possible Duplicate: How to interpret the divergence of Fisher information expectation? Consider translated Weibull distribution with probability density function: $$ f(x ; k, \lambda, ...
5
votes
2answers
284 views

Is it appropriate to use the term “bits” to discuss a log-base-2 likelihood ratio?

I'm quite enamoured with likelihood ratios as a means of quantifying relative evidence in scientific endeavours. However, in practice I find that the raw likelihood ratio can get unprintably large, so ...
3
votes
2answers
1k views

Calculating the transfer entropy in R

The transfer entropy, from information theory, is an effective way to measure the one-way information dependence between two variables. A nice high-level summary is here: ...
6
votes
0answers
183 views

Fisher information in a hierarchical model

Given the following hierarchical model, $$ X \sim {\mathcal N}(\mu,1), $$ and, $$ \mu \sim {\rm Laplace}(0, c) $$ where $\mathcal{N}(\cdot,\cdot)$ is a normal distribution. Is there a way to get an ...