We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [information]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
5 views

Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
53
votes
10answers
15k views

Measuring entropy/ information/ patterns of a 2d binary matrix

I want to measure the entropy/ information density/ pattern-likeness of a two-dimensional binary matrix. Let me show some pictures for clarification: This display should have a rather high entropy: ...
0
votes
0answers
25 views

Compute the information matries related to normal distribution

This is a problem that I have trouble with. Suppose that we have $X_{1}, \ldots, X_{m}$ are iid $N\left(\mu, \sigma^{2}\right), Y_{1}, \ldots, Y_{n}$ are iid $N\left(0, \sigma^{2}\right),$ the $X$ 's ...
0
votes
0answers
27 views

Nested sampling: estimate of bulk posterior support over prior

Going through the details of the Nested sampling Skilling paper, and I've encountered an estimate in Section 5 which I cannot reproduce. Rephrasing what's mentioned in the paper: we assume to have a ...
1
vote
0answers
36 views

Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
1
vote
0answers
42 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
1
vote
1answer
202 views

Fisher information matrix of two parameter exponential distribution

Is it possible to find the Fisher Information matrix of the two parameter (scale and location) exponential distribution? Any hint?
7
votes
5answers
5k views

Calculating the transfer entropy in R

The transfer entropy, from information theory, is an effective way to measure the one-way information dependence between two variables. A nice high-level summary is here: http://lizier.me/joseph/...
0
votes
0answers
17 views

Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
5
votes
1answer
125 views

Why not normalizing mutual information with harmonic mean of entropies?

This is a similar question to this one (which has unfortunately no answer yet), although I believe my question is more specific. Let $X$ and $Y$ be two discrete random variables with outcome space, ...
1
vote
0answers
75 views

Information about parameters using priors distributions [duplicate]

When using the "non-informative" prior $\pi(\mu,\sigma)\propto\frac{1}{\sigma^2}$ where $\pi(\mu)\propto1$ and $\pi(\sigma^2)\propto\frac{1}{\sigma^2}$ Where is the no information for the ...
2
votes
2answers
276 views

Bayesian Statistics -Prior and Posterior distributions

Please is it ever possible for the prior distribution to contain more information about parameter(s) than the posterior distribution? If yes, when can that occur? Is it the same concept as the ...
1
vote
1answer
33 views

Are there any measures of Nonspecificity that take into account unreachable states?

The Hartley function, the main measure of Nonspecificity, is essentially based on the count of all possible states. However, while coding the Hartley, I realized that "possible" means all remaining ...
0
votes
0answers
22 views

What are most recent research work on the problem of key phrases extraction from a text corpus?

I am interested in the problem of extracting key phrases from a text corpus. This is different from the keyword extraction problem, which is only for a particular document. This problem helps us, for ...
5
votes
2answers
2k views

Word entropy / frequency in human speech

I am wondering how to best approximate the information value of a word $x$ in general human speech. By information value I literally mean its entropy: $H(X) = \mathbb{E}_{X} [I(x)] = -\sum_{x \in \...
0
votes
1answer
28 views

Finding Fisher Matrix for Line Fitting

I am going through the "Fitting a Line" example from here. $f_1 = ax_1 + b$ and $f_2 = ax_2 + b$ are the models used to observed two data points in $R^2$. If $\sigma_1$ and $\sigma_2$ is the ...
0
votes
0answers
80 views

Mutual Information from Multiple Sources

The mutual information gain expression is $$ H(X) - H(X | Y) $$ If I have a set of data sources, $ \mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace $, then I start with the simplest mutual ...
0
votes
0answers
129 views

What is information gain ratio?

With respect to data mining, what is information gain ratio? I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.
1
vote
0answers
26 views

Identifying / visualising the most informative data source or combination of data sources

I have observations on the status (say, dead = 0, alive = 1) of a number of subjects as recorded in three distinct data sources at the same point in time: +---------+---------+---------+---------+ | ...
44
votes
7answers
5k views

Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to ...
1
vote
0answers
77 views

Difference between two entropic states of the same variable

I am interested in finding the difference of entropy between two states of the same variable. The probability distribution associated with the variable X changes at each discrete point in time t by an ...
2
votes
1answer
61 views

Metric for temporal deviation or variation from mean? Hurst or one-sample K-S test?

I am not sure I am articulating this question properly, and my unfamiliarity with the proper terminology has hindered my ability to research this question. I am looking for a metric that captures ...
0
votes
0answers
234 views

Perplexity, cross/conditional entropy and law of total variance

I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference. The law of total variance is: $$...
1
vote
1answer
59 views

Is there an online guide on how to visualise different kinds of survey question types (multiple choice, ranking the options, etc) in Microsoft Excel?

Is there an online guide on how to visualise different kinds of survey question types (multiple choice, ranking the options, etc) in Microsoft Excel? In particular, I am interested in how to ...
20
votes
2answers
599 views

Fisher information in a hierarchical model

Given the following hierarchical model, $$ X \sim {\mathcal N}(\mu,1), $$ and, $$ \mu \sim {\rm Laplace}(0, c) $$ where $\mathcal{N}(\cdot,\cdot)$ is a normal distribution. Is there a way to get an ...
2
votes
0answers
92 views

Gibbs entropy and Shannon entropy

The two formulations seem identical to me: $H(x) = \sum p(x) log(1/p(x))$ why would the equation be attributed to Shannon rather than Gibbs (in the context of information)?
2
votes
0answers
274 views

When is BIC reasonable approximation to evidence?

I've recently seen a few papers in physics using the Bayesian information criterion (BIC) to evaluate models. I'm much more familiar with Bayesian evidence, $p(x|M)$. I've read in a few places, e.g. ...
1
vote
0answers
51 views

Finding a likelihood function for similarities

in order to compare human action sequences and computer modeled predictions for action sequences I use a similarity measure for these sequences. All similarities can have a value between 0 (terrible ...
0
votes
0answers
44 views

Interpretation of Model When Intentionally Excluding a Control Variable!

I am looking at factors affecting firm value (the dependent variable). I fitted the following model using OLS: $log(FirmValue_i)=\beta_0 + \beta_1Divers_i+ \beta_2HQLoc_i + \epsilon_i$ where $...
5
votes
1answer
8k views

Why is variance (instead of standard deviation) the default measure of information content in principal components?

The information content of principal components is almost always expressed as a variance (e.g., in scree plots or in statements like "the first three PCs contain 95% of the total data variance"). The ...
7
votes
1answer
2k views

Is it appropriate to use the term “bits” to discuss a log-base-2 likelihood ratio?

I'm quite enamoured with likelihood ratios as a means of quantifying relative evidence in scientific endeavours. However, in practice I find that the raw likelihood ratio can get unprintably large, so ...
4
votes
1answer
6k views

Information gain as a feature selection for 3-class classification problem

I am facing a sentiment analysis task where I am using Naive Bayes to classify documents as Positive, Negative or Neutral. I have thought of using Information Gain as my filter for feature selection. ...
0
votes
1answer
969 views

Standard errors of the MLEs

Can anybody tell me how to find numerical values for standard errors of the MLEs of Weibull distribution using the uncensored real data set on the breaking stress of carbon fibres (in Gba) reported by ...
1
vote
1answer
789 views

Differential Entropy of Gaussian Process

I have $N$ datapoints that have $d$ features in a GP and their covariance matrix $K$ and I want to calculate the differential entropy of that GP. Is this formula right? $E(I)= \frac{1}{2} \log((2πe)^...
16
votes
2answers
3k views

Observed information matrix is a consistent estimator of the expected information matrix?

I am trying to prove that the observed information matrix evaluated at the weakly consistent maximum likelihood estimator (MLE), is a weakly consistent estimator of the expected information matrix. ...
1
vote
0answers
177 views

Observed info matrix via Hessian

In some resources, I saw that the observed information matrix is the negative of the expected value of the Hessian matrix. However, in some other resources I saw that it is just the negative of the ...
3
votes
1answer
587 views

The Fisher information matrix

I have read two definitions of the Fisher information matrix. Are they equivalent? (The formulas given below are slightly modified versions of those given in the indicated sources, so as to bring out ...
10
votes
3answers
4k views

Does dimension reduction always lose some information?

Like the title says, does dimension reduction always lose some information? Consider for example PCA. If the data I have is very sparse, I would assume a "better encoding" could be found (is this ...
4
votes
1answer
633 views

How to interpret the divergence of Fisher information expectation?

Consider translated Weibull distribution with probability density function: $$ f(x ; k, \lambda, \theta) = \frac{k}{\lambda} \left( \frac{x-\theta}{\lambda} \right)^{k-1} \exp\left( - \left(\frac{...
3
votes
0answers
63 views

How to interpret the divergence of Fisher information expectation [duplicate]

Possible Duplicate: How to interpret the divergence of Fisher information expectation? Consider translated Weibull distribution with probability density function: $$ f(x ; k, \lambda, \theta) =...
4
votes
1answer
380 views

Item information in IRT

According to item information curves, item information for a 2PL IRT model is $I(\theta)=a^2_i p_i(\theta) q_i(\theta)$ To determine $p_i(\theta)$ and $q_i(\theta)$, do you just use the observed ...
1
vote
0answers
1k views

What can be going wrong when Maximum Likelihood standard errors are high?

In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of ...