# Questions tagged [information]

The tag has no usage guidance.

20 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
93 views

### Gibbs entropy and Shannon entropy

The two formulations seem identical to me: $H(x) = \sum p(x) log(1/p(x))$ why would the equation be attributed to Shannon rather than Gibbs (in the context of information)?
287 views

### When is BIC reasonable approximation to evidence?

I've recently seen a few papers in physics using the Bayesian information criterion (BIC) to evaluate models. I'm much more familiar with Bayesian evidence, $p(x|M)$. I've read in a few places, e.g. ...
41 views

### Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
75 views

### Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
26 views

### Identifying / visualising the most informative data source or combination of data sources

I have observations on the status (say, dead = 0, alive = 1) of a number of subjects as recorded in three distinct data sources at the same point in time: +---------+---------+---------+---------+ | ...
81 views

### Difference between two entropic states of the same variable

I am interested in finding the difference of entropy between two states of the same variable. The probability distribution associated with the variable X changes at each discrete point in time t by an ...
55 views

### Finding a likelihood function for similarities

in order to compare human action sequences and computer modeled predictions for action sequences I use a similarity measure for these sequences. All similarities can have a value between 0 (terrible ...
177 views

### Observed info matrix via Hessian

In some resources, I saw that the observed information matrix is the negative of the expected value of the Hessian matrix. However, in some other resources I saw that it is just the negative of the ...
1k views

### What can be going wrong when Maximum Likelihood standard errors are high?

In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of ...
9 views

### How to measure the information of covariates in a ML task?

Background Recently, I do 2 different ML projects. One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment....
18 views

### Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
5 views

### Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
34 views

### Nested sampling: estimate of bulk posterior support over prior

Going through the details of the Nested sampling Skilling paper, and I've encountered an estimate in Section 5 which I cannot reproduce. Rephrasing what's mentioned in the paper: we assume to have a ...
18 views

### Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
22 views

### What are most recent research work on the problem of key phrases extraction from a text corpus?

I am interested in the problem of extracting key phrases from a text corpus. This is different from the keyword extraction problem, which is only for a particular document. This problem helps us, for ...
32 views

### Finding Fisher Matrix for Line Fitting

I am going through the "Fitting a Line" example from here. $f_1 = ax_1 + b$ and $f_2 = ax_2 + b$ are the models used to observed two data points in $R^2$. If $\sigma_1$ and $\sigma_2$ is the ...
81 views

### Mutual Information from Multiple Sources

The mutual information gain expression is $$H(X) - H(X | Y)$$ If I have a set of data sources, $\mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace$, then I start with the simplest mutual ...
138 views

### What is information gain ratio?

With respect to data mining, what is information gain ratio? I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.
I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference. The law of total variance is: ...
I am looking at factors affecting firm value (the dependent variable). I fitted the following model using OLS: $log(FirmValue_i)=\beta_0 + \beta_1Divers_i+ \beta_2HQLoc_i + \epsilon_i$ where \$...