Questions tagged [information]

The tag has no usage guidance.

20 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
2
votes
0answers
93 views

Gibbs entropy and Shannon entropy

The two formulations seem identical to me: $H(x) = \sum p(x) log(1/p(x))$ why would the equation be attributed to Shannon rather than Gibbs (in the context of information)?
2
votes
0answers
287 views

When is BIC reasonable approximation to evidence?

I've recently seen a few papers in physics using the Bayesian information criterion (BIC) to evaluate models. I'm much more familiar with Bayesian evidence, $p(x|M)$. I've read in a few places, e.g. ...
1
vote
0answers
41 views

Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
1
vote
0answers
75 views

Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
1
vote
0answers
26 views

Identifying / visualising the most informative data source or combination of data sources

I have observations on the status (say, dead = 0, alive = 1) of a number of subjects as recorded in three distinct data sources at the same point in time: +---------+---------+---------+---------+ | ...
1
vote
0answers
81 views

Difference between two entropic states of the same variable

I am interested in finding the difference of entropy between two states of the same variable. The probability distribution associated with the variable X changes at each discrete point in time t by an ...
1
vote
0answers
55 views

Finding a likelihood function for similarities

in order to compare human action sequences and computer modeled predictions for action sequences I use a similarity measure for these sequences. All similarities can have a value between 0 (terrible ...
1
vote
0answers
177 views

Observed info matrix via Hessian

In some resources, I saw that the observed information matrix is the negative of the expected value of the Hessian matrix. However, in some other resources I saw that it is just the negative of the ...
1
vote
0answers
1k views

What can be going wrong when Maximum Likelihood standard errors are high?

In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of ...
0
votes
0answers
9 views

How to measure the information of covariates in a ML task?

Background Recently, I do 2 different ML projects. One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment....
0
votes
0answers
18 views

Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
0
votes
0answers
5 views

Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
0
votes
0answers
34 views

Nested sampling: estimate of bulk posterior support over prior

Going through the details of the Nested sampling Skilling paper, and I've encountered an estimate in Section 5 which I cannot reproduce. Rephrasing what's mentioned in the paper: we assume to have a ...
0
votes
0answers
18 views

Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
0
votes
0answers
22 views

What are most recent research work on the problem of key phrases extraction from a text corpus?

I am interested in the problem of extracting key phrases from a text corpus. This is different from the keyword extraction problem, which is only for a particular document. This problem helps us, for ...
0
votes
1answer
32 views

Finding Fisher Matrix for Line Fitting

I am going through the "Fitting a Line" example from here. $f_1 = ax_1 + b$ and $f_2 = ax_2 + b$ are the models used to observed two data points in $R^2$. If $\sigma_1$ and $\sigma_2$ is the ...
0
votes
0answers
81 views

Mutual Information from Multiple Sources

The mutual information gain expression is $$ H(X) - H(X | Y) $$ If I have a set of data sources, $ \mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace $, then I start with the simplest mutual ...
0
votes
0answers
138 views

What is information gain ratio?

With respect to data mining, what is information gain ratio? I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.
0
votes
0answers
240 views

Perplexity, cross/conditional entropy and law of total variance

I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference. The law of total variance is: $$...
0
votes
0answers
44 views

Interpretation of Model When Intentionally Excluding a Control Variable!

I am looking at factors affecting firm value (the dependent variable). I fitted the following model using OLS: $log(FirmValue_i)=\beta_0 + \beta_1Divers_i+ \beta_2HQLoc_i + \epsilon_i$ where $...