# Questions tagged [information]

The tag has no usage guidance.

43 questions
Filter by
Sorted by
Tagged with
15k views

### Measuring entropy/ information/ patterns of a 2d binary matrix

I want to measure the entropy/ information density/ pattern-likeness of a two-dimensional binary matrix. Let me show some pictures for clarification: This display should have a rather high entropy: ...
6k views

### Why would someone use a Bayesian approach with a 'noninformative' improper prior instead of the classical approach?

If the interest is merely estimating the parameters of a model (pointwise and/or interval estimation) and the prior information is not reliable, weak, (I know this is a bit vague but I am trying to ...
623 views

### Fisher information in a hierarchical model

Given the following hierarchical model, $$X \sim {\mathcal N}(\mu,1),$$ and, $$\mu \sim {\rm Laplace}(0, c)$$ where $\mathcal{N}(\cdot,\cdot)$ is a normal distribution. Is there a way to get an ...
3k views

### Observed information matrix is a consistent estimator of the expected information matrix?

I am trying to prove that the observed information matrix evaluated at the weakly consistent maximum likelihood estimator (MLE), is a weakly consistent estimator of the expected information matrix. ...
5k views

### Does dimension reduction always lose some information?

Like the title says, does dimension reduction always lose some information? Consider for example PCA. If the data I have is very sparse, I would assume a "better encoding" could be found (is this ...
2k views

### Is it appropriate to use the term “bits” to discuss a log-base-2 likelihood ratio?

I'm quite enamoured with likelihood ratios as a means of quantifying relative evidence in scientific endeavours. However, in practice I find that the raw likelihood ratio can get unprintably large, so ...
5k views

### Calculating the transfer entropy in R

The transfer entropy, from information theory, is an effective way to measure the one-way information dependence between two variables. A nice high-level summary is here: http://lizier.me/joseph/...
8k views

### Why is variance (instead of standard deviation) the default measure of information content in principal components?

The information content of principal components is almost always expressed as a variance (e.g., in scree plots or in statements like "the first three PCs contain 95% of the total data variance"). The ...
2k views

207 views

### Fisher information matrix of two parameter exponential distribution

Is it possible to find the Fisher Information matrix of the two parameter (scale and location) exponential distribution? Any hint?
61 views

### Is there an online guide on how to visualise different kinds of survey question types (multiple choice, ranking the options, etc) in Microsoft Excel?

Is there an online guide on how to visualise different kinds of survey question types (multiple choice, ranking the options, etc) in Microsoft Excel? In particular, I am interested in how to ...
41 views

### Entropy evolution while learning?

It is fairly well known that $$H(X|Y)\le H(X),$$ the posterior entropy is smaller than the prior entropy. This is similar to $$\mathbb{E}_Y[\mathbb{V}ar_X[X|Y]]\le \mathbb{V}ar_X[X]$$ which follows ...
74 views

### Combining categories by Weight of Evidence

When calculating Information Value and Weight of Evidence, it's possible to draw a chart of WoE for each variable to study its effect on the state of the target variable. Now, I know it's possible to ...
79 views

### Information about parameters using priors distributions [duplicate]

When using the "non-informative" prior $\pi(\mu,\sigma)\propto\frac{1}{\sigma^2}$ where $\pi(\mu)\propto1$ and $\pi(\sigma^2)\propto\frac{1}{\sigma^2}$ Where is the no information for the ...
26 views

### Identifying / visualising the most informative data source or combination of data sources

I have observations on the status (say, dead = 0, alive = 1) of a number of subjects as recorded in three distinct data sources at the same point in time: +---------+---------+---------+---------+ | ...
81 views

### Difference between two entropic states of the same variable

I am interested in finding the difference of entropy between two states of the same variable. The probability distribution associated with the variable X changes at each discrete point in time t by an ...
55 views

### Finding a likelihood function for similarities

in order to compare human action sequences and computer modeled predictions for action sequences I use a similarity measure for these sequences. All similarities can have a value between 0 (terrible ...
177 views

### Observed info matrix via Hessian

In some resources, I saw that the observed information matrix is the negative of the expected value of the Hessian matrix. However, in some other resources I saw that it is just the negative of the ...
1k views

### What can be going wrong when Maximum Likelihood standard errors are high?

In maximum likelihood estimation (MLE) a useful result is that the standard errors for some estimated coefficient vector can be computed as the square roots of the diagonal entries of the inverse of ...
1k views

### Standard errors of the MLEs

Can anybody tell me how to find numerical values for standard errors of the MLEs of Weibull distribution using the uncensored real data set on the breaking stress of carbon fibres (in Gba) reported by ...
9 views

### How to measure the information of covariates in a ML task?

Background Recently, I do 2 different ML projects. One is lending club loan prediction, another is a pravite dataset in online experiment field to predict whether a customer will take the treatment....
18 views

### Kullback–Leibler and the Brier score?

Both seem to be quite obviously about prediction and sort of map one probability distribution onto another one. Whereas with the DKL (https://en.wikipedia.org/wiki/Kullback%E2%80%...
5 views

### Analyzing 1000s of time-stamped tweets. How to automatically identify spikes in terms?

I have about 30,000 tweets from a corporate customer service account, all harvested according to the Twitter TOC. Naturally, each message is time-stamped and not extremely long. I'm trying to ...
34 views

### Nested sampling: estimate of bulk posterior support over prior

Going through the details of the Nested sampling Skilling paper, and I've encountered an estimate in Section 5 which I cannot reproduce. Rephrasing what's mentioned in the paper: we assume to have a ...
18 views

### Information Gain property

Studying about information gain I found in the web (from the presentation of a lecture) that $IG(C|X) = IG(X|C)$ is it true? How I prove it?
22 views

### What are most recent research work on the problem of key phrases extraction from a text corpus?

I am interested in the problem of extracting key phrases from a text corpus. This is different from the keyword extraction problem, which is only for a particular document. This problem helps us, for ...
32 views

### Finding Fisher Matrix for Line Fitting

I am going through the "Fitting a Line" example from here. $f_1 = ax_1 + b$ and $f_2 = ax_2 + b$ are the models used to observed two data points in $R^2$. If $\sigma_1$ and $\sigma_2$ is the ...
81 views

### Mutual Information from Multiple Sources

The mutual information gain expression is $$H(X) - H(X | Y)$$ If I have a set of data sources, $\mathbf{X} = \lbrace X_0, X_1,\ldots,X_m \rbrace$, then I start with the simplest mutual ...
138 views

### What is information gain ratio?

With respect to data mining, what is information gain ratio? I'm a complete beginner to data analytics and mining, so please explain at a low level of understanding.
I was reading about the concept of Perplexity and was thinking whether there's a connexion with the law of law of total variance, but I couldn't find any reference. The law of total variance is: ...
I am looking at factors affecting firm value (the dependent variable). I fitted the following model using OLS: $log(FirmValue_i)=\beta_0 + \beta_1Divers_i+ \beta_2HQLoc_i + \epsilon_i$ where \$...