# Tagged Questions

The tag has no wiki summary.

32 views

### Maximum entropy priors for hypotheses

Suppose I have a random variable X which can take integer values 0 through 99. It can be, for example, numbers written on balls in a large urn. I have three hypotheses about the distribution of these ...
80 views

### Maximum entropy and non-informative distribution

Is Maximum Entropy rule equivalent to non-informativeness? In other words, when maximizing the entropy of a distribution, given some known stuff, is it equivalent to finding to most non-informative ...
45 views

### Maximum Entropy and Multinomial Logistic Function

I have a newbie question. I tried to find the answer from Google but couldn't get a clear answer. Is the MaxEnt model exactly same as multinomial logistic regression (i.e., softmax regression)? It ...
89 views

### Why is Entropy maximised when the probability distribution is uniform?

I know that entropy is the measure of randomness of a process/variable and it can be defined as follows. for a random variable $X \in$ set $A$ :- $H(X)= \sum_{x_i \in A} -p(x_i) \log (p(x_i))$. In ...
91 views

### Entropy of (Sum of Gaussians) versus Sum of (Entropy of Gaussians)

Short version: How can the joint entropy of two independent variables be less than the sum of those independent variables? The joint entropy should encode all information that a scalar function can, ...
50 views

### Maximum Entropy Markov Model Calculations

I am new to ML and I'm having trouble figuring out how to implement a Maximum Entropy Markov Model for a sequence labeling task. Given this MEMM equation  ...
94 views

### Entropy calculations and Process ranking

Still working on my financial datasets, I am now trying to observe the effects of the pre-processing on the results of my study. To sum it up: I have thirty time series of prices (the main indices, ...
51 views

### MaxEnt prior for positive variable that decreases monotonically to zero at a specified bound

I'm trying to devise a prior for a model parameter $x$ about which I know the following things: It is strictly positive. There is a maximum possible value $x_m$. Larger values are less likely than ...
93 views

### Multiplying two entropy values

I have seen formulas that either sum or subtract entropy values (eg Information Gain). However, I have not seen a "multiplication/division of entropy values" anywhere. I would like to know if there is ...
162 views

### Entropy-based methods in R

I was wondering if anybody knows of the existence of an R package which implements entropy-based methods (maximum empirical likelihood, maximum exponential empirical likelihood, minimum discrepancy ...
216 views

### Maximum entropy of dice

How can be proven that the entropy of a dice with equal probability for all its faces is at its maximum? It's clear that the entropy will be smaller if there are more chances for a particular face, ...
83 views

### Maximum entropy priors in infinite dimensional spaces

Has the idea of a maximum entropy probability distribution been explored for function spaces, and if so what are some key papers, books, or terms to look for? For $\mathbb{R}^n$ (and discrete ...
37 views

### What makes values from two features equal for the purposes of constructing a maximum entropy model?

I'm looking to construct a maximum entropy model using general iterative scaling. In the process, I've come across the SharpEntropy project for .NET and I'm using that as a reference for the ...
173 views

### Feature selection for SVM and Maximum Entropy

In text classification problems where the number of features >> number of documents, is it useful to perform feature selection with filters (e.g. Information Gain) when using Naive Bayes. However, ...
196 views

### “Since $x$ is near-gaussian, its PDF can be written as…”

Short question: Why is this true?? Long question: Very simply, I am trying to figure out what justifies this first equation. The author of the book I am reading, (context here if you want it, but ...
184 views

### Maximum entropy inference for k-means clustering

I am taking a course on Maximum Entropy Inference (MEI), where its application to k-Means was discussed. I am confused about the problem setting. As far as I understand, our goal is to find ...
1k views

### Entropy-based refutation of Shalizi's Bayesian backward arrow of time paradox?

In this paper, the talented researcher Cosma Shalizi argues that to fully accept a subjective Bayesian view, one must also accept an unphysical result that the arrow of time (given by the flow of ...
220 views

### Maximum entropy sampler

I want to sample from a distribution which has fixed to a given values mean(=0), standard deviation(=1), skewness(=0) and kurtosis. I also want this distribution to be as general as possible, i.e. to ...
497 views

### Maximum entropy classifier and sentiment analysis

I am doing a project work in sentiment analysis (on Twitter data) using machine learning approach. In order to find the 'best' way to this I have experimented with naive Bayesian and maximum entropy ...
557 views

### Entropy based on euclidian distances between datapoints / clusters centers?

Is it possible to define any useful entropy or conditional entropy which is based on the distance between datapoint(s) and cluster center(s), instead of basing on the number of points assigned to ...
208 views

### Creating a maximum entropy Markov model from an existing multi-input maximum entropy classifier

I am intrigued by the concept of a Maximum Entropy Markov Model (MEMM), and I am thinking of using it for a Part of Speech (POS) tagger. At the moment, I am using a conventional Maximum Entropy (ME) ...