Entropy of a non-representative sample of discrete distribution Let $X$ be a discrete random variable which can take $K$ distinct values. Let $S$ be a random i.i.d. sample of $X$ containing $n$ values, $n \ll K$. What can be said about the relation between the entropy of true distribution of $X$ and the entropy of the empirical distribution calculated from sample $S$? Are there any scaling laws?
 A: It is not a simple problem but special cases have been researched intensively and some techniques are quite mature. 
Most of them assume a certain structural property about the distribution of $X$ (often stated as a Bayesian prior). The most classical case is when $X$ is "a word chosen in usual language". Some techniques work very well with small $n$ (compared to $K$ that is total vocabulary size). This applies to a variety of situations (not just language).
There is a very good summary of entropy estimation methods (when $n≪K$) here:
https://math.stackexchange.com/questions/604654/estimating-the-entropy
Additional notes:
I've only looked at NSB so far. A rule of thumb is that NSB starts providing useful results when $n$ is above $2^{S/2}$ while a naive method works only when $n$ is above $2^S$. It relies on exploiting the number of collisions. This text is an easy to read introduction ot NSB: http://www.nowozin.net/sebastian/blog/estimating-discrete-entropy-part-3.html. I don't mean I recommend NSB over other methods, I'm far from being an expert in all this. I have asked the same question as you a while ago, and my naive thoughts about it may be of some use: Entropy estimation with fewer data lines than bins
