Suppose we estimate the probability mass function $p$ from a iid training sample $\{x_1,\dots,x_n\}$. Then we want to use the estimated pmf $\hat{p}$ to estimate the probabilities $p(y_j)$ for a iid test sample $\{y_1, \dots,y_m\}$ which are supposed to come from the same distribution as the training sample.

However, we don't know the entire sample space when we estimate the pmf from the training sample. Specifically, we anticipate that there may be some test outcomes, $y_j$'s, which have not been seen in the training example, i.e. $y_j \neq x_i, i \neq 1,\dots, n$, and furthermore, we don't know what $y_j$'s are and how many of them. My question is how we can estimate the pmf from the training sample to deal with such unknown testing outcomes?

Note that, this is different from the case when we know the entire sample space, but not observe some outcomes. In this case, we can use Laplace smoothing to move probability mass to those with zero occurrences.

The question arises from estimating the distributions for N-gram models in NLP.


  • $\begingroup$ This is impossible in general without imposing some structure on the unexplored part of the sample space. If there are 3 outcomes and you've only ever seen 1, how can you say the relative likelihood of the other two? A bayesian approach would give you a prior to start from, at least. In practice that should make the out-of-sample probabilities very small. $\endgroup$ – MichaelChirico Jul 30 at 3:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.