I have about 30000 observations of a variable $Y$ for different categories of a treatment $X$. Some treatments have over 500 samples and it's very reasonable to take just these cases ($P(Y|X) \simeq freq(Y|X)$), other potential treatments have never been seen, and thus it is reasonable to apply all the data (or the mean over all seen categories, $P(Y) \simeq freq(\mathbb E[Y|X])$. What I'm wondering is how to address those categories that have very few samples (many have just 1).

Ideally I'd like to have a parameter that allows me to interpolate between the observed distribution for the entire sample and the distribution for the sample of a given treatment, which places more wight in the latter the more samples we have.

I need to interpolate the distributions (not just the means) because I haven't got a good model for the distributions of $X$ (they're certainly not Gaussian, Poisson or anything I can identify), and I need to keep track of confidence intervals.


1 Answer 1


This problem reminds me a bit of the settings for the Krichevsky-Trofimov estimator, where symbols are to be predicted, some of which might have not been observed, or observed with very low counts. The solution there is to add a-priori a low (possibly fractional) count to any symbol before starting the observation-based counting.

In the question, you (implicitly) describe a "neutral" distribution over the data - say $E$ - which could be defined in at least two ways:

  1. The empirical distribution of $Y$.

  2. The empirical distribution of the mean of $Y|x$ per category $x$, taken over all $x$.

The choice of how to determine $E$ depends on the settings, but let's say we have such a neutral distribution $E$.

Set $q > 0$, and find a sample $E'$ obtained by the boundaries of $q$ quantiles of $E$. (Note that for larger $q$, $E' \sim E$.) Set another positive $w$. For each category $x$ of $X$, approximate the distribution as the distribution obtained by the union of $E'$ and $w$ copies of $Y | x$.

So, for example if $q = 10$, $w=1$, then:

  • for an $x$ that hasn't been observed, the distribution will be exactly that of the neutral distribution approximation - the 10 points of $E'$

  • for an $x$ that has been observed once, the distribution will be almost that of the neutral distribution approximation - the 10 points of $E'$, and the single point of $Y | x$.

  • for an $x$ that has been observed 500 times, the distribution will be very similar to $Y | x$ - the 10 points of $E'$, and the 500 points of $Y | x$.

The parameter $w$ is a tradeoff: higher values will give tighter confidence intervals for both large categories and low categories - only you can set it (or at least you need to define an objective). The parameter $q$ is mainly for computational shortcuts. Without computational restrictions, $q$ can be made the number of samples determining $E$. (Of course, $w$ needs to scale with $q$.)

  • $\begingroup$ thanks for the hint. It sounds reasonable. Do you have a reference to read up more on this? $\endgroup$ Oct 13, 2016 at 9:40
  • $\begingroup$ @AlexMonras Krichevsky & Trofimov's articles seem to be accessible through IEEE paysites. The closest accessible material I could find was on Good Turing smoothing, which shares similar ideas. Please note that, irrespectively, there's something of a leap by applying the idea from symbol frequency estimation, to conditional distribution estimation. I would say that this is an "inspired by" heuristic. $\endgroup$
    – Ami Tavory
    Oct 13, 2016 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.