regularization for categories with few samples I have about 30000 observations of a variable $Y$ for different categories of a treatment $X$. Some treatments have over 500 samples and it's very reasonable to take just these cases ($P(Y|X) \simeq freq(Y|X)$), other potential treatments have never been seen, and thus it is reasonable to apply all the data (or the mean over all seen categories, $P(Y) \simeq freq(\mathbb E[Y|X])$. What I'm wondering is how to address those categories that have very few samples (many have just 1).
Ideally I'd like to have a parameter that allows me to interpolate between the observed distribution for the entire sample and the distribution for the sample of a given treatment, which places more wight in the latter the more samples we have. 
I need to interpolate the distributions (not just the means) because I haven't got a good model for the distributions of $X$ (they're certainly not Gaussian, Poisson or anything I can identify), and I need to keep track of confidence intervals.
 A: This problem reminds me a bit of the settings for the Krichevsky-Trofimov estimator, where symbols are to be predicted, some of which might have not been observed, or observed with very low counts. The solution there is to add a-priori a low (possibly fractional) count to any symbol before starting the observation-based counting.

In the question, you (implicitly) describe a "neutral" distribution over the data - say $E$ - which could be defined in at least two ways:


*

*The empirical distribution of $Y$.

*The empirical distribution of the mean of $Y|x$ per category $x$, taken over all $x$.
The choice of how to determine $E$ depends on the settings, but let's say we have such a neutral distribution $E$. 
Set $q > 0$, and find a sample $E'$ obtained by the boundaries of $q$ quantiles of $E$. (Note that for larger $q$, $E' \sim E$.) Set another positive $w$. For each category $x$ of $X$, approximate the distribution as the distribution obtained by the union of $E'$ and $w$ copies of $Y | x$. 
So, for example if $q = 10$, $w=1$, then:


*

*for an $x$ that hasn't been observed, the distribution will be exactly that of the neutral distribution approximation - the 10 points of $E'$

*for an $x$ that has been observed once, the distribution will be almost that of the neutral distribution approximation - the 10 points of $E'$, and the single point of $Y | x$.

*for an $x$ that has been observed 500 times, the distribution will be very similar to $Y | x$ - the 10 points of $E'$, and the 500 points of $Y | x$.
The parameter $w$ is a tradeoff: higher values will give tighter confidence intervals for both large categories and low categories - only you can set it (or at least you need to define an objective). The parameter $q$ is mainly for computational shortcuts. Without computational restrictions, $q$ can be made the number of samples determining $E$. (Of course, $w$ needs to scale with $q$.)
