# regularization for categories with few samples

I have about 30000 observations of a variable $Y$ for different categories of a treatment $X$. Some treatments have over 500 samples and it's very reasonable to take just these cases ($P(Y|X) \simeq freq(Y|X)$), other potential treatments have never been seen, and thus it is reasonable to apply all the data (or the mean over all seen categories, $P(Y) \simeq freq(\mathbb E[Y|X])$. What I'm wondering is how to address those categories that have very few samples (many have just 1).

Ideally I'd like to have a parameter that allows me to interpolate between the observed distribution for the entire sample and the distribution for the sample of a given treatment, which places more wight in the latter the more samples we have.

I need to interpolate the distributions (not just the means) because I haven't got a good model for the distributions of $X$ (they're certainly not Gaussian, Poisson or anything I can identify), and I need to keep track of confidence intervals.

This problem reminds me a bit of the settings for the Krichevsky-Trofimov estimator, where symbols are to be predicted, some of which might have not been observed, or observed with very low counts. The solution there is to add a-priori a low (possibly fractional) count to any symbol before starting the observation-based counting.

In the question, you (implicitly) describe a "neutral" distribution over the data - say $E$ - which could be defined in at least two ways:

1. The empirical distribution of $Y$.

2. The empirical distribution of the mean of $Y|x$ per category $x$, taken over all $x$.

The choice of how to determine $E$ depends on the settings, but let's say we have such a neutral distribution $E$.

Set $q > 0$, and find a sample $E'$ obtained by the boundaries of $q$ quantiles of $E$. (Note that for larger $q$, $E' \sim E$.) Set another positive $w$. For each category $x$ of $X$, approximate the distribution as the distribution obtained by the union of $E'$ and $w$ copies of $Y | x$.

So, for example if $q = 10$, $w=1$, then:

• for an $x$ that hasn't been observed, the distribution will be exactly that of the neutral distribution approximation - the 10 points of $E'$

• for an $x$ that has been observed once, the distribution will be almost that of the neutral distribution approximation - the 10 points of $E'$, and the single point of $Y | x$.

• for an $x$ that has been observed 500 times, the distribution will be very similar to $Y | x$ - the 10 points of $E'$, and the 500 points of $Y | x$.

The parameter $w$ is a tradeoff: higher values will give tighter confidence intervals for both large categories and low categories - only you can set it (or at least you need to define an objective). The parameter $q$ is mainly for computational shortcuts. Without computational restrictions, $q$ can be made the number of samples determining $E$. (Of course, $w$ needs to scale with $q$.)

• thanks for the hint. It sounds reasonable. Do you have a reference to read up more on this? Oct 13, 2016 at 9:40
• @AlexMonras Krichevsky & Trofimov's articles seem to be accessible through IEEE paysites. The closest accessible material I could find was on Good Turing smoothing, which shares similar ideas. Please note that, irrespectively, there's something of a leap by applying the idea from symbol frequency estimation, to conditional distribution estimation. I would say that this is an "inspired by" heuristic. Oct 13, 2016 at 10:01