# Kernel density estimation (KDE) for data points with different variance

Consider the following situation: An experiment was done in 15 different conditions, and a value of the parameter 'A' was measured in each experiment (A can have any integer value between 0 and infinity). 3 repetitions were done in each condition, so that there are a total of 45 measurements of 'A'.

I want an estimate for the probability density function to measure a certain value of 'A'.

The first thing I tried was to average the 3 repetitions in each of the 15 different conditions, and do Kernel density estimation (KDE) over the 15 data points with a Gaussian kernel (I use a Matlab function that automatically chooses the optimal bandwidth for the kernel).

Then I realized that in different conditions the variance between repetitions is different, and I believe that this variance is important information that should be taken into account. The following plot shows this situation: the blue line is the KDE, and the points are the measurements of 'A' in each condition, averaged over 3 repetitions. The error bars indicate the standard deviation between the 3 repetitions in each condition (The hight of the points don't convey any information, they are just spread in the y axis so their error bars don't intersect). I think this graph shows an example where the variance of each data point has important information that the KDE misses (I think there should be one small peak instead of two, considering the large variances of the points in the two smaller peaks).

My question: what is the best way to estimate the probability distribution function in this situation, taking into account the variance of each data point?

Solutions I considered:

1. should I choose different kernel bandwidth for each point, larger bandwidth for larger variance?
2. should I just do KDE on all 45 data points (15 conditions with 3 repetitions each)?
3. should I use a different method, other than KDE?

**Edit: I'm adding some details about the data: This is gene expression data in bacteria (RNA-seq). If you know nothing about this, just know that gene expression is the how many of a certain type of molecule (that is related to a certain gene) are present in a population of bacteria. Each experiment starts with a small number of bacteria, that are then grown in one of 15 conditions (different PH levels, different nutrition etc). After the bacteria divide and grow to large numbers, we measure the gene expression in the population for every gene (the graph I'm showing is for one gene). For every condition we do 3 repetition, mean that 3 small groups of bacteria are grown separately in the same condition. Another consideration: the variation of gene expression between repetitions of the same condition is not expected to be Gaussian. We know that some genes in some conditions it's a Poisson distribution. For some it is not at all Poisson, and sadly for us the standard deviation is much larger than the mean. My objective was to find out if I can define sates of gene expression that can be considered "on" or "off". This is why I tried KDE.

I'm new to Kernel density estimation.

• Why do you want to use KDE in here instead of mixture of Gaussians? – Tim Feb 18 at 13:46
• I don't want to assume the number of Gaussian in my data.... I want the estimation to tell me how many peaks are there in the probability density function... Even if I do use mixture of Gaussians, should I use it on all 45 data points, ignoring the fact that they actually came from 15 conditions? – Adi Ro Feb 18 at 14:48
• OK, but this can be done by model selection. – Tim Feb 18 at 14:49
• @Tim is right about model selection. So perhaps you mean to ask: what is the best model selection method for 1D data in which we expect 10-20 Gaussians over a specifiable range of means [0-700] and a specifiable range of variances [10-1000]? – Peter Leopold Feb 18 at 15:44
• In my experience KDE is good for finding critical points: maxes, mins and inflection points. You can get those on one pass -- but only AFTER you've come up with a "suitable" bandwidth. The density profile itself from basic KDE has bias proportional to the second curvature of the density profile, so bias correction involves using KDE multiple times to estimate derivatives of densities. Once you go down the KDE hole, expect to be there for a while. – Peter Leopold Feb 18 at 15:51