Consider the following situation: An experiment was done in 15 different conditions, and a value of the parameter 'A' was measured in each experiment (A can have any integer value between 0 and infinity). 3 repetitions were done in each condition, so that there are a total of 45 measurements of 'A'.
I want an estimate for the probability density function to measure a certain value of 'A'.
The first thing I tried was to average the 3 repetitions in each of the 15 different conditions, and do Kernel density estimation (KDE) over the 15 data points with a Gaussian kernel (I use a Matlab function that automatically chooses the optimal bandwidth for the kernel).
Then I realized that in different conditions the variance between repetitions is different, and I believe that this variance is important information that should be taken into account. The following plot shows this situation: the blue line is the KDE, and the points are the measurements of 'A' in each condition, averaged over 3 repetitions. The error bars indicate the standard deviation between the 3 repetitions in each condition (The hight of the points don't convey any information, they are just spread in the y axis so their error bars don't intersect).
I think this graph shows an example where the variance of each data point has important information that the KDE misses (I think there should be one small peak instead of two, considering the large variances of the points in the two smaller peaks).
My question: what is the best way to estimate the probability distribution function in this situation, taking into account the variance of each data point?
Solutions I considered:
- should I choose different kernel bandwidth for each point, larger bandwidth for larger variance?
- should I just do KDE on all 45 data points (15 conditions with 3 repetitions each)?
- should I use a different method, other than KDE?
**Edit: I'm adding some details about the data: This is gene expression data in bacteria (RNA-seq). If you know nothing about this, just know that gene expression is the how many of a certain type of molecule (that is related to a certain gene) are present in a population of bacteria. Each experiment starts with a small number of bacteria, that are then grown in one of 15 conditions (different PH levels, different nutrition etc). After the bacteria divide and grow to large numbers, we measure the gene expression in the population for every gene (the graph I'm showing is for one gene). For every condition we do 3 repetition, mean that 3 small groups of bacteria are grown separately in the same condition. Another consideration: the variation of gene expression between repetitions of the same condition is not expected to be Gaussian. We know that some genes in some conditions it's a Poisson distribution. For some it is not at all Poisson, and sadly for us the standard deviation is much larger than the mean. My objective was to find out if I can define sates of gene expression that can be considered "on" or "off". This is why I tried KDE.
I'm new to Kernel density estimation.