How to determine the 'significance' of a new observation based on previous observations

Question

Problem: I have carried out a series of biological experiments where the output of the experiment is a N x N matrix of counts. I then created a custom distance metric that takes in two rows of counts and calculates the 'difference' between them (I will call this difference metric D). I calculated D for all pairwise comparisons and now have an array of difference metrics D called D_array.

My assumption based on biology is that the majority of D in D_array represent that there is no significant difference between the two rows of counts and only the >= 95% interval of D metrics actually represent real differences between two rows of counts. Let us assume that this is true, even if it doesn't make sense.

So this means if D_array = [0, 1, 2, 3, 4 ... 99] (100 metrics) then only a D score of 95-99 are actually representative of a real difference between two rows of counts.

Note: D_array is not representative of my data. My actual data actually has a distribution of values like this (black line represents the mean): https://i.stack.imgur.com/XFFY4.jpg

Given D_array I want to be able to determine whether a newly calculated distance value D' is "significant" based on my previous data: the distribution of my D_array. Ideally, I would like to provide some sort of metric of 'significance' such as a p-value. By significance I mean the probability / significance of having gotten a result as extreme as D'.

After a bit of reading, I found that I can use bootstrapping to calculate a 95% confidence interval for D_array, and then essentially ask if D' is outside of the 95% CI range. However, I am unsure if there is a way to determine how significant having obtained a value of D' is based on D_array.

My questions are:

1. Does asking if D' is outside of the 95% CI of bootstrapped D_array in order to determine whether D' represents a 'real' difference between two rows of counts make sense?

2. Given D' and D_array how can I determine the significance of having gotten a value as extreme as D' as a result. I have seen bootstrapping used to calculate P-values, but this usually requires the mean of two different distributions which I do not have in this case.

3. Is there a better way to determine whether a new observation is 'significantly' different from my prior distribution of 'null' (D_array) data. If so, how?

Answer 1

Given D_array I want to be able to determine whether a newly calculated distance value D' is "significant" based on my previous data: the distribution of my D_array. Ideally, I would like to provide some sort of metric of 'significance' such as a p-value. By significance I mean the probability / significance of having gotten a result as extreme as D'.

To what might be a good enough approximation, you already have what you need. You display an empirical distribution of D values. If you think that an observation in the top 5% of that distribution is "significant," you could simply use the 95th percentile of the empirical cumulative distribution function as your cutoff.

I found that I can use bootstrapping to calculate a 95% confidence interval for D_array, and then essentially ask if D' is outside of the 95% CI range.

Bootstrapping can be a useful way to get the sampling distribution of a statistic calculated from a data sample. Bootstrapping doesn't really give you a 95% confidence interval (CI) for the entire D_array. As you do bootstrap sampling with replacement, the average distribution among the bootstrap samples should be the same as the original D_array. You get the confidence interval for a statistic by doing multiple bootstrap samples, calculating the statistic on each of them, and examining the distribution of the statistic among the bootstrap samples. This answer discusses relative advantages of different ways to do that.

Perhaps you are interested in a 95% confidence interval for the 95th percentile of the population distribution from which you sampled your D_array. Bootstrapping probably won't help much with that. Quoting from that answer:

The first, general, issue is how well the empirical distribution $\hat F$ represents the population distribution $F$. If it doesn't, then no bootstrapping method will be reliable. In particular, bootstrapping to determine anything close to extreme values of a distribution can be unreliable. This issue is discussed elsewhere on this site, for example here and here. The few, discrete, values available in the tails of $\hat F$ for any particular sample might not represent the tails of a continuous $F$ very well.

One way to proceed would be to base your analysis on an model of the distribution of your D_array. Your data look like they might be from a log-normal distribution, which sometimes works well for calculations based on count data. If a log-normal distribution is appropriate, you might be better off fitting such a distribution to your data and basing your cutoff on the upper 5% tail of the fitted distribution.

Finally, I fear that you are putting too much emphasis on "statistical significance" rather than practical significance. What is significant for your application? How will you be using the cases that you identify as "significant" versus "non-significant"? Are you more interested in avoiding false-positive or false-negative findings?* For example, using a cutoff based on the upper 95% CI of the 95th percentile of the distribution would tend to put that tradeoff far toward the side of avoiding false positives. Is that what you really want? Those questions must be answered based on understanding of the subject matter. Once you answer those fundamental questions, statistical analysis can point the way to meeting your objectives reliably.

---

*Large-scale biological studies like RNA-seq typically control the false-discovery rate rather than the false-positive (Type I error) rate.