How can I control the false positives rate?

Question

I feel I'm pretty new to this, since some time passed since my last statistics assignment, so please bear with me.

I am analyzing results of a biological experiment. Basically, I'm looking at a some graph over a genome, where each position in the genome has a value, and I'm looking for local minima (peaks).

Now, I have to set some threshold since relatively high local minima also occur by chance. I can simulate the experiment computationally and get new data, but this is quite resource demanding (I can't run 1000 simulations, perhaps 100 or even only 20).

What I'm currently doing is run some simulations; for each simulation: find all local minima, build a CDF for the local minima values. I then average all the simulation CDFs over all simulations so I have one 'average' CDF (CDF_simulations) which is supposed to show how would local minima distribute if everything in my genome is random.

I do the same for the real data: find minima and build CDFs for their values, so I now have two CDFs - one for the real data and one for the average of the simulations.

I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is < 10%. I report all minima in the real data with value < x as "true".

I think this method should get me to a FP rate of 10%.

• Does this make sense?

• What is this method called and where can I find more about it?

• How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) > CDF_realdata(x).

• Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?

I think this is quite common, and the name FDR also pops to mind, but when reading about FDR I couldn't exactly figure how to apply it to my situation.

Any comments and references (hopefully user-friendly ones) will be appreciated!

Thanks Dave

Answer 1

• Does this make sense: To me, mostly yes... although I think you might be doing something I don't expect (see below).
• What is this method called and where can I find more about it: You are building up an empirical reference distribution through permutation of your genome labels. There may be fancier terms. I don't know what a good citation might be, consider: Good, P. (2005) Permutation, Parametric, and Bootstrap Tests of Hypotheses, Springer-Verlag, NY, 3rd edition.
• How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) > CDF_realdata(x): This is the part that makes less sense to me. I'm not sure what you are doing here exactly. Maybe the thing to do is to find the 90th percentile for the CDF_simulations and use that as your cutoff for saying there might be something interesting going on in CDF_realdata?
• Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?: The number of simulations you run will produce a larger and more reliable reference distribution. Your averaged CDF approach seems a little odd to me.