How can I control the false positives rate?

I feel I'm pretty new to this, since some time passed since my last statistics assignment, so please bear with me.

I am analyzing results of a biological experiment. Basically, I'm looking at a some graph over a genome, where each position in the genome has a value, and I'm looking for local minima (peaks).

Now, I have to set some threshold since relatively high local minima also occur by chance. I can simulate the experiment computationally and get new data, but this is quite resource demanding (I can't run 1000 simulations, perhaps 100 or even only 20).

What I'm currently doing is run some simulations; for each simulation: find all local minima, build a CDF for the local minima values. I then average all the simulation CDFs over all simulations so I have one 'average' CDF (CDF_simulations) which is supposed to show how would local minima distribute if everything in my genome is random.

I do the same for the real data: find minima and build CDFs for their values, so I now have two CDFs - one for the real data and one for the average of the simulations.

I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is < 10%. I report all minima in the real data with value < x as "true".

I think this method should get me to a FP rate of 10%.

• Does this make sense?

• What is this method called and where can I find more about it?

• How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) > CDF_realdata(x).

• Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?

I think this is quite common, and the name FDR also pops to mind, but when reading about FDR I couldn't exactly figure how to apply it to my situation.

Any comments and references (hopefully user-friendly ones) will be appreciated!

Thanks Dave

• Probably I didn't understand something, but can you please clarify how can you simulate "real" CDFs? (I am assuming you are doing a real life data analysis, as you mention the analysis over a genome) Nov 2, 2010 at 20:01
• @suncoolsu: I guess it was a bad choice of words. I don't simulate real data, I simulate the experiment under the assumption the genome has nothing special in it. Nov 2, 2010 at 20:12
• It makes sense. I wonder whether it would help to consider the magnitudes of the local minima along with their values (where the "magnitude" is the amount by which its value differs from neighboring local maxima). Regardless, start by reading about resampling statistics on Wikipedia: en.wikipedia.org/wiki/Resampling_%28statistics%29 .
– whuber
Nov 5, 2010 at 21:23
• Can you describe "I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is < 10%. I report all minima in the real data with value < x as 'true'" more precisely? Feb 12, 2013 at 23:12