5
$\begingroup$

I feel I'm pretty new to this, since some time passed since my last statistics assignment, so please bear with me.

I am analyzing results of a biological experiment. Basically, I'm looking at a some graph over a genome, where each position in the genome has a value, and I'm looking for local minima (peaks).

Now, I have to set some threshold since relatively high local minima also occur by chance. I can simulate the experiment computationally and get new data, but this is quite resource demanding (I can't run 1000 simulations, perhaps 100 or even only 20).

What I'm currently doing is run some simulations; for each simulation: find all local minima, build a CDF for the local minima values. I then average all the simulation CDFs over all simulations so I have one 'average' CDF (CDF_simulations) which is supposed to show how would local minima distribute if everything in my genome is random.

I do the same for the real data: find minima and build CDFs for their values, so I now have two CDFs - one for the real data and one for the average of the simulations.

I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is < 10%. I report all minima in the real data with value < x as "true".

I think this method should get me to a FP rate of 10%.

  • Does this make sense?

  • What is this method called and where can I find more about it?

  • How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) > CDF_realdata(x).

  • Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?

I think this is quite common, and the name FDR also pops to mind, but when reading about FDR I couldn't exactly figure how to apply it to my situation.

Any comments and references (hopefully user-friendly ones) will be appreciated!

Thanks Dave

Improve this question
$\endgroup$
4
  • $\begingroup$ Probably I didn't understand something, but can you please clarify how can you simulate "real" CDFs? (I am assuming you are doing a real life data analysis, as you mention the analysis over a genome) $\endgroup$
    – suncoolsu
    Nov 2, 2010 at 20:01
  • $\begingroup$ @suncoolsu: I guess it was a bad choice of words. I don't simulate real data, I simulate the experiment under the assumption the genome has nothing special in it. $\endgroup$
    – David B
    Nov 2, 2010 at 20:12
  • $\begingroup$ It makes sense. I wonder whether it would help to consider the magnitudes of the local minima along with their values (where the "magnitude" is the amount by which its value differs from neighboring local maxima). Regardless, start by reading about resampling statistics on Wikipedia: en.wikipedia.org/wiki/Resampling_%28statistics%29 . $\endgroup$
    – whuber ♦
    Nov 5, 2010 at 21:23
  • $\begingroup$ Can you describe "I now search for the max x such that CDF_simulations(x) / CDF_realdata(x) is < 10%. I report all minima in the real data with value < x as 'true'" more precisely? $\endgroup$
    – russellpierce
    Feb 12, 2013 at 23:12

1 Answer 1

Reset to default
2
$\begingroup$
  • Does this make sense: To me, mostly yes... although I think you might be doing something I don't expect (see below).
  • What is this method called and where can I find more about it: You are building up an empirical reference distribution through permutation of your genome labels. There may be fancier terms. I don't know what a good citation might be, consider: Good, P. (2005) Permutation, Parametric, and Bootstrap Tests of Hypotheses, Springer-Verlag, NY, 3rd edition.
  • How should I scan the CDFs to find the right x? Sometimes for low x's CDF_simulations(x) > CDF_realdata(x): This is the part that makes less sense to me. I'm not sure what you are doing here exactly. Maybe the thing to do is to find the 90th percentile for the CDF_simulations and use that as your cutoff for saying there might be something interesting going on in CDF_realdata?
  • Where does the number of simulations come into play? Does it make sense to simply build an averaged CDF as I did?: The number of simulations you run will produce a larger and more reliable reference distribution. Your averaged CDF approach seems a little odd to me.
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.