Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 1 down vote favorite
1

I have a dataset, on which I am doing some data mining. My mining method has certain parameters which I can fiddle with, presumably leading to better or worse quality results. I measure the quality of these results with some statistical tests, which gives me a low P-value for "good" results.

This generates a lot of P-values over a large range. I want to visualize how the P-value changes in response to my parameter. Currently I do this by plotting the P-value over the parameter, however the plot doesn't come out smooth at all. For my personal use, I tried "smoothing" by doing a moving average of the P-values. I know this is a nonsense method but it helps me see the general trend.

I have two questions:

  1. What is the "right" way to smooth such a curve, ie. a curve generated by a function f(x)=p which returns a P-value corresponding to each parameter x?
  2. Does the answer to 1 change if I was plotting the log of P-values, rather than the values themselves?
share|improve this question
1  
Hi there, could you let us know how many p-values you're talking about and the research question that has led to this technique? – Michelle Mar 9 '12 at 20:26
1. I don't think there is any single right way. But you seem to think that doing the smoothing at all is wrong :-) I don't see how it could be wrong, unless your application of it ends up misleading your readers somehow. 2. What specifically do you mean by "what should I do"? – rolando2 Mar 10 '12 at 1:32
I cite some papers in my tables post on the blog that talk about visualizing simulation studies. Although your use case is unique compared to them, perhaps they would give some motivation (or at least useful advice about how to present results). – Andy W Mar 10 '12 at 21:59

1 Answer

up vote 2 down vote

When you have many p-values, QQ plots of -log10 of them are useful. In R you can make these with e.g.

madeup.pvalues <- runif(10000,0,1)

plot( x=-log10(ppoints(10000)), 
      y=-log10(sort(madeup.pvalues)),
      xlab="Expected (-log10)", ylab="Observed (-log10)") 
abline(0,1,lty=45)

More deviation from the dashed line indicates more deviation from $U(0,1)$ p-values. In your example, you could try overlaying results from different choices of your tuning parameter in different colors, to see which one looks "best".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.