4
$\begingroup$

I have 50 p-values. I want to show the mean of these 50 and the standard deviation:

0.06 ± 0.19

The standard deviation is quite large, because although almost all of the time, the p-value is close to 0, occasionally, there is a large value, close to 1.

0.06 ± 0.19

doesn't seem quite right though, because it seems to imply that the p-value could drop below zero. Is there a better way to state the mean and standard deviation in this situation?

Example of p-values:

[0.00001,0.03,0.0007,0.1,0.00005,0.78 ...]

More info:

The p-values come from testing a correlation between 2 variables in a simulation I have written. There are a few elements of randomness and so even if the variables are actually correlated, the results can sometimes show no correlation and hence I get a p-value close to 1.

Due to this random nature, I run the simulation 50 times and then I know I am getting a more reliable p-value. I then want to say something about the spread, which is where this question came from.

|cite|improve this question
$\endgroup$
  • 2
    $\begingroup$ Why are you trying to construct confidence intervals on $p$-values? $p$-values are known and don't have to be estimated so confidence intervals don't apply. Are you instead trying to estimate unknown probabilities? $\endgroup$ – dsaxton Mar 2 '16 at 15:17
  • 1
    $\begingroup$ @dsaxton: Where does OP talk about confidence intervals? Perhaps they simply want to describe a bunch of p-values from various experiments; providing mean and SD could seem as a reasonable idea. $\endgroup$ – amoeba says Reinstate Monica Mar 2 '16 at 15:45
  • 1
    $\begingroup$ Perhaps a mean and standard deviation of the logarithms of p-values? $\endgroup$ – amoeba says Reinstate Monica Mar 2 '16 at 16:04
  • 3
    $\begingroup$ You have a bunch of numbers in [0, 1] and want to describe them. How about a plot of some sort, e.g. histogram or kernel density estimate? Or, if you prefer giving numbers, how about some quantiles? $\endgroup$ – Adrian Mar 2 '16 at 16:34
  • 4
    $\begingroup$ It would help if you could provide a bit more information on where these "p-values" come from. If these are p-values of statistical significance tests, then averaging them doesn't serve much of a useful purpose that I can see off-hand. If they are instead ratios, say, of successes to total attempts at some task, then there may be better ways to summarize and display results. $\endgroup$ – EdM Mar 2 '16 at 16:54
3
$\begingroup$

Given that your interest is in the correlations between variables among your simulations, you would do yourself and your audience a better service by displaying the values of the correlation coefficients rather than the p-values derived from them. The p-values you have are presumably based upon an assumption of bivariate normality with zero correlation under the standard null hypothesis, which might not be met by the processes that you are simulating, and the p-values depend on the number of data pairs examined.

Plot a histogram or a kernel density plot of the correlations for a large number of simulations (say 1000 or so). That plot will nicely show what might be expected of random variability in your simulation scheme, and you could even use it to estimate confidence intervals (or p-values) for the correlations, based on your own simulated process rather than on the assumption of bivariate normality. You could use that approach to examine how the distribution of correlations will change depending on the assumptions of your simulation. This will be much more informative than reporting p-value distributions.

|cite|improve this answer
$\endgroup$
0
$\begingroup$

If your 50 $p$-values are $p=(p_1,p_2,...,p_{50})$, is sd$(p)=0.19$? If so, this is incorrect. As you are reporting the mean of a sample of $p$-values you need to report the SD of the mean. Thus you should report $$0.06\pm \frac{0.19}{\sqrt{50}}=0.06\pm 0.027$$

|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ -1. Why is this "incorrect" and why does one "need" to report the SEM (standard error of the mean)? Quite often one does want to report mean±SD and not mean±SEM. Each of them has its own usage and purpose. $\endgroup$ – amoeba says Reinstate Monica Mar 2 '16 at 21:14
  • $\begingroup$ One cannot do mean $\pm$ SD. That is simply incorrect, since that supports that idea that there error in the estimate for the mean is SD, which it is not. $\endgroup$ – Greenparker Mar 2 '16 at 21:52
  • $\begingroup$ @Greenparker I replied here. $\endgroup$ – amoeba says Reinstate Monica Mar 2 '16 at 21:56
  • $\begingroup$ @amoeba Yes, I should have been more clear in my answer. It is incorrect because they are reporting the mean. If they wish to use simple SD, they should choose one p-value at random from the 50 they computed and report the SD with that as the SD applies to a single random variable, not the mean of random variables. $\endgroup$ – MaximumLikelihood Mar 7 '16 at 15:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.