Are 25 participants enough to evaluate P value?

Question

I read plenty of social science research, and I have a constant scepticism towards the statistics behind social science research. Therefore, I have some specific questions.

Can you trust the P-value of a ANOVA test with 25 participants? The results of my specific example is: (F(1, 1148) = 13.25, *p* < 0.05).

Is this enough information to evaluate the "correctness" of the statistics behind this research? In my mind I see p is less than alpha .05 which means "yay, a finding".

Somehow the research papers (and plenty like this example) kind of "p-hacks" towards a statistical significant result, just to get it published, and I am annoyingly sceptic towards how correct the work actually is.

In addition to the information given in this example, what other parameters or numbers should I as a reader get from the author to review the correctness of this work? Or is it sufficient?

Answer 1

Your question is all over the place. As far as the title: the simple answer is yes.

I would offer a different perspective from Wes' answer above. As an historical note, Fisher's famous Lady Tasting Tea experiment involved an $n$ of 8 cups of tea, half of which prepared in a manner different from the other. This type of experiment was highly amenable to analysis by Fisher's exact test which, unlike the Pearson chi-square test, did not consider the approximate sampling distribution of a test statistic, but instead considered each possible permutation of the contingency table, and produced a $p$-value as a proportion of permutations which were equally or more inconsistent under the null hypothesis that Lady Bristol could not discern either tea.

That having been said, I cannot rationalize the denominator degrees of freedom in your ANOVA. Is this a repeated measures design? N=25 participants should not produce such a large DF without some form of replication and random effects.

I think you have many reasons to be dubious of statistics in social research. "p-hacking" is an accusatory term that aims to throw statistics and statisticians under the bus as a fault of bad science. You can definitely "attempt to slice an analysis many ways" (multiple testing without correction), or identify subgroups, the statistical rationale for these approaches is really symptomatic of a greater problem: not identifying a scientific problem of interest. If the authors cannot clearly state a hypothesis and carry out an experiment or analysis which directly answers the question of interest, the fault is not a statistical issue, but a scientific one.

Answer 2

Well the short answer is yes, you can calculate the p.value for your estimate...

However, the main problem with using such a small sample is that it would be difficult to generalise such a finding to your given target population, assuming it is larger and more diverse than your sample of 25 individuals. If random sampling were used, how likely would it be for the sample to be representative of your target population? One outlier or uncommon observation could have a significant impact upon your estimates, or a subgroup may be over represented. This needs to be considered throughout the study.

If you are estimating parameters of your model using OLS, a larger sample size generally provides an estimate closer to the true value of the parameter of interest. This is due to the consistency of the OLS estimator.

If $$n_{i}$$ represents the number of observations in a sample and

$$n_{1}<n_{2}<n_{3}$$

(Wooldridge, 2006)

You may get an accurate estimate for the sample of 25, but whether that sample it representative of your target population is another matter. It should be noted that the OLS estimator is not biased in cases where a small sample is present. It is the precision of the estimate which increases with the sample size. Though there is no guarantee that the true parameter lies within your given confidence interval for an individual sample, this increase in precision is useful in justifying subsequent inference.

There is also much debate as to the usefulness of p.values in general, and in many cases it is preferable to present confidence intervals as well/instead (Sterne and Davey Smith, 2001). Such statistics should be interpreted within the context of previous theory and study design elements. Essentially, use it as part of your interpretation, but not necessarily as the basis of your interpretation.

I hope that is of some help :)

References

Sterne J, Davey Smith G., 2001, “Sifting the evidence - what's wrong with significance tests?” Physical Therapy vol.81, pp.1464-1469 (reprint of article 249) and the American Statistical Association statement and the 21 supporting papers)

Wooldridge, J. M. 2006. Introductory econometrics: a modern approach. Mason, OH, Thomson/South-Western.