1
$\begingroup$

I read plenty of social science research, and I have a constant scepticism towards the statistics behind social science research. Therefore, I have some specific questions.

Can you trust the P-value of a ANOVA test with 25 participants? The results of my specific example is: (F(1, 1148) = 13.25, *p* < 0.05).

Is this enough information to evaluate the "correctness" of the statistics behind this research? In my mind I see p is less than alpha .05 which means "yay, a finding".

Somehow the research papers (and plenty like this example) kind of "p-hacks" towards a statistical significant result, just to get it published, and I am annoyingly sceptic towards how correct the work actually is.

In addition to the information given in this example, what other parameters or numbers should I as a reader get from the author to review the correctness of this work? Or is it sufficient?

$\endgroup$
  • $\begingroup$ I have subjective Bayesian friend who gets $p$-values without any data. $\endgroup$ – usεr11852 Apr 2 at 22:17
1
$\begingroup$

Your question is all over the place. As far as the title: the simple answer is yes.

I would offer a different perspective from Wes' answer above. As an historical note, Fisher's famous Lady Tasting Tea experiment involved an $n$ of 8 cups of tea, half of which prepared in a manner different from the other. This type of experiment was highly amenable to analysis by Fisher's exact test which, unlike the Pearson chi-square test, did not consider the approximate sampling distribution of a test statistic, but instead considered each possible permutation of the contingency table, and produced a $p$-value as a proportion of permutations which were equally or more inconsistent under the null hypothesis that Lady Bristol could not discern either tea.

That having been said, I cannot rationalize the denominator degrees of freedom in your ANOVA. Is this a repeated measures design? N=25 participants should not produce such a large DF without some form of replication and random effects.

I think you have many reasons to be dubious of statistics in social research. "p-hacking" is an accusatory term that aims to throw statistics and statisticians under the bus as a fault of bad science. You can definitely "attempt to slice an analysis many ways" (multiple testing without correction), or identify subgroups, the statistical rationale for these approaches is really symptomatic of a greater problem: not identifying a scientific problem of interest. If the authors cannot clearly state a hypothesis and carry out an experiment or analysis which directly answers the question of interest, the fault is not a statistical issue, but a scientific one.

$\endgroup$
  • $\begingroup$ I appreciate the historical note, thank you. I will be less worried of n. For your question regarding repeated measure design, yes it is. All participants are tested in two configurations. First without manipulation, then with. I am curious, however, why will the denominator degree of freedom be impacted by it being a repeated measures design? Thank you, a nice clarification that it is a scientific limitation, not a statistical issue. $\endgroup$ – jeyss Nov 24 '16 at 9:31
5
$\begingroup$

Well the short answer is yes, you can calculate the p.value for your estimate...

However, the main problem with using such a small sample is that it would be difficult to generalise such a finding to your given target population, assuming it is larger and more diverse than your sample of 25 individuals. If random sampling were used, how likely would it be for the sample to be representative of your target population? One outlier or uncommon observation could have a significant impact upon your estimates, or a subgroup may be over represented. This needs to be considered throughout the study.

If you are estimating parameters of your model using OLS, a larger sample size generally provides an estimate closer to the true value of the parameter of interest. This is due to the consistency of the OLS estimator.

If $$n_{i}$$ represents the number of observations in a sample and

$$n_{1}<n_{2}<n_{3}$$

enter image description here

(Wooldridge, 2006)

You may get an accurate estimate for the sample of 25, but whether that sample it representative of your target population is another matter. It should be noted that the OLS estimator is not biased in cases where a small sample is present. It is the precision of the estimate which increases with the sample size. Though there is no guarantee that the true parameter lies within your given confidence interval for an individual sample, this increase in precision is useful in justifying subsequent inference.

There is also much debate as to the usefulness of p.values in general, and in many cases it is preferable to present confidence intervals as well/instead (Sterne and Davey Smith, 2001). Such statistics should be interpreted within the context of previous theory and study design elements. Essentially, use it as part of your interpretation, but not necessarily as the basis of your interpretation.

I hope that is of some help :)

References

Sterne J, Davey Smith G., 2001, “Sifting the evidence - what's wrong with significance tests?” Physical Therapy vol.81, pp.1464-1469 (reprint of article 249) and the American Statistical Association statement and the 21 supporting papers)

Wooldridge, J. M. 2006. Introductory econometrics: a modern approach. Mason, OH, Thomson/South-Western.

$\endgroup$
  • 1
    $\begingroup$ couple of points: you should clarify in what sense results are "difficult to generalize to a population of interest": specifically you are referring to precision (and not bias) of those estimates. It's true that an underpowered study renders imprecise estimates. I think the virtue of the $p$-value is more evident in small samples, as the focus of discussion becomes strength of evidence and its inconsistency with a null hypothesis. $\endgroup$ – AdamO Nov 23 '16 at 17:05
  • $\begingroup$ Thanks for the input. Support for p.values really seems to vary, and it is true that the p.value can be useful in small studies (giving you an idea of where to focus further research, etc.) Probably the most important thing is just to understand where it comes from and its limitations. There is a reason we use them after all :) $\endgroup$ – Wes Nov 23 '16 at 17:25
  • 1
    $\begingroup$ Thank you, Wes! Your distinction between precision and bias when evaluating the P-value was good! I will check out your references. In my own research as well, I will strive more towards the confidence interval instead of the P-value. $\endgroup$ – jeyss Nov 24 '16 at 9:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.