0
$\begingroup$

In my current analysis, I have not found statistical significance to some of my conditions. So I was wondering how to predict or calculate the effect size with more subjects and items. I have 20 subjects and each have taken 120 items. My dependent variable is Reaction Time (RT) and my independent variables are Related (2 levels -- control and experimental) and PrimeType (5 levels). In testing significance, I look at RT significance of Related for each Prime Type level. For example, the significance level for PrimeType 3 when compared in RT between Related yield a p-value = .075

So I wonder what packages or functions that I could use where I put in the current observation numbers and significance level and compare it or calculate it with a bigger sample. I want to check whether the the significance level would change or become statistically significant in a bigger sample size.

Any ideas?

$\endgroup$
  • $\begingroup$ Sequential testing is inflating type 1 error rate. Thus, increasing sample size without correction for multiple testing is just cheating. $\endgroup$ – Michael M Dec 16 '13 at 6:38
  • $\begingroup$ I am trying to see whether the results changes if I have more participants, more like a prediction. In my paper, I will only share the current results and, if possible, mention how the results might or might not change with a bigger sample. As a Statistician, @MichaelMayer, what do you suggest I should do? Would re-sampling or bootstrapping decrease type 1 error rate and have a correct prediction? $\endgroup$ – ama Dec 16 '13 at 15:14
  • $\begingroup$ Resampling or bootstrapping does not directly account for multiplicity. You might get some ideas from Wiki's en.wikipedia.org/wiki/Sequential_analysis (section "Bias") about sequential testing. I hope you will find some useful links or informations about your problem. Another term related to the question is "adaptive design" or "group sequential design". $\endgroup$ – Michael M Dec 16 '13 at 15:28
2
$\begingroup$

The best estimate of the effect size in a larger sample is the effect size in your sample.

A large enough sample size will make any effect size statistically significant.

You can estimate this by recreating your data with the same effect size and a larger N, or by looking at the effect size in a table of p values and seeing how large it has to be in order to be significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.