# T-test for small-sample with outliers and skewed data good enough?

I am conducting a fMRI study in which there are only 8 participants (and 4 control participants). I will be seeking differences in the 8 participants between periods of time, and also seeking differences between the 8 participants and the 4 controls at one particular time-point.

There is quite a lot of variation between participants however, given that I am looking at participants which have a medical condition. The main variables being tested are all ratio data.

I am not sure the best way to approach the data. Ideally, I'd stick with independent t-tests and paired samples t-tests. I have also considered not focusing too much on the statistical tests, but more focus on examining the plotted data - and making general observations from it.

Would this make sense, or would I be better going for a Mann Whitney U test, or applying bootstrapping to the t-tests?

My statistics knowledge isn't great - so I am not sure what to do knowing that my study is underpowered.

• At how many time points are they measured? Twice? Commented Jul 29, 2017 at 23:31
• There are several time points. Although I'll likely only use two. I'm comparing data collected this year, with data collected from the same participants 5 years ago and seeing what the changes are
– user171671
Commented Jul 29, 2017 at 23:44
• And you are also interested in the difference between both groups, right? You can look up the repeated measures ANOVA. I could find a tutorial depending on your statistics software. Commented Jul 30, 2017 at 0:04
• Yeah I've thought about that. But with such a small sample, wouldn't an ANOVA increase type 11 error? I am using SPSS btw
– user171671
Commented Jul 30, 2017 at 0:22
• No. Since you have two groups, ANOVA and t test are equivalent. However, this is different. You'll be using a repeated measures ANOVA, and repeated measures designs are often quite powerful. Moreover, worrying about power now is strange. Also use plots, to guide your work. And don't make claims with too much certainty given the small sample size. Commented Jul 30, 2017 at 0:39

To be a bit blunt, there isn't much you can do by altering statistical tests to make up for the fact that you simply don't have enough data to detect the effect sizes that are likely.

Choosing what statistical tests to use is a decision that should be made based on what types of data you have, what assumptions you can make about the data, and similar.

To kind of envision this imagine the following:

You have a coin. It may be a fair coin or it may be a loaded coin. To tell which it is you flip the coin three times. All times it comes up heads. Do you have enough data to tell if it is a fair coin? Probably not. This is because there is a 12.5% chance of this happening even if the coin is fair.

Bootstrapping won't help here. This is because bootstrapping would be roughly analogous to assuming you had 10 coins that all came up heads by extrapolating the data that came from your three flips. And then running the probabilities. You would get a good P value, but that wouldn't change the fact that you only flipped three coins. And using a different statistical test could only help if the test loaded assumptions that aren't actually right into your analysis.

What bootstrapping is good for is power analysis. What if your study came up with some results that were intriguing, but you don't have a good enough P value or power? How big a sample size would you need to demonstrate those intriguing results (if they hold up with the larger study)? Bootstrapping can help tell you this. This also means that your small sample size experiment may be helpful for estimating what types of effect sizes you may see when you do get the resources to run a properly sized study, and thus estimate what sample size you need.

When you get slightly larger sample sizes you can start to detect very large effect sizes. Imagine your coin was so loaded that it almost always comes up heads. You flip it five times. All times come up heads. There was only a 0.03125 probability of a fair coin doing this, so, your P-value on this test is 0.03125. You have sufficient evidence to conclude that the coin is loaded. But in a case where the effect size was weaker, ie. you got three heads and two tails, you can't tell that the coin is loaded.

Or more formally, let's go back to what Power means. Power is 1 minus the probability of a false negative. power = 1 – β. It is dependent on the effect size, the sample size, and the alpha cutoff you use (ie. the P-Value threshold you use, often 0.05). (Unfortunately, there isn't a good and simple formula for calculating power by hand, which is where methods like bootstrapping, and packages in R and such come into play).

Tldr. You don't have enough data to discover anything but very large effect sizes, but you may collect data that is useful in planning larger studies.

• Thanks for the reply! How would you use boostrapping in a power analysis exactly? Do you have a citations for this? Also, in my analysis, I have already ran t-tests (the ones that have come back sig. have yielded large effects). I don't know if this is acceptable given that there are clear outliers in the data (maybe I should use non-parametric alternatives). Would it be appropriate to report these t-test results - but argue that further research (with a bigger sample size) should further explore it so that real conclusions can be drawn. I am clearly having trouble writing my report.
– user171671
Commented Jul 29, 2017 at 22:40
• I think what I was asking in my last comment was - how do I use my experiment to estimate what types of effect sizes I'd see when a properly sized study is ran (and therefore estimate the correct n)?
– user171671
Commented Jul 29, 2017 at 22:44
• I think this is a good example of how to use a bootstrapping to do a power analysis in R r-bloggers.com/…. (the particulars will depend on what statistics package you are using). See, this paper for SAS lexjansen.com/pharmasug/2005/statisticspharmacokinetics/… This R script is one I created for homework in a college class. It probably wouldn't be what you would want to do (use a premade package for this instead) but it shows the logic here dropbox.com/s/tyn8wsrk9r592lc/poweranalysisexample.R?dl=0 Commented Jul 29, 2017 at 22:54
• Thanks a lot :) I'll take a look. I am actually using SPSS and don't know how exactly to do a power analysis using it
– user171671
Commented Jul 29, 2017 at 22:55
• This blog goes into some of the details of what power means, similar to my initial answer. effectsizefaq.com/category/statistical-power I've done a bit with SPSS, but haven't yet done a power analysis using SPSS, so, I can't help you much there. Commented Jul 29, 2017 at 22:56