I am supporting a psychology experiment, and having problems analyzing some of the data.
By way of background, I’m a programmer at a research organization who’s taken several stat courses recently. A relatively junior researcher (Ph.D. candidate) asked me if I would help her analyze some data in preparation for an internal review.
The experiment involves measuring pupil diameter as a function of cognitive load.
In a nutshell, a person sits in front of a computer screen for about 20 minute reading. From time to time the subject is given a task, such as performing a mathematical calculation, that is meant to induce cognitive load. Sensors at the bottom of the computer monitor measure pupil dilation on the order of every few milliseconds. The idea is that under cognitive load, pupils have been shown to dilate.
The overall 20-minute time periods are divided into four sections, each of which has a different numbers of tasks. A representative 20-minute session looks like this:
Section A B C D # Observations 13908 20416 19635 23269 # Tasks 0 11 28 6
At the end of a session, the data might look something like this:
Section A B C D # Observations 13908 20416 19635 23269 # Tasks 0 11 28 6 Mean Dilation 4.073 4.195 4.216 4.118 StdDev 0.202 0.254 0.256 0.239
So the task is to determine if there are significant differences between the mean dilations across the sections, and I was asked to run one-way ANOVAs on the data.
In R I did the following on each data set:
df$section <- as.factor(df$section) testANOVA <- oneway.test(dilation ~ section, data = df) testANOVA
I got outputs like the following:
One-way analysis of means (not assuming equal variances) data: df$dilation and df$section F = 1436.55, num df = 3.00, denom df = 41130.44, p-value < 2.2e-16
I believe 2.2e-16 is the lowest p-value that R can give, and it did not seem right to me. I basically interpreted this as, “There is almost no possible way that the null hypothesis could be true.”
I'm relatively new to statistics and haven’t studied ANOVAs yet, but in just looking at the means and standard deviations of all the data sets it does not seem reasonable that the null hypothesis would be so strongly rejected across almost the entirety of the data sets.
The researcher told me early on not to worry about assumptions, but given those results, I started digging into the assumptions of a one-way ANOVA.
These are some of the issues I came up with regarding this data:
--Unequal variance. The variances are not equal across the blocks (although my understanding of the oneway.test in R is that it accounts for unequal variance).
--Lack of independence. The data from each segment comes from the same person during the same session.
--Lack of normalcy. Each block exhibits normal histograms when looked at in its entirety, but I started taking random samples of each block (without replacement), and once I got down to below 1,000 observations in a block, it became apparent that some of the blocks were pretty skewed, and in some cases the histogram almost looked like a uniform distribution.
Also of note, the segment sizes are different (greatly so in some cases).
I tried two different approaches to getting some insights: (1) I created simulated data using the real means and standard deviations, but imposing normal distributions; (2) I standardized the size of the segments of the real data by sampling (without replacement).
In both of the above scenarios, I noticed that with very large numbers of observations, I still invariably got very low p-values. If I started reducing the number of observations, the p-values started rising.
In digging deeper, I came across this post in which someone says that with a large number of observations, normality tests are essentially useless:
Which made me wonder if the same holds true for one-way ANOVAs.
I’m obviously way out of my league at this point, and I’m going to tell the researcher that. But for my own edification I have a simple question:
Given the way I’ve described the data, (1) are there ways – via transformations and perhaps non-parametric approaches – to perform ANOVAs to glean some sorts of accurate insights into segment-level differences, or (2) with all of the assumption issues and whatnot, might there be a fundamental flaw in the design of the experiment such that it’s generally producing just noise?
If it's the first, I'd be inclined to keep digging around, just to educate myself. But if it's the latter, I don't really want to spend a whole lot of time on it.