# Paired t-test with small standard error

I am comparing two algorithms applied to all frames of a video clip. To evaluate the result I am doing a paired t-test analysis, i.e., I am subtracting the algorithm scores for each frame and then doing a regular t-test on the resulting diff population.

My problem is that because the number of frames is large (500+) my standard error, s / sqrt(n), is so small that even when I run the same algorithm on the video twice, my statistical analysis gives me the result that mu != 0 although the x_bar of the difference are in the order of 10^-2 (score range is [0, 1] and thus the diff range is [-1, 1])

What is the best way to blunt the analysis? Random selection from the population?

• To me, for a large dataset, measuring effect size seems more important than hypothesis testing for the kind of reasons you provide. You could do a bootstrap procedure to estimate confidence interval around your x_bar. But I see an other problem here. Don't you have non independent data points ? I mean if an algorithm is better on a frame, I guess he has good chances to be better on the adjacent frame, no ? If so, it violates t-test assumptions. Do you have access to other clips or can you cut your video clip in various scenes ? Jun 23, 2016 at 9:04
• Could you please explain what you mean with measuring effect size? Yes, I am aware that my datapoints are probably not independent but I am at my wits end with how to do this analysis. I have access to a set of clips that spout various characteristics, but I was planning on using each of them on their own to ensure the algorithm copes with various scenarios. Jun 23, 2016 at 10:08

To follow up on the comments, classicaly, for a large dataset, measuring effect size tends to be important. By effect size I meant measuring how much algo A is better than B. You can look up for example what is cohen's d and compute its confidence interval using boot package with r (or bootES which seems dedicated to this task). For large dataset, significance is easily reached with small effect size so this need to be checked to complement your analysis.