Problem I have a computer algorithm, and am attempting to measure the affect of a parameter (NS) on the output of the algorithm (SC). My hypothesis is that if the level of NS is lowered, there should be a corresponding decrease in SC as well. I'm doing all of my statistical analysis in the R language (see below).
I've performed an analysis of variance on my data, and though the results aren't statistically significant, it looks like there is a pattern to the data (that lowering NS does decrease SC). I tried increasing my sample sizes, which does result in more pairwise comparisons becoming statistically significant. However when I do this, the results of Levene's Test show that my variances are no longer homoscedastic, and thus the analysis of variance isn't valid...
If you look at a graph of the mean values of the output (sc) you can see what I'm talking about when I say "it looks like there is a pattern". I've generated different data sets several times and each time, the means of SC are scattered similarly to graph below (despite the pairwise comparisons being statistically insignificant).
What I am Doing
Here is my R code and some output:
t <- read.table("output.dat") names(t) <- c("sc", "ns") leveneTest(t$sc, group=t$ns, center=median) Levene's Test for Homogeneity of Variance (center = median) Df F value Pr(>F) group 24 1.0447 0.4018 124975 Warning message: In leveneTest.default(t$sc, group = t$ne, center = median) : t$ne coerced to factor. t.aov <- aov(t$sc ~ as.factor(t$ns)) summary(t.aov) Df Sum Sq Mean Sq F value Pr(>F) as.factor(t$ne) 24 1448 60.32 5.488 <2e-16 *** Residuals 124975 1373548 10.99 --- Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 TukeyHSD(t.aov) #This prints out a HUGE table which I'm not going to include #The point is all but a few of the comparisons aren't statistically #significant at the p = 0.05 level plot(TukeyHSD(t.aov))
So... at this point I'd be ready to just say that adjusting the level of NS doesn't affect SC (for all but the 1-24, 2-24 etc, comparisons, its hard to see in the graph, but its there) and thus lowering the level of NS doesn't result in a minimization of SC.
However I just can't reconcile the graphs of the means of SC with the statistical implications from the analysis of variance... is my intuition leading my astray here, should I simply reject my hypothesis? Is there a way I can still increase my sample size to get a significant result, even though the Levene Test says my data is no longer homoscedastic? Should I be using a different statistical tool instead of analysis of variance to decide these things?
Any suggestions, advice, or criticism is appreciated.
P.S I'm not a statistician, so if I'm doing something glaringly stupid, please let me know.