Analysis of variance not statistically significant… but is there still a pattern to the data?

Question

Problem I have a computer algorithm, and am attempting to measure the affect of a parameter (NS) on the output of the algorithm (SC). My hypothesis is that if the level of NS is lowered, there should be a corresponding decrease in SC as well. I'm doing all of my statistical analysis in the R language (see below).

I've performed an analysis of variance on my data, and though the results aren't statistically significant, it looks like there is a pattern to the data (that lowering NS does decrease SC). I tried increasing my sample sizes, which does result in more pairwise comparisons becoming statistically significant. However when I do this, the results of Levene's Test show that my variances are no longer homoscedastic, and thus the analysis of variance isn't valid...

If you look at a graph of the mean values of the output (sc) you can see what I'm talking about when I say "it looks like there is a pattern". I've generated different data sets several times and each time, the means of SC are scattered similarly to graph below (despite the pairwise comparisons being statistically insignificant).

What I am Doing

Here is my R code and some output:

t <- read.table("output.dat") 
names(t) <- c("sc", "ns")
leveneTest(t$sc, group=t$ns, center=median)

Levene's Test for Homogeneity of Variance (center = median)
          Df F value Pr(>F)
group     24  1.0447 0.4018
  124975               
Warning message:
In leveneTest.default(t$sc, group = t$ne, center = median) :
t$ne coerced to factor.

t.aov <- aov(t$sc ~ as.factor(t$ns))
summary(t.aov)

                    Df  Sum Sq Mean Sq F value Pr(>F)    
as.factor(t$ne)     24    1448   60.32   5.488 <2e-16 ***
Residuals       124975 1373548   10.99                   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

TukeyHSD(t.aov)  #This prints out a HUGE table which I'm not going to include
                 #The point is all but a few of the comparisons aren't statistically         
                 #significant at the p = 0.05 level 
plot(TukeyHSD(t.aov))

So... at this point I'd be ready to just say that adjusting the level of NS doesn't affect SC (for all but the 1-24, 2-24 etc, comparisons, its hard to see in the graph, but its there) and thus lowering the level of NS doesn't result in a minimization of SC.

However I just can't reconcile the graphs of the means of SC with the statistical implications from the analysis of variance... is my intuition leading my astray here, should I simply reject my hypothesis? Is there a way I can still increase my sample size to get a significant result, even though the Levene Test says my data is no longer homoscedastic? Should I be using a different statistical tool instead of analysis of variance to decide these things?

Any suggestions, advice, or criticism is appreciated.

P.S I'm not a statistician, so if I'm doing something glaringly stupid, please let me know.

Answer 1

You're thinking about your ANOVA incorrectly. It's OK, lots of people are taught ANOVA that way. The ANOVA does not mean there are any significant differences between levels of the predictor variable. None of them can be significant and yet the ANOVA is. It means that the pattern of data has meaning. Simply report your significant ANOVA and describe the pattern of data. That sounds exactly like what you want to do anyway.

As a suggestion for improvement, it would be a much more meaningful analysis if you did a regression and had some idea about the mathematical relationship between your two variables. It looks a bit exponential but even a simple line would fit pretty good. In fact, a comparison between an ANOVA and linear regression here would show very little advantage from all of the degrees of freedom of the ANOVA and allow you to make a more direct statement about the relationship between the variables.

Answer 2

You are running into one of the fundamental problems with p-values: They are partly dependent on sample sizes.

So, when you increase sample size, smaller effect sizes become significant. This accounts for both changes that you report 1) More comparisons become significant because smaller effect sizes are (this seems like a good thing) 2) The test of heterogeneity becomes significant for the same reason.

It is better, in my view, to concentrate on effect sizes and confidence intervals.