Inference from linear regression slope and Pearson

Question

Sorry if this has been asked before but I've already done quite a bit of work here and I feel like I'm quite close to an answer.

I am interested in testing whether the PHP function array_key_exists depends on the size of the array. If it did, we should expect that the growth of the computation time for the function would depend on the size of the array. If it does not, then the computation time should be constant. In other words, I want to test the strength of a possible linear relationship.

I measured the computation time for varying sizes of arrays. A few things I see, in ascending order of importance:

1. There lots of points (almost half of them) where the time actually decreases as the size increases. This basically gives me grounds to accept the null hypothesis immediately, because any linear relationship would entail a monotonically increasing function of the size.
2. Very low standard deviation in the time, indicating that the values are tightly clustered.
3. The slope of the simple linear regression is quite small.
4. Exceedingly low Pearson coefficient and its square has practically vanished.

If my null hypothesis is that the real Pearson coefficient is zero, namely there is no relationship between the array size and computation time, how small does my sample Pearson coefficient or its square need to be to reject it? What assumptions do I need to make (perhaps acceptable probabilities of Type I and Type II) to do so?

Is it better to formulate the null hypothesis from the Pearson coefficient or from the linear regression slope? Are they about the same as far as explanatory power in this circumstance?

Answer 1

There are many potential issues.

Your first problem is that the first point really doesn't fit the (roughly) pattern of the rest (look at your data - see below). The linear model simply doesn't describe the data.

You could apply a more complex model, or you could try fitting a simpler model to a subset of the data.

The red line is the line of fit that corresponds to your Pearson correlation; clearly that Pearson correlation is meaningless with the anomalous point included.

The green line is the line if you omit it; it's imperfect, but does make it clear that there is some relationship between Computation Time and Array Size - apart from the that one point at the low end, it's a slowly increasing relationship.

The blue curve is a quadratic fit (without the stray point) - that turns out to add almost nothing to the description of the data. To be honest, to me it looks flat between 100 and 700 and then appears to increase above 700, but I'd hesitate to fit a model for that without an a priori reason (humans are great at identifying patterns in noise).

To revisit the question you asked:

I am interested in testing whether the PHP function array_key_exists depends on the size of the array.

The answer - even for a quick inspection of the data - is fairly plainly, yes - the data at hand are inconsistent with the idea that the computation time doesn't depend on array size. The relationship is noisy, and not of an obvious form, but there's a fairly clear increasing relationship between expected Computation Time and Array Size (for the latter between 100 and 1000) .

So if the relationship is changing but not obvious, we might just try smoothing it. Below are loess smooths to (i) all the data (grey), (ii) the data omitting the first point (in red, which lies over the grey curve in the right half of the plot), and (iii) the data also omitting the last two points (in green, in case you might be tempted to say there's no relationship without those two end points).

In each case, there's some indication of some kind of increase with array size, except for the effect of the 50.