Which high performance methods exist to determine the statistical significance of web analytics metrics?

In the field of web analytics, how can one calculate the statistical significance of a metric (for instance Repeat Visitor's). Imagine that we have millions of visitors daily and we are registering each visit, through cookies maybe. We store data per visit/visitor in a relational database and want to show a number for interesting metrics on demand in a report. Performance is an issue here, so we need fast queries and computations. Also we don't want to make any assumptions, about the data distribution (definitely not binomial for all metrics). What are the best relevant techniques to solve this problem?

• We'll need you to describe what you mean by "the statistical significance of a metric." You haven't yet posed a statistical problem to be solved. – rolando2 Jun 10 '12 at 17:30
• Well, there is one answer that makes sense to me. I think it is clear enough. – Navi Jun 11 '12 at 18:22

1 Answer

Could you elaborate on why you're worried about the performance of the analysis code? I completely understand why you'd need a highly tuned database for collecting the data if you're getting millions of hits every day. On the other hand, if you're doing the usual sorts of traffic analysis, you probably don't want to re-run the entire analysis for each visitor. In addition to being overkill, that might set you up for some nasty multiple comparisons issues if, for example, you keep rerunning an A-B test until you get a significant result.

Instead, you might want to analyze snapshots of your data. That shouldn't require a ton of computing power. Just to put some numbers on it, I ran some basic analyses on 10 million data points using matlab on my (somewhat older) computer. Linear regression, T tests and sign-rank tests took about a second each; ANOVAs were a little longer (~10 seconds). Permutation tests took a while, but most of that was generating the permutations (and could probably be sped up).

Furthermore, you probably don't even need to keep all the data around. For example, the classic binomial tests just need the proportions and counts. Similarly, a t-test just requires means and variances, which are easy enough to compute online. In fact, you could have triggers on your database update the means/proportions/variances/etc on insert, which would make those analyses pretty much instantaneous.

• The analysis is not going to be done with Matlab, but on demand for multiple metrics in a website. At least this is the current plan. This could potentially hang the UI. We are not dealing with "binomial proportions" most of the time. Was also thinking of triggers for variance, I guess that's the best solution. – Navi Jun 11 '12 at 18:24
• I included matlab as a not-quite worst case. The language itself isn't terribly fast and a lot of stuff that makes it fast (e.g, BLAS, FFTW) is available separately (and, often, for free). More generally, I'd encourage you to think about what sort of analysis you want first, then figure out how to make it fast. It's not always possible (MCMC or resampling is, by definition, slow), but it usually is and it's the right thing to do statistically too. – Matt Krause Jun 13 '12 at 1:27