Is this a correct way of calculating statistical significance

Question

I have the following problem:

I have a website category (Baby products) that during September last year went through a process of reorganisation. So I have a specific traffic value for August and a traffic value for October.

The stakeholder wants to see if the modifications worked and if the difference is "significant".

The statistician went ahead with treating the traffic for August as a sample, the traffic from October as a sample.

The test used was https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test.

He used August and October as sample for a virtual, possible population that is composed of people that might enter the category, in the past, present, or at some point in the future; and for all the website users, regardless of category.

The data is represented by the traffic on each product from the website category, and not on the category as a whole. Thus, if we have 300 products in the category, the data is given by the traffic on each of the 300 products, separately. We thus have 300 data points. The 2 samples were compared using Wilcoxon's signed rank (paired) test, and not the more customary student's t-test for the obvious reason: the distributions had (statistically) significant departures from normality, as acknowledged by several tests (Q-Q plots, Shapiro-Wilk and D'Agostino).

Now, I fear that this might be wrong because:

1. You have all the data, traffic from August and October. You cannot run a significance test if you have all the data.

2. The statistician says that we don't have all the data and these two months of August/October are sample from a larger possible population.

3. If 2 is true, then, the sample is seems one of convenience because it is composed of people that choose to enter the site in that specific time frame.

4. If 3 is true, then you still have a problem because the Wilcoxon test assumes that the samples are independent. But website traffic/conversions are not independent. If, for example, I have a baby, I need to get diapers every X weeks.

Can you help me out guys?

Answer 1

I don't see how he could have done a Wilcoxon signed-rank test comparing just two numbers (traffic from October vs August) - are you sure about that? Or did I understand that wrong?

However, I agree with him that you don't have "all the data". What you want to test here is not the number of people that did happen to visit your site in each month, but an underlying number that reflects the expected monthly average number of visitors. In a given month, the actual number of visitors is somewhat random and follows a distribution around this mean value. The question is: if the traffic was different this month, how likely is it that this was due to chance alone, rather than due to a fundamental change in the traffic we're attracting? The less likely the change in traffic is to be explained by chance, the more significant it is.

Answer 2

I think the statistician is mostly right, the question makes sense: given two number of visits, are these numbers compatible with the hypothesis of an unchanged attractiveness of the website?

I don’t get how the traffic was compared with the Wilcoxon test. My first move would be to model it as Poisson, and use the approximation "square root of a Poisson = normal variable vith variance 0.25", thus $$ \sqrt 2(\sqrt{X_{\text{aug}}} - \sqrt{X_{\text{oct}}}) \sim \mathcal N(0,1),$$ leading to a test procedure.

However the problem is that the traffic can change simply because of the month — I suspect that August and October are very different months in this regard, independently of the site attractiveness.

Here is what I would do, assuming that the traffic in one week is large enough and that the site is several years old.

1. Compute the mean traffic $mw_i$ ($i = 1,\dots, 52$) on each of the 52 weeks of the year, across the last 5 years for example
2. Denote $w_{ij}$ the traffic for each week $i$ of year $j$. You are interested in the difference between $w_{ij}$ and the mean traffic at this time of the year $mw_i$. Just plot the values $w_{ij} - mw_i$ (time on $x$-axis, value on $y$-axis). This should be easy to interpret.
3. Asses the effect of the site reorganization simply by eyeballing — I frankly don’t think that computing a $p$-value is meaningful. If you don’t see an increasing slope after the reorganization, there is no change, that's it.