I have the following problem:
I have a website category (Baby products) that during September last year went through a process of reorganisation. So I have a specific traffic value for August and a traffic value for October.
The stakeholder wants to see if the modifications worked and if the difference is "significant".
The statistician went ahead with treating the traffic for August as a sample, the traffic from October as a sample.
The test used was https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test.
He used August and October as sample for a virtual, possible population that is composed of people that might enter the category, in the past, present, or at some point in the future; and for all the website users, regardless of category.
The data is represented by the traffic on each product from the website category, and not on the category as a whole. Thus, if we have 300 products in the category, the data is given by the traffic on each of the 300 products, separately. We thus have 300 data points. The 2 samples were compared using Wilcoxon's signed rank (paired) test, and not the more customary student's t-test for the obvious reason: the distributions had (statistically) significant departures from normality, as acknowledged by several tests (Q-Q plots, Shapiro-Wilk and D'Agostino).
Now, I fear that this might be wrong because:
You have all the data, traffic from August and October. You cannot run a significance test if you have all the data.
The statistician says that we don't have all the data and these two months of August/October are sample from a larger possible population.
If 2 is true, then, the sample is seems one of convenience because it is composed of people that choose to enter the site in that specific time frame.
If 3 is true, then you still have a problem because the Wilcoxon test assumes that the samples are independent. But website traffic/conversions are not independent. If, for example, I have a baby, I need to get diapers every X weeks.
Can you help me out guys?