# How to test if change is significant across multiple categories?

Here is a look at my data. We asked the same respondents (n=~400) to provide us with their current and future consumption as a proportion of total expenditure. Plotted here are the mean proportions of total expenditure for each category for "Now" and "Later", respectively current and future.

What I'm looking for is a statistical method that I can use to test if the other categories are increasing while the largest category is decreasing. The graph shows this, to some degree, but I would like to make sure that it is statistically valid.

-
@mbq, thanks I wasn't exactly sure how to title it! – Brandon Bertelsen Nov 7 '10 at 17:48
Because this question appears to have nothing to do with R I have removed the "r" tag. – whuber Nov 7 '10 at 17:50
Please add to your question the N, and also what you predicted before you obtained these data. If you predicted that later would always be greater than now that's very different from predicting that there would be 'some effect' of time. – John Nov 7 '10 at 17:59
@whuber, I chose the R tag because a solution using R would be incredibly useful. – Brandon Bertelsen Nov 7 '10 at 17:59
Sample size informs us about the power of any test but it is not needed to conduct a valid test. – whuber Nov 7 '10 at 19:09

There are subtle issues involving the difference between designed comparisons and post-hoc comparisons, of which this likely is an example.

If, before collecting the data, you anticipated this kind of pattern, you could employ a simple nonparametric test. The null hypothesis would be that all changes are due to chance with the alternative being that a specified category was increasing and the other eight categories were decreasing. Under the null, positive changes have a 50% chance of occurring, implying the chance of the alternative is $(0.50)^8(1 - 0.50)^1$ = $0.002$: highly significant evidence for the alternative.

The analysis for a post-hoc observation is difficult because we can't even get started with describing the situation. Exactly what kind of pattern would you happen to notice and considered worthy of testing? So many are possible, with no accurate description available, that all we can say (from experience) is that (a) it is highly likely that any interested investigator would notice some pattern in the data and (b) a post-hoc hypothesis test could be constructed to "demonstrate" the "high significance" of that pattern, exactly as I did above. For these reasons, applying hypothesis tests after the fact to support claims of "statistical validity" for exploratory results is frowned upon. (Among statisticians, who should know better, it is called "data snooping" or worse.)

One way out is to conduct your analysis with c. half the data, randomly selected. Look for any patterns you like. Construct an appropriate suite of hypothesis tests for those patterns and then apply them to the held-out data only. This is in the spirit of the scientific requirement for replication. If you don't do this, then you would be obliged to repeat your experiment to confirm whatever you're seeing in the data you currently have.

-
yes, I did expect this pattern as it represents a well researched natural decline in a product life cycle. In this scenario the largest category represents the company that was first to market. As we would expect a larger and larger proportion of future expenditure is going to competitors with a smaller share going to the first to market company. – Brandon Bertelsen Nov 7 '10 at 18:04
that understood how would I implement this test using R? – Brandon Bertelsen Nov 7 '10 at 18:38
@Brandon I am glad to hear that this is not a post-hoc test. You don't need R to implement it: the second paragraph of my response contains all the calculations needed! In effect, this well-researched pattern would have a 1/512 chance of occurring by chance in the case that all nine possibilities (increase/decrease) were independent and equally likely. Because 1/512 is so small (less than the standard threshold of 1/20), you have a valid claim that it is evidence that the pattern is not due to chance. – whuber Nov 7 '10 at 19:08
I should clarify that saying this movement or change in share of wallet is expected from secondary/tertiary/etc market entrants. As time passes the second/third/etc gain larger share or wallet and also take up some of the first entrants share of wallet. By well-researched, I meant that this phenomenon is well researched not that this specific instance or usage case is well researched beyond what I'm attempting to show right now. – Brandon Bertelsen Nov 7 '10 at 19:47

Given the additional information you've subsequently posted I'm not sure any statistical test is going to be that informative. If you had a strong prediction of a pattern such as this or similar, this is such a low probability event that you're pretty much set just getting these data. With an N of 400 almost any tests will most definitely be significant. Some good descriptive stats like confidence intervals would be very useful.

I would suggest that caution be made in your description of the downward trend being remotely meaningful. It's such a tiny amount that, yeah, if your N is big enough it will be significant. But is that tiny drop in percentage meaningful? I think the more meaningful statement is that it's not an increase like the others and that it is staying roughly flat. Don't try to change the story of really small effects with statistical tests.

-
Thanks John, I guess what I'm trying to get at here is if the market leader has reached their potential or saturation point. The other market entrants are continuing to increase their share of wallet whereas the first to market's share of wallet is apparently decreasing or staying the same. To be able to say this with any confidence has substantial marketing and strategic consequences. – Brandon Bertelsen Nov 7 '10 at 19:52
The value didn't go up, which is what you predicted. The amount it went down is insignificant. No statistical test is going to allow you to say with more confidence that the value didn't go up than the value actually going down. Some descriptive statistics, like confidence intervals of the effects would be good. It would be best to graph all of your values as the difference scores and a confidence interval around those difference scores. – John Nov 8 '10 at 4:00
(in addition to your current graph) – John Nov 8 '10 at 4:06
@Brandon A subtler point revealed by this conversation is that the effects are not independent: they are linked by the size of the market. Thus, given that the market size is practically unchanged, a decrease in the leader's share necessarily leads to increases in the shares of at least one competitor. This calls for a different model and a different test, as suggested by @John's comments: it would suffice to demonstrate that the leader's decrease is unlikely to be due to random market fluctuations or sampling error. A better model would also account for changes in market size. – whuber Nov 8 '10 at 13:38
@Brandon Good enough. But now we are aware that the validity of the analysis may depend on this fact of significant growth, indicating it's an important thing to know and to document. – whuber Nov 8 '10 at 19:03