# How to test if change is significant across multiple categories?

Here is a look at my data. We asked the same respondents (n=~400) to provide us with their current and future consumption as a proportion of total expenditure. Plotted here are the mean proportions of total expenditure for each category for "Now" and "Later", respectively current and future.

What I'm looking for is a statistical method that I can use to test if the other categories are increasing while the largest category is decreasing. The graph shows this, to some degree, but I would like to make sure that it is statistically valid.

-
@mbq, thanks I wasn't exactly sure how to title it! –  Brandon Bertelsen Nov 7 '10 at 17:48
Because this question appears to have nothing to do with R I have removed the "r" tag. –  whuber Nov 7 '10 at 17:50
Please add to your question the N, and also what you predicted before you obtained these data. If you predicted that later would always be greater than now that's very different from predicting that there would be 'some effect' of time. –  John Nov 7 '10 at 17:59
@whuber, I chose the R tag because a solution using R would be incredibly useful. –  Brandon Bertelsen Nov 7 '10 at 17:59
Sample size informs us about the power of any test but it is not needed to conduct a valid test. –  whuber Nov 7 '10 at 19:09
show 1 more comment

There are subtle issues involving the difference between designed comparisons and post-hoc comparisons, of which this likely is an example.

If, before collecting the data, you anticipated this kind of pattern, you could employ a simple nonparametric test. The null hypothesis would be that all changes are due to chance with the alternative being that a specified category was increasing and the other eight categories were decreasing. Under the null, positive changes have a 50% chance of occurring, implying the chance of the alternative is $(0.50)^8(1 - 0.50)^1$ = $0.002$: highly significant evidence for the alternative.

The analysis for a post-hoc observation is difficult because we can't even get started with describing the situation. Exactly what kind of pattern would you happen to notice and considered worthy of testing? So many are possible, with no accurate description available, that all we can say (from experience) is that (a) it is highly likely that any interested investigator would notice some pattern in the data and (b) a post-hoc hypothesis test could be constructed to "demonstrate" the "high significance" of that pattern, exactly as I did above. For these reasons, applying hypothesis tests after the fact to support claims of "statistical validity" for exploratory results is frowned upon. (Among statisticians, who should know better, it is called "data snooping" or worse.)

One way out is to conduct your analysis with c. half the data, randomly selected. Look for any patterns you like. Construct an appropriate suite of hypothesis tests for those patterns and then apply them to the held-out data only. This is in the spirit of the scientific requirement for replication. If you don't do this, then you would be obliged to repeat your experiment to confirm whatever you're seeing in the data you currently have.

-
yes, I did expect this pattern as it represents a well researched natural decline in a product life cycle. In this scenario the largest category represents the company that was first to market. As we would expect a larger and larger proportion of future expenditure is going to competitors with a smaller share going to the first to market company. –  Brandon Bertelsen Nov 7 '10 at 18:04
that understood how would I implement this test using R? –  Brandon Bertelsen Nov 7 '10 at 18:38
@Brandon I am glad to hear that this is not a post-hoc test. You don't need R to implement it: the second paragraph of my response contains all the calculations needed! In effect, this well-researched pattern would have a 1/512 chance of occurring by chance in the case that all nine possibilities (increase/decrease) were independent and equally likely. Because 1/512 is so small (less than the standard threshold of 1/20), you have a valid claim that it is evidence that the pattern is not due to chance. –  whuber Nov 7 '10 at 19:08
I should clarify that saying this movement or change in share of wallet is expected from secondary/tertiary/etc market entrants. As time passes the second/third/etc gain larger share or wallet and also take up some of the first entrants share of wallet. By well-researched, I meant that this phenomenon is well researched not that this specific instance or usage case is well researched beyond what I'm attempting to show right now. –  Brandon Bertelsen Nov 7 '10 at 19:47