What is the most appropriate test for a multi-year different group/yr experiment?

Question

I've been working on a research project for close to five years now. For my thesis I have to show how "well" did my approach improve things.

Setup: Every year we use a tool A to brainstorm and negotiate software requirements. The tool was wiki-based and had very low participation from technical and non-technical (client) stakeholders. (Yes we have real living and breathing clients for our class :). I looked at the state of affairs and saw that perhaps social networking based/influenced technologies could help increase participation. So I created tool B to replace tool A. However, tool A was used for the first 2 years and tool B for the latter 3 years.

Environment: The students changed every year but the overall composition of the class was relatively same (i.e., we had the similar amount of awesome, average and underperforming teams/students). We strive for selecting client projects which are similar level of difficulty and are doable in the duration of the class. They can all be considered projects of the same 'class' (class as in category - same level of complexity etc).

Here's my hypothesis: Tool B will increase stakeholder participation as compared to Tool A (it was initially in past tense, then changed to present and now to future. Not sure what's right. Keep getting corrections for tense from advisors.)

Here are my measurements:

1. Client (non-technical) participation using Tool B vs Tool A - via access logs
2. Team (student) participation using Tool B vs Tool A - via access logs + observational data
3. Number of requirements captured/negotiated (new/updated) in Tool A vs Tool B
4. Client surveys (for ascertaining the usefulness of tool A for capturing/negotiating requirements. Already have for Tool B.)

One of my advisors suggests I use MANCOVA with 1-3 above as DVs and covariates capturing "sense of complexity of projects", "average number of use-cases per project" and "some metric for team composition/makeup" (which are pretty much the same across the years). Another advisor thinks that simple t-tests would work just fine: That is, I compare the average performance of groups across the years (Group 1 = Tool A; Group 2 = Tool B) and it should be sufficient or maybe an ANOVA at most. Another advior says not to do anything since the data itself is highly skewed i.e., using Tool B has really increased each of the above! He said that doing a statistical test is only to increase the perceived success of the tool and just makes a pompous show of the rigor in analysis.

I'm not really sure what would be a good approach here? I'm familiar with t-tests but have never done a MANCOVA ever and am afraid that I may just crunch the numbers and falsify underlying assumptions. What would be an appropriate test for such an experimental design, which is done across multiple years, with different groups, keeping the environment relatively constant? I have many such hypotheses w.r.t. tool B since there are many things that it enables from a process standpoint than what was doable before. It's really confusing with 3 advisors giving different advice and I not being a statistician to be able to decide.

Answer 1

Guess what, all 3 of them might have a point. The issues with "A causes B" are tricky. :)

But first things first. If your hypothesis is:

Tool B will increase stakeholder participation as compared to Tool A.

Stick with it. That's probably the most straightforward and honest thing to measure (honest, in my opinion, since you don't use any covariates such as "sense of complexity of projects" that are unreliable in terms of how well can you measure that etc.). As you correctly pointed out, and so did Prof. 2, a two sample t-test is the right tool to measure mean differences between two groups.

However, then you seem to be wanting to analyze other four things you are measuring, out of which three weren't measured for both tools according to your description: 1:B, 2:B, 3:{A,B}, 4:A. Statistically, I don't see how you want to determine differences between groups for anything other than measurement 3. Which leaves you with a t-test again.

Your main problem is that a t-test will allow you to say that there is a significant difference. You always need a controlled experiment to claim causality, which in your case is tricky, as you didn't deploy both tools simultaneously using a control group, but rather you first test one, than the other. The obvious problem with this approach is that in the meantime, the usage pattern might have changed due to a number of factors so diverse such as better smartphones, more savvy users, you don't know.

• But I would still claim that what you have is more than observational data.
• I would roll with a t-test for the things measured for both tools.
• I would avoid metrics such as "perceived complexity of XY".
• If the densities of the "Number of requirements" of A and B are visibly different, plot them.

You can also run one of the plethora of tests for testing whether your two empirical distributions are the same. (I agree with Prof. 3 that if you can clearly see a different distribution shape, centered around a different value, formal tests are a bit of a showmanship, but he is a professor and can get away with saying that, you probably won't).

Best of luck!

Answer 2

The tests you mention are not appropriate for your situation. They will only tell you the probability of getting a difference difference between years 1-2 and years 3-5 as or more extreme than the difference you observed if there was exactly zero difference from year to year. This null hypothesis is highly unlikely to be true regardless of whether tools were changed. It is also unlikely that classes in the future will be exactly the same type of students as in the past.

What you care about should be (I think) whether tool B will lead to higher participation in the future than tool A. This is an "analytic" problem, while the statistical tests you are attempting to use are meant for "enumerative problems". Yes, this type of use is very common and it has lead to about 80 years of misleading results in many fields.

The only way to make rational decisions is to have understanding of the underlying data generating process. If there is little background knowledge all you can do is plot the data and look for patterns that indicate there may be some lurking/confounding variable that offers an alternative explanation for the increase in participation. You should try to break up the data into as many plausible subgroups as possible (e.g., type of student) and look for patterns.

If you and other experts cannot think of any plausible alternative explanations then it would be rational to decide to continue using tool B. If an alternative explanation is available then further study is necessary to determine which is responsible. A good source on this issue would be William Edwards Deming.

https://en.wikipedia.org/wiki/Analytic_and_enumerative_statistical_studies

https://www.deming.org/media/pdf/081.pdf

EDIT: Here is a quote from Deming (decide for yourself whether he is a credible source, but he knew both Fisher and J Neyman):

Limitations of statistical inference. All results are conditional on (a) the frame whence came the units for test; (b) the method of investigation (the questionnaire or the test-method and how it was used) ; (c) the people that carry out the interviews or measurements. In addition (d), the results of an analytic study are conditional also on certain environmental states, such as the geographic locations of the comparison, the date and duration of the test, the soil, rainfall, climate, description and medical histories of the patients or subjects that took part in the test, the observers, the hospital or hospitals, duration of test, levels of radiation, range of voltage, speed, range of temperature, range of pressure, thickness (as of plating), number of flexures, number of jolts, maximum thrust, maximum gust, maximum load.

The exact environmental conditions for any experiment will never be seen again. Two treatments that show little difference under one set of environmental circumstances or even within a range of conditions, may differ greatly under other conditions-other soils, other climate, etc. The converse may also be true: two treatments that show a large difference under one set of conditions may be nearly equal under other conditions.

There is no statistical method by which to extrapolate to longer usage of a drug beyond the peritd of test, nor to other patients, soils, climates, higher voltages, nor to other limits of severity outside the range studied. Side effects may develop later on. Problems of maintenance of machinery that show up well in a test that covers three weeks may cause grief and regret after a few months. A competitor may stop in with a new product, or put on a blast of advertising. Economic conditions change, and upset predictions and plans. These are some of the reasons why information on an analytic problem can never be complete, and why computations by use of a loss-function can only be conditional. The gap beyond statistical inference can be filled in only by knowledge of the subject-matter (economics, medicine, chemistry, engineering, psychology, agricultural science, etc.), which may take the formality of a model [12], [14], [15].

Deming, W. Edwards "On probability as a basis for action" The American Statistician, volume 29, 1975

https://www.deming.org/media/pdf/145.pdf