I've been working on a research project for close to five years now. For my thesis I have to show how "well" did my approach improve things.
Setup: Every year we use a tool A to brainstorm and negotiate software requirements. The tool was wiki-based and had very low participation from technical and non-technical (client) stakeholders. (Yes we have real living and breathing clients for our class :). I looked at the state of affairs and saw that perhaps social networking based/influenced technologies could help increase participation. So I created tool B to replace tool A. However, tool A was used for the first 2 years and tool B for the latter 3 years.
Environment: The students changed every year but the overall composition of the class was relatively same (i.e., we had the similar amount of awesome, average and underperforming teams/students). We strive for selecting client projects which are similar level of difficulty and are doable in the duration of the class. They can all be considered projects of the same 'class' (class as in category - same level of complexity etc).
Here's my hypothesis: Tool B will increase stakeholder participation as compared to Tool A (it was initially in past tense, then changed to present and now to future. Not sure what's right. Keep getting corrections for tense from advisors.)
Here are my measurements:
- Client (non-technical) participation using Tool B vs Tool A - via access logs
- Team (student) participation using Tool B vs Tool A - via access logs + observational data
- Number of requirements captured/negotiated (new/updated) in Tool A vs Tool B
- Client surveys (for ascertaining the usefulness of tool A for capturing/negotiating requirements. Already have for Tool B.)
One of my advisors suggests I use MANCOVA with 1-3 above as DVs and covariates capturing "sense of complexity of projects", "average number of use-cases per project" and "some metric for team composition/makeup" (which are pretty much the same across the years). Another advisor thinks that simple t-tests would work just fine: That is, I compare the average performance of groups across the years (Group 1 = Tool A; Group 2 = Tool B) and it should be sufficient or maybe an ANOVA at most. Another advior says not to do anything since the data itself is highly skewed i.e., using Tool B has really increased each of the above! He said that doing a statistical test is only to increase the perceived success of the tool and just makes a pompous show of the rigor in analysis.
I'm not really sure what would be a good approach here? I'm familiar with t-tests but have never done a MANCOVA ever and am afraid that I may just crunch the numbers and falsify underlying assumptions. What would be an appropriate test for such an experimental design, which is done across multiple years, with different groups, keeping the environment relatively constant? I have many such hypotheses w.r.t. tool B since there are many things that it enables from a process standpoint than what was doable before. It's really confusing with 3 advisors giving different advice and I not being a statistician to be able to decide.