I've been working on a research project for close to five years now. For my thesis I have to show how "well" did my approach improve things.

Setup: Every year we use a tool A to brainstorm and negotiate software requirements. The tool was wiki-based and had very low participation from technical and non-technical (client) stakeholders. (Yes we have real living and breathing clients for our class :). I looked at the state of affairs and saw that perhaps social networking based/influenced technologies could help increase participation. So I created tool B to replace tool A. However, tool A was used for the first 2 years and tool B for the latter 3 years.

Environment: The students changed every year but the overall composition of the class was relatively same (i.e., we had the similar amount of awesome, average and underperforming teams/students). We strive for selecting client projects which are similar level of difficulty and are doable in the duration of the class. They can all be considered projects of the same 'class' (class as in category - same level of complexity etc).

Here's my hypothesis: Tool B will increase stakeholder participation as compared to Tool A (it was initially in past tense, then changed to present and now to future. Not sure what's right. Keep getting corrections for tense from advisors.)

Here are my measurements:

  1. Client (non-technical) participation using Tool B vs Tool A - via access logs
  2. Team (student) participation using Tool B vs Tool A - via access logs + observational data
  3. Number of requirements captured/negotiated (new/updated) in Tool A vs Tool B
  4. Client surveys (for ascertaining the usefulness of tool A for capturing/negotiating requirements. Already have for Tool B.)

One of my advisors suggests I use MANCOVA with 1-3 above as DVs and covariates capturing "sense of complexity of projects", "average number of use-cases per project" and "some metric for team composition/makeup" (which are pretty much the same across the years). Another advisor thinks that simple t-tests would work just fine: That is, I compare the average performance of groups across the years (Group 1 = Tool A; Group 2 = Tool B) and it should be sufficient or maybe an ANOVA at most. Another advior says not to do anything since the data itself is highly skewed i.e., using Tool B has really increased each of the above! He said that doing a statistical test is only to increase the perceived success of the tool and just makes a pompous show of the rigor in analysis.

I'm not really sure what would be a good approach here? I'm familiar with t-tests but have never done a MANCOVA ever and am afraid that I may just crunch the numbers and falsify underlying assumptions. What would be an appropriate test for such an experimental design, which is done across multiple years, with different groups, keeping the environment relatively constant? I have many such hypotheses w.r.t. tool B since there are many things that it enables from a process standpoint than what was doable before. It's really confusing with 3 advisors giving different advice and I not being a statistician to be able to decide.


2 Answers 2


Guess what, all 3 of them might have a point. The issues with "A causes B" are tricky. :)

But first things first. If your hypothesis is:

Tool B will increase stakeholder participation as compared to Tool A.

Stick with it. That's probably the most straightforward and honest thing to measure (honest, in my opinion, since you don't use any covariates such as "sense of complexity of projects" that are unreliable in terms of how well can you measure that etc.). As you correctly pointed out, and so did Prof. 2, a two sample t-test is the right tool to measure mean differences between two groups.

However, then you seem to be wanting to analyze other four things you are measuring, out of which three weren't measured for both tools according to your description: 1:B, 2:B, 3:{A,B}, 4:A. Statistically, I don't see how you want to determine differences between groups for anything other than measurement 3. Which leaves you with a t-test again.

Your main problem is that a t-test will allow you to say that there is a significant difference. You always need a controlled experiment to claim causality, which in your case is tricky, as you didn't deploy both tools simultaneously using a control group, but rather you first test one, than the other. The obvious problem with this approach is that in the meantime, the usage pattern might have changed due to a number of factors so diverse such as better smartphones, more savvy users, you don't know.

  • But I would still claim that what you have is more than observational data.
  • I would roll with a t-test for the things measured for both tools.
  • I would avoid metrics such as "perceived complexity of XY".
  • If the densities of the "Number of requirements" of A and B are visibly different, plot them.

You can also run one of the plethora of tests for testing whether your two empirical distributions are the same. (I agree with Prof. 3 that if you can clearly see a different distribution shape, centered around a different value, formal tests are a bit of a showmanship, but he is a professor and can get away with saying that, you probably won't).

Best of luck!

  • $\begingroup$ I don't really have causuality. A doesn't cause B. I only say B is better than A for increasing participation. Period. Also, I do have each of the measures for both tools. I've edited the question to reflect that. However, the way the class is designed it's almost impossible to have a control group since there is only so much one can do/manage. So we have to completely remove something we intend to change and replace it for the next years' class. $\endgroup$
    – PhD
    Oct 27, 2013 at 20:13
  • $\begingroup$ Causality in the sense that tool B causes higher participation than tool A. Right? You don't just want to say there is a difference between the different classes. You want to say there is an increase in usage, and it's thanks to the tool B. Hence you are declaring B as causal for a surge in users/user activity. And of course, I understand that you are limited by the circumstances. Which is why I would simply point these difficulties out when I describe the methodology. $\endgroup$ Oct 27, 2013 at 20:17
  • $\begingroup$ Ah! I see. I misconstrued your allusion to causality :) $\endgroup$
    – PhD
    Oct 27, 2013 at 20:32

The tests you mention are not appropriate for your situation. They will only tell you the probability of getting a difference difference between years 1-2 and years 3-5 as or more extreme than the difference you observed if there was exactly zero difference from year to year. This null hypothesis is highly unlikely to be true regardless of whether tools were changed. It is also unlikely that classes in the future will be exactly the same type of students as in the past.

What you care about should be (I think) whether tool B will lead to higher participation in the future than tool A. This is an "analytic" problem, while the statistical tests you are attempting to use are meant for "enumerative problems". Yes, this type of use is very common and it has lead to about 80 years of misleading results in many fields.

The only way to make rational decisions is to have understanding of the underlying data generating process. If there is little background knowledge all you can do is plot the data and look for patterns that indicate there may be some lurking/confounding variable that offers an alternative explanation for the increase in participation. You should try to break up the data into as many plausible subgroups as possible (e.g., type of student) and look for patterns.

If you and other experts cannot think of any plausible alternative explanations then it would be rational to decide to continue using tool B. If an alternative explanation is available then further study is necessary to determine which is responsible. A good source on this issue would be William Edwards Deming.



EDIT: Here is a quote from Deming (decide for yourself whether he is a credible source, but he knew both Fisher and J Neyman):

Limitations of statistical inference. All results are conditional on (a) the frame whence came the units for test; (b) the method of investigation (the questionnaire or the test-method and how it was used) ; (c) the people that carry out the interviews or measurements. In addition (d), the results of an analytic study are conditional also on certain environmental states, such as the geographic locations of the comparison, the date and duration of the test, the soil, rainfall, climate, description and medical histories of the patients or subjects that took part in the test, the observers, the hospital or hospitals, duration of test, levels of radiation, range of voltage, speed, range of temperature, range of pressure, thickness (as of plating), number of flexures, number of jolts, maximum thrust, maximum gust, maximum load.

The exact environmental conditions for any experiment will never be seen again. Two treatments that show little difference under one set of environmental circumstances or even within a range of conditions, may differ greatly under other conditions-other soils, other climate, etc. The converse may also be true: two treatments that show a large difference under one set of conditions may be nearly equal under other conditions.

There is no statistical method by which to extrapolate to longer usage of a drug beyond the peritd of test, nor to other patients, soils, climates, higher voltages, nor to other limits of severity outside the range studied. Side effects may develop later on. Problems of maintenance of machinery that show up well in a test that covers three weeks may cause grief and regret after a few months. A competitor may stop in with a new product, or put on a blast of advertising. Economic conditions change, and upset predictions and plans. These are some of the reasons why information on an analytic problem can never be complete, and why computations by use of a loss-function can only be conditional. The gap beyond statistical inference can be filled in only by knowledge of the subject-matter (economics, medicine, chemistry, engineering, psychology, agricultural science, etc.), which may take the formality of a model [12], [14], [15].

Deming, W. Edwards "On probability as a basis for action" The American Statistician, volume 29, 1975


  • $\begingroup$ So, I'm not sure what you mean. Just don't do any significance tests? Does changing the tense of the hypothesis matter? I'm not sure I understand what you're getting at. Honestly, whether it'll continue to support in the future is less of a concern than has it been so over the past? What you are suggesting seems to be a form of "believe me, here's the trend" analysis without anything substantive to back it up. $\endgroup$
    – PhD
    Oct 31, 2013 at 0:33
  • $\begingroup$ Yes, significance tests are of limited value for you. They may be highly misleading. Read the Deming paper in the second link, I found his distinction between enumerative and analytic purposes highly illuminating. Just because you can calculate a p value does not mean you have done something substantive. You can perform a statistical test on before/after but this requires making a number of assumptions that you know a priori are false. $\endgroup$
    – Flask
    Oct 31, 2013 at 0:37
  • $\begingroup$ This paper may be more useful in explaining my point. $\endgroup$
    – Flask
    Oct 31, 2013 at 0:53
  • $\begingroup$ @PhD I was thinking a good plot would be to take the year on year change in participation vs year. Then you should see a large spike when tool B was introduced if it helped. There may be some signal detection algorithm that could help you get a "substantive" number to back up your claims. If you find one please post your own answer. $\endgroup$
    – Flask
    Nov 1, 2013 at 22:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.