Initially things were done in a particular way (A) and I changed to in some key ways to do things a bit differently (B). Now I wish to evaluate whether the change was for the better. What tests can I do to validate this claim? I have historical and observational data for 3 years on each of the approaches. Which statistical tests can I use to measure/quantify the effectiveness of the change?

For example: Items were selected for implementation were selected in a particular manner. I changed the selection process to be more rigorous and value-centric (for some notion of value) and claimed that the items selected by my approach are superior with respect to their value for the organization/implementation.

I now wish to conduct a retrospective analysis to see if things are better as a result of my approach or just due to chance. I was suggested to use MANCOVA which I'd never heard of/used before. I'm working on understanding it but is there something else that I should look at? How are process changes statistically evaluated 'in the wild'? (Or just left to observational data?)

  • $\begingroup$ This is (again) a very broad question, there are entire books on the topic, e.g. Experimental and Quasi-Experimental Designs for Research by Donald Campbell. To select a particular technique, you need to give more details on what you measured exactly, what other data you have and what you want to learn precisely. $\endgroup$ – Gala Jun 5 '13 at 20:20
  • $\begingroup$ I see what you mean...let me see if I can make it more concrete/focused. $\endgroup$ – PhD Jun 5 '13 at 21:31
  • $\begingroup$ @chl - Not exactly. The release is purely from business standpoint. This is from an internal process POV. The underlying 'problem structure' could be the same, but are two completely different projects in different domains/organizations. $\endgroup$ – PhD Jun 11 '13 at 17:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.