Initially things were done in a particular way (A) and I changed to in some key ways to do things a bit differently (B). Now I wish to evaluate whether the change was for the better. What tests can I do to validate this claim? I have historical and observational data for 3 years on each of the approaches. Which statistical tests can I use to measure/quantify the effectiveness of the change?
For example: Items were selected for implementation were selected in a particular manner. I changed the selection process to be more rigorous and value-centric (for some notion of value) and claimed that the items selected by my approach are superior with respect to their value for the organization/implementation.
I now wish to conduct a retrospective analysis to see if things are better as a result of my approach or just due to chance. I was suggested to use MANCOVA which I'd never heard of/used before. I'm working on understanding it but is there something else that I should look at? How are process changes statistically evaluated 'in the wild'? (Or just left to observational data?)