0
$\begingroup$

The tool I'm working on is used to process large number of documents. Recently, I've implemented small feature for this tool. The feature affects some % of all documents. After few days I received a message from other engineer that size of documents we processed increased substantially and one of the possible reason behind it is my change. I want to measure if it's true. I have following data for each document we processed:

  • processing time
  • flag indicating if my feature affected the document
  • document size

How can I measure hypothesis whether my change could cause increase in documents' size? The system is complicated, so it could happen by (non-deterministic) interaction with other components. I would like to come up with some metric that could tell me if it's the case.

$\endgroup$
  • $\begingroup$ Depending on what you are measuring you would use a simple test, for ex. t-test to compare mean document size. $\endgroup$ – user2974951 Dec 14 '18 at 7:58
1
$\begingroup$

I would use the following experimental set-up:

You should randomly select documents and assign them to two groups (let's say group A and group B).

Documents in group A will be processed with your newly implemented feature.

Documents in group B will be procesesed with the 'original' approach.

Then for each document you measure your variables of interest.

So you obtain a dataset with, for each of the documents in your experiment, the treatment group and the outcome measurement.

Then you apply an independent samples t-test on your outcomes (only on one outcome variable at a time) and you look at the p-value to see if there is a significant difference between the two treatments.


Note: Having a random selection could be difficult in practice, but it is necessary to be sure that it is the treatment that causes the difference and not some other variable that was under or over represented in your samples. A scientifically sound experimental set-up is the best way to make sure you are looking at causality and not just at some coincidental correlation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.