Consider we want to compare algorithm A and algorithm A* (a modified version of algorithm A). To evaluate them, both are executed on multiple datasets (D1, D2, ...) and their performance is measured by a reasonable metric M.

Is the relation for improvement of algorithm A* w.r.t algorithm A, with the following formula a standard effect size to compare their performance on dataset $D_i$?

$\frac{M(A*, D_i)-M(A, D_i)}{M(A, D_i)}$

For example let $M(A, D_i)$ be 1000, $M(A*, D_i)$ be 1050. Then:

$\frac{M(A*, D_i)-M(A, D_i)}{M(A, D_i)} = \frac{1050-1000}{1000} = 5%$

Does this simple formula have a name in literature? Can this be considered a standard effect size?


I would not consider this as an estimate of a standardized effect size, because the term "standardized" means, you express the effect in terms of the standard deviation of the measurements: $\frac{\mu_1-\mu_0}{\sigma}$.

In your case, the denominator is not a standard deviation.

If I would like to invent a name for your performance measure, I would call it e.g. the "relative improvement".

  • $\begingroup$ In my field (Empirical Software Engineering), few papers have used something like Cohens d (en.wikiversity.org/wiki/Cohen%27s_d) and many papers have used this "relative improvement". How can this be explained? $\endgroup$ – Mostafa Mahdieh Nov 1 '19 at 11:59
  • 1
    $\begingroup$ Cohens d can be used to get a sense about the relevance of an effect in a population. How large is the effect in relation to the variation, that can be observed across the population. Silly purely invented example: maybe an apple a day raises the mean IQ about half a point, but it is not really relevant, considering the IQ variation across the population. With $100 \times$ "relative improvement", you ask another question: How many percent points is the new method better, than the old one. The variation of your $M$ is not your primary interest. $\endgroup$ – ghlavin Nov 1 '19 at 12:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.