# Is the improvement of one algorithm with respect to another one, a standard effect size?

Consider we want to compare algorithm A and algorithm A* (a modified version of algorithm A). To evaluate them, both are executed on multiple datasets (D1, D2, ...) and their performance is measured by a reasonable metric M.

Is the relation for improvement of algorithm A* w.r.t algorithm A, with the following formula a standard effect size to compare their performance on dataset $$D_i$$?

$$\frac{M(A*, D_i)-M(A, D_i)}{M(A, D_i)}$$

For example let $$M(A, D_i)$$ be 1000, $$M(A*, D_i)$$ be 1050. Then:

$$\frac{M(A*, D_i)-M(A, D_i)}{M(A, D_i)} = \frac{1050-1000}{1000} = 5%$$

Does this simple formula have a name in literature? Can this be considered a standard effect size?

I would not consider this as an estimate of a standardized effect size, because the term "standardized" means, you express the effect in terms of the standard deviation of the measurements: $$\frac{\mu_1-\mu_0}{\sigma}$$.

In your case, the denominator is not a standard deviation.

If I would like to invent a name for your performance measure, I would call it e.g. the "relative improvement".

• In my field (Empirical Software Engineering), few papers have used something like Cohens d (en.wikiversity.org/wiki/Cohen%27s_d) and many papers have used this "relative improvement". How can this be explained? – Mostafa Mahdieh Nov 1 '19 at 11:59
• Cohens d can be used to get a sense about the relevance of an effect in a population. How large is the effect in relation to the variation, that can be observed across the population. Silly purely invented example: maybe an apple a day raises the mean IQ about half a point, but it is not really relevant, considering the IQ variation across the population. With $100 \times$ "relative improvement", you ask another question: How many percent points is the new method better, than the old one. The variation of your $M$ is not your primary interest. – ghlavin Nov 1 '19 at 12:08