2
$\begingroup$

I would like to compare the performance of two algorithms A and B under various conditions. Performance is measured on a ratio scale. In a simplified setting, the input data set is a factor (DS1, DS2, ...). The problem is that there are parameters (nominal factor levels) specific to A, which I cannot apply to B, similar to a nested ANOVA. For instance, algorithm A could be run in three flavors: A1, A2, and A3. Algorithm B could come in variants B1 and B2 but I can apply only one algorithm to a data set at a time.

Using variant of algorithm A (1,2,3) and variant of algorithm B (1,2) as two factors doesn't make sense, because I cannot combine A1 with B2. As EdM pointed out in the first comment, it might be easier to interpret the problem as having to compare five algorithms, A1, A2, A3, B1, and B2. In this case, would a ANOVA or a MANOVA be favorable? Below are two designs.

Is it a problem that the five algorithms are variants of two base algorithms, i.e. that there is a relationship between A1, A2 and A3 as well as between B1 and B2? If for instance it turns out that A1 and A2 perform the same, the algorithm factor levels in the ANOVA wouldn't be independent. Is that a problem? Is the MANOVA favorable in this case because it naturally handles relationships between independent variables? 

Factorial ANOVA Design
╔═══════════╦═══════════╦═════════════╗
║ Data Set  ║ Algorithm ║ Performance ║
╠═══════════╬═══════════╬═════════════╣
║ DS1       ║ A1        ║ result A1   ║
║ DS1       ║ A2        ║ result A2   ║
║ DS1       ║ A3        ║ result A3   ║
║ DS1       ║ B1        ║ result B1   ║
║ DS1       ║ B2        ║ result B2   ║
║ DS2       ║ A1        ║ result A1'  ║
║ DS2       ║ ...       ║             ║
╚═══════════╩═══════════╩═════════════╝

MANOVA Design
╔══════════╦═══════════════════╦═════════════╦═══════════╦═══════════╦═══════════╗
║ Data Set ║ Performance of A1 ║ Perf. of A2 ║ P. of A3  ║ P. of B1  ║ P. of B2  ║
╠══════════╬═══════════════════╬═════════════╬═══════════╬═══════════╬═══════════╣
║ DS1      ║ result A1         ║ result A2   ║ result A3 ║ result B1 ║ result B2 ║
║ DS2      ║ result A1         ║ result A2   ║ result A3 ║ result B1 ║ result B2 ║
║ DS3      ║ ...               ║             ║           ║           ║           ║
╚══════════╩═══════════════════╩═════════════╩═══════════╩═══════════╩═══════════╝

Edit: The idea of MANOVA was inspired by this example:

we may conduct a study where we try two different textbooks, and we are interested in the students' improvements in math and physics.

The same source seems to be supporting my worries about correlations in the performances of the different algorithms, but I'm not sure:

Multicollinearity and Singularity – When there is high correlation between dependent variables, one dependent variable becomes a near-linear combination of the other dependent variables. Under such circumstances, it would become statistically redundant and suspect to include both combinations.

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Might it be simpler to think of this as comparing 4 algorithms: vector/VL2, vector/VLInf, histogram/HC, and histogram/HB? It's not clear why, in your table, you are specifying histogram options for cases where the analysis is done with vectors. $\endgroup$
    – EdM
    Commented Jun 16, 2015 at 21:10
  • $\begingroup$ That's the problem. I added your feedback to the question. $\endgroup$
    – cw'
    Commented Jun 17, 2015 at 8:09
  • $\begingroup$ quantitudepod.org/s2e09-manova-must-die $\endgroup$
    – kjetil b halvorsen ♦
    Commented Jul 29, 2023 at 0:49

1 Answer 1

Reset to default
1
$\begingroup$

It sounds like you want to test multiple hypothesis at the same time.

  1. Is there a difference in performance between A and B?
  2. Is there a difference in performance between A1, A2 and A3?
  3. Is there a difference in performance between B1 and B2?

These hypothesis need different tests. For 1. you'll have an ANOVA whereby the overall performance scores of A are compared to B. For 2. and 3. you'll only use the data relevant for that question. In the end you could consider pairwise comparisons between specific algorithms to find the best one (e.g. A1 vs B2) but keep in mind that you'll have to correct for multiple testing.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you, that adds some clarity. Do you have an opinion about the MANOVA approach? Wouldn't that avoid the problem of multiple testing? $\endgroup$
    – cw'
    Commented Jun 17, 2015 at 15:03
  • $\begingroup$ I don't really see how a MANOVA would look like as you still have one outcome variable (performance). If I understand you correctly than the DS1, DS2, are your experimental subjects (the things you test on). You are trying different algorithms on them (you could see that as different treatments) so there is some dependence going on and you might consider a mixed model: testing the difference in performance per DS (like the table you made that described your MANOVA example). In that case DS would be included as a random variable and you test for structural difference per algorithm. $\endgroup$
    – Ivo
    Commented Jun 17, 2015 at 15:14
  • $\begingroup$ I thought about using each performance as an outcome variable, performance of A1, performance of A2, etc. $\endgroup$
    – cw'
    Commented Jul 17, 2015 at 17:29
  • 1
    $\begingroup$ OK. But then you'll still have 1 outcome variable, namely performance. MANOVA would make sense if you have multiple outcome variables for every observation. $\endgroup$
    – Ivo
    Commented Jul 26, 2015 at 13:03
  • 1
    $\begingroup$ But that would be a multivariate outcome variable (in |R^5) – I'd say that's what MANOVA is made for and models with multiple univariate variables. $\endgroup$
    – cw'
    Commented Jul 26, 2015 at 13:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.