I would like to compare the performance of two algorithms A and B under various conditions. Performance is measured on a ratio scale. In a simplified setting, the input data set is a factor (DS1, DS2, ...). The problem is that there are parameters (nominal factor levels) specific to A, which I cannot apply to B, similar to a nested ANOVA. For instance, algorithm A could be run in three flavors: A1, A2, and A3. Algorithm B could come in variants B1 and B2 but I can apply only one algorithm to a data set at a time.
Using variant of algorithm A (1,2,3) and variant of algorithm B (1,2) as two factors doesn't make sense, because I cannot combine A1 with B2. As EdM pointed out in the first comment, it might be easier to interpret the problem as having to compare five algorithms, A1, A2, A3, B1, and B2. In this case, would a ANOVA or a MANOVA be favorable? Below are two designs.
Is it a problem that the five algorithms are variants of two base algorithms, i.e. that there is a relationship between A1, A2 and A3 as well as between B1 and B2? If for instance it turns out that A1 and A2 perform the same, the algorithm factor levels in the ANOVA wouldn't be independent. Is that a problem? Is the MANOVA favorable in this case because it naturally handles relationships between independent variables?
Factorial ANOVA Design ╔═══════════╦═══════════╦═════════════╗ ║ Data Set ║ Algorithm ║ Performance ║ ╠═══════════╬═══════════╬═════════════╣ ║ DS1 ║ A1 ║ result A1 ║ ║ DS1 ║ A2 ║ result A2 ║ ║ DS1 ║ A3 ║ result A3 ║ ║ DS1 ║ B1 ║ result B1 ║ ║ DS1 ║ B2 ║ result B2 ║ ║ DS2 ║ A1 ║ result A1' ║ ║ DS2 ║ ... ║ ║ ╚═══════════╩═══════════╩═════════════╝ MANOVA Design ╔══════════╦═══════════════════╦═════════════╦═══════════╦═══════════╦═══════════╗ ║ Data Set ║ Performance of A1 ║ Perf. of A2 ║ P. of A3 ║ P. of B1 ║ P. of B2 ║ ╠══════════╬═══════════════════╬═════════════╬═══════════╬═══════════╬═══════════╣ ║ DS1 ║ result A1 ║ result A2 ║ result A3 ║ result B1 ║ result B2 ║ ║ DS2 ║ result A1 ║ result A2 ║ result A3 ║ result B1 ║ result B2 ║ ║ DS3 ║ ... ║ ║ ║ ║ ║ ╚══════════╩═══════════════════╩═════════════╩═══════════╩═══════════╩═══════════╝
Edit: The idea of MANOVA was inspired by this example:
we may conduct a study where we try two different textbooks, and we are interested in the students' improvements in math and physics.
The same source seems to be supporting my worries about correlations in the performances of the different algorithms, but I'm not sure:
Multicollinearity and Singularity – When there is high correlation between dependent variables, one dependent variable becomes a near-linear combination of the other dependent variables. Under such circumstances, it would become statistically redundant and suspect to include both combinations.