How to determine the combination of factor levels for which the response variable is highest

Question

I have run an experiment with 5 categorical factors. The factors have anywhere between 2 and 8 levels each. I have one response variable, which is continuous in the range of 0 to 100. All-in-all, I have run a fully factorial experiment with 800-something combinations. Each combination has 10 samples. In total, in R-speak, I have a data frame with 6 columns and 8607 rows.

My goal: determine the level of each factor that results in the best performance. For example, I want to be able to say "Performance is generally the best when factor1 is level "A", factor2 is level"C", ..., and factor5 is level "E". Conclusions: always use level "A" for factor1 ....".

How do I achieve this?

I first thought of PCA, but this isn't quite correct because the components that PCA finds are combinations of factors, but I need to be able to say which factor level is best, for each and every factor. I want to keep the factors in tact.

I also thought of ANOVA, which may be what I want, but I'm not sure how to use its output. For example, in R, I get:

> summary(aov(...))
              Df  Sum Sq Mean Sq  F value    Pr(>F)    
preprocess     7  21.430   3.061  180.771 < 2.2e-16 ***
bugData        2   5.276   2.638  155.782 < 2.2e-16 ***
fileData       5   6.462   1.292   76.315 < 2.2e-16 ***
param1         2 255.766 127.883 7551.306 < 2.2e-16 ***
param2         1  15.579  15.579  919.887 < 2.2e-16 ***
Residuals   8589 145.457   0.017                       
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

I don't know how to interpret these results. Is it that param1 has the largest effect, because it's "Sum Sq" is largest? How do I know what level of param1 is best?

So, this is my idea: For each factor, compare the "winning-percentage" of each level against every other level. That is, the number of times that level X "beats" level Y, given that all other factors are equal. I can compare level X and level Y a lot of times, because there are so many other factors and levels of those factors. So, I change the level of the other factors, compare level X and level Y in the current factor, and keep track of who won. Doing this, I should end up with something like "For factor1, levelX beats levelY 85% of the time, and therefore is the better choice."

Does this approach make sense? Is there a name for it? Or is there another approach altogether that achieves what I want?

Any help or pointers is greatly appreciated. I would prefer if my answer is implementable in R, but I can adapt. I have a very beefy machine to use (16 processors, 196G RAM), so I'm not too worried about the efficiency of the algorithm that solves my problem.

Answer 1

This question is amenable to a decision tree analysis technique. With a continuous outcome, the software will simply put cut-points in the middle of measured values, so the cuts will fall between levels you have measured, rather than being actual levels. The categorical predictors work well with this method, as you'll see which levels lead to which outcomes. You'll get a box at the end of each terminal which will contain information like the mean of performance. The branches associated with higher levels of performance will be based on your 5 categorical factors so, all going well, it's a relatively simple and quick way to see if there are any clear associations between factor levels and your performance measure.

The main con to decision trees is they tend to be stepwise, with an "F-to-enter" calculation, so they can suffer from the same drawbacks as stepwise regression, in that you are not guaranteed the optimal solution.

Have a look here and here for an R example. I've only used SAS Enterprise Miner and AnswerTree but the R code looks easy to follow once you've got the general idea of how trees work. This is a nice introduction to decision trees, with some pretty images. :)

There are some issues you'll need to become familiar with if you decide to go down this path, basically these are all around the familiar problem of not using the same data to develop the model and test the model. Some references which you might find useful are here and here.

Update: the rule used by decision trees to choose the variable I believe is based on the amount of variance explained: the decision tree software will go through every available variable split and choose the one that explains the most variance, for example see here. So there is no input from the user on the splits, unless one changes the F-to-enter (for example) criterion.

The resulting tree will have a number of branches. The number of splits you get will depend on which method you use as some restrict the number of splits to two. The path that has the terminal node with the largest mean will show you the factor levels associated with that large mean. The variables, and the values of them, will be shown against each node in that branch. You can conclude that is the optimal path, but remember to do some testing with the model (e.g. cross validation). You also could end up with two nodes with similar means. It would pay do some data visualisation of the tree classifications to see whether it worked, for example the one with the largest mean could also have an excessively large standard deviation and may not classify as well as it appears just from the mean.

The terminal nodes may not be in any order of mean either, in my experience one doesn't get a nice ascending or descending order of nodes from left to right.

Have fun with the method. :)

Update 2: the tree will stop forming branches (nodes) when, across node, this situation occurs:

1. none of the variables that have not been included in the splits meet the F-to-enter rule (e.g. including them does not meet a minimum requirement for explaining variance) and
2. for the variables that have been entered, there are no more splits available that meet the F-to-enter rule, and
3. for the variables that have been entered, all splits have occurred higher up the branch, OR
4. if a maximum number of branches was stipulated, the maximum has been hit on every node where an F-to-enter rule could otherwise still be applied.

Assuming your maximum number of branches hasn't been reached, you can interpret your results this way, and note that these results are exploratory and not definitive unless you have validated them with a suitable validation method:

• all variable splits that explain a reasonable amount of the variance in the model have been entered, so
• any variables not entered either do not explain sufficient variance or are possibly correlated with other variables in the model (it is not possible to tell which of these two situations is more correct with a stepwise model), and
• similarly, when variable splits don't occur (e.g. when you have two levels of a variable sitting together, like filedata levels "d" and "e"), the split will not explain sufficient variance, so the outcome may be invariant to which of the levels occurs. Because this is a stepwise method, the results may not produce the optimal outcome. I recommend grouping the variables on the basis of the results of the decision tree and then looking at boxplots of performance based on these groups. That will clearly show whether the decision tree has given you a good result.

I'm pleased the method appears to have worked. I hope these additional suggestions provide you with a way forward with your analysis. Again, best of luck!