I have a fairly complicated problem I am trying to analyse. Thanks for reading - it's a long one.
I am looking at the performance of a program, this program takes as input a file and a set of arguments on how to process that file: [partitions, cores, splitter, queue, timeout]. Each argument has a number of possible values (between three and five each). Each execution of the program takes a certain amount of time, a different amount of time for each file, and for each combination of arguments for each file.
For each file there is an "optimal" set of arguments, that will yield the best performance. In some (very few) cases, multiple sets of arguments yield the same optimal time. Each file has a number of parameters ${x_1}, \dotsc, {x_n}$. All parameters are known for all files.
I have 76 test files, each input was ran ~370 times with every combination of values for each argument (>28000 trials). So, for these 76 input files I know the optimal set of arguments to yield the best performance. My question is, how can I predict the optimal set of arguments prior to execution.
I have looked into random forests and linear discriminant analysis, with limited success.
I have tried three different "types" of model here:
isfastest ~ ${x_i}$ where for each trial I record whether it is the optimal trial for the given file.
bestQueue (and bestSplitter, etc) ~ ${x_i}$ this essentially limits my data to the 76 files, and doesnt consider the individual trials.
label ~ ${x_i}$ - where "label" contains the assignments for each of the arguments as a string, e.g. "splitterA queueB partitionsA coresD timeoutC"
The problem with the third approach is that I have almost as many unique "labels" as I have data points - 76 files, and 65 different "optimal" labels. This has led to overfitting of the model I use, making it useless (I used k-fold cross validation to check this).
The problem with the best ... model is that it limits my data points and doesn't consider interactions between the arguments, e.g. splitterA may be best only when combined with queueB. The best prediction rate I get here (with k-fold cross validation, k=76 is ~40% correct).
The problem with the first approach is that random forests and LDA both always predict FALSE as the prior probability of that is very high (99%).
To add to this, while for most files there is only one "optimal" set of arguments, there is sometimes a second best, that is only marginally slower - so a prediction error that chose one of those, rather than one with a huge performance hit is better.
My questions are:
Is LDA/random forests the right way to go?
Which of the above approaches is "better"?
How can I handle the issue regarding the impact of different incorrect predictions? Is there a technique that allows for multiple classes, e.g. the "best queue" is both queueA and queueB?
