I'm working on a system that trains an arbitrarily large amount of classifiers (e.g. Support Vector Machines, k-Neighbors Classifiers, Neural Networks, Decision Trees, ...) on the same training set and collects a bunch of performance metrics for each model. Now, most of these are your standard run-of-the-mill metrics like precision, recall, overall accuracy and all that, but some are more complex (or should I say "different"?), for example:
- Runtime- and memory complexity of the learning algorithm
- Number of performed preprocessing/dimensionality reduction steps
- The absolute amount of time it took to train the model on the training set
- The average amount of time it took to classify a single data point from the test set
I want to find a good way of ranking these models based on user-specified weights for a subset of the aforementioned performance metrics. Simplified example:
- Precision: 92%
- no. of preprocessing steps: 2
- Precision: 89%
- no. of preprocessing steps: 1
- Precision: 72%
- no. of preprocessing steps: 2
If a user's goal was to find the model that was least "complex" while still achieving reasonable precision, they would likely assign a higher weight to the "no. of preprocessing steps" attribute and see which model gets ranked highest (probably model 2, but it really depends on the concrete values of the weights of course).
So, in short, I am faced with a so-called Multiple-criteria decision-making (MCDM) problem, and I need to solve it. I've looked into several MCDM methods and figured it would be a good idea to start off simple by ranking the models using the weighted sum of all their metrics. But I'm afraid it's not that straightforward...
- The first problem I've been facing here is that the metrics have different meanings in terms of which values are actually "better" than others. In the above example, higher accuracy ("more is better") with less preprocessing steps ("less is better") would form a desirable combination. But how should the ranking algorithm know that less is better in terms of the number of preprocessing steps? One obvious way to deal with this would be to simply negate the values for all metrics where less is better. That way the general assumption "more is better" holds for all metrics in question no matter what, ultimately causing the weighted average to be higher for more desirable models. Does this approach have any caveats? I guess not (please let me know if it does)...
Now, on to the "real" problem:
- The metrics come in different units, and some of the metrics are theoretically unbounded (e.g. the number of preprocessing steps), so it becomes sort of hard to scale them to a fixed range. I figured, however, that since I'm only trying to find the best model relative to the other models in the set (as opposed to finding a "universal" score/rank) I could simply scale all metrics to [0,1] by diving each metric by the sum of all of its occurrences in the set. That way the "more is better" assumption still holds, but now all metrics will also have the same unit (%). I don't know why, but to me this sort of feels like the kind of solution that seems plausible on the surface but can backfire in some cases, but I don't really know why. I'm not that great at math or stats so I lack the mathematical expertise (or at least intuition) to come up with a reason or a concrete example where this may be the case.
I'd be really glad if anyone could help me assess the solutions I came up with for both problems.
Note: I've already looked into MCDM methods that aim to eliminate the need for a common unit among the metrics, namely the weighted product model. Here, however, all the metrics need to be greater than one. Again, I could just scale all metrics to the range [1,2] or similar, but the question whether or not that's a good idea still stands.