In machine learning and in statistics there exist plenty of measures which estimate the performance of a predictive model. For example, classification accuracy, area under ROC curve ... for classification or $R^2$ for regression models.

Is there any generalisation of such measures for multiple dependent variables in order to evaluate predictive accuracy of Multivariate Multiple Regression, Partial Least Squares Regression or even more complex predictive models where the dependent variables can be of mixed type?

  • $\begingroup$ It will depend on how you weigh the different variables, but you can obviously combine any number of marginal measures as you see fit. I'm not sure, though, that it makes much sense from a prediction point of view to combine a 0-1-loss with a sum of squares. Why do you want a single measure of predictive accuracy, could you provide an example? $\endgroup$ – NRH Jun 21 '11 at 16:23
  • $\begingroup$ Your outcome variable lives in a $p$-dimensional space. If you're talking about discrete valued outcomes, you can make a prediction accuracy statistic out of how often points are correctly placed into subsets of that space. If they are continuous you can use an appropriate distance metric. $\endgroup$ – Macro Jun 21 '11 at 18:50
  • $\begingroup$ @NRH: there are a lot of such examples - depending on your age, marital status, level of education I would like to predict how much do you use Internet, read newspapers and if you are satisfied with your life. Yes, it is possible to weight accuracy in each dimension but doing so you may disregard possible dependencies among the dependent variables. $\endgroup$ – Lan Jun 21 '11 at 19:45
  • $\begingroup$ @Macro - it would be possible, yes. But I would prefer that such measure to be on [0,1] interval. Maybe one could compare the average distance between predicted and actual value to the "predictive accuracy" of mean predictor? $\endgroup$ – Lan Jun 21 '11 at 19:57
  • 2
    $\begingroup$ It sounds to me like you are not really interested in prediction, but rather in modeling, in which case a joint likelihood of the dependent variables given the independent variables seems appropriate. I don't understand the concern about the dependence. Prediction accuracy is measured by some metric, and risk (expected error) is evaluated under the joint distribution. $\endgroup$ – NRH Jun 22 '11 at 5:34

In Machine Learning, many algorithms directly minimise a loss function with some form of capacity control (regularisation). This gives a direct measure of the performance of the classifier on future data, through the use of the loss function that was being minimised. If the specific problem you are dealing with can be framed in an optimisation framework, then a measure may fall out naturally from the loss function. For example, Kristin Bennett in this paper showed that PLS can be formulated as $$ \min_w \left\| \bf{X} - \bf{y}\bf{w}' \right\|_2, \; s.t. \bf{w}'\bf{w} = 1, $$ which they later show bounds the usual least squares loss. More complex predictive models can be phrased in terms of composite loss functions - see for example these slides from Mike Jordan.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.