# Random forests for multivariate regression

I have a multi-output regression problem with $d_x$ input features and $d_y$ outputs. The outputs have a complex, non-linear correlation structure.

I'd like to use random forests to do the regression. As far as I can tell, random forests for regression only work with a single output, so I would have to train $d_y$ random forests - one for each output. This ignores their correlations.

Is there an extension to random forests that takes output correlations into account? Maybe something like Gaussian process regression for multi-task learning.

• here's what i've been able to dig up so far: scikit-learn.org/dev/modules/tree.html#multi-output-problems Jul 24, 2012 at 6:13
• sure. i have high dimensional "images" (actually dI/dV spectra) of peptides. the goal is to figure out the locations & classes of the amino acids that make up the peptide. my first approach was image segmentation, but CRFs and pixel-wise random forests failed. so now, instead of saying each pixel "belongs" to one & only one amino acid (not really true), i'm assigning each pixel a relative "influence" value from nearby amino acids. this results in a $d_y$ dimensional histogram for each pixel. hence, multiple output regression! Jul 24, 2012 at 14:53
• It might be a belated reply: in Crimisini et al. Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning they use RF in a way that might suits you for organs boundaries identification. Jan 29, 2013 at 1:29
• This might be late as well, but might help anyone stumbling upon this post. Random Forest can easily be trained using multivariate data. Everything happens in the same way, however instead of using variance for information gain calculation, we use covariance of the multiple output variables. And more importantly, the leaves now contain N-dimensional PDFs. Sep 24, 2014 at 14:12
• I don't know that the RF "This ignores their correlations". Given the ensemble nature of the RF, I think they might account for the correlations. If they took univariate input and gave univariate output then they wouldn't be accounting for correlations. Feb 3, 2015 at 12:10