27
$\begingroup$

I have a multi-output regression problem with $d_x$ input features and $d_y$ outputs. The outputs have a complex, non-linear correlation structure.

I'd like to use random forests to do the regression. As far as I can tell, random forests for regression only work with a single output, so I would have to train $d_y$ random forests - one for each output. This ignores their correlations.

Is there an extension to random forests that takes output correlations into account? Maybe something like Gaussian process regression for multi-task learning.

Improve this question
$\endgroup$
11
  • 4
    $\begingroup$ here's what i've been able to dig up so far: scikit-learn.org/dev/modules/tree.html#multi-output-problems $\endgroup$
    – sergeyf
    Jul 24 '12 at 6:13
  • 1
    $\begingroup$ sure. i have high dimensional "images" (actually dI/dV spectra) of peptides. the goal is to figure out the locations & classes of the amino acids that make up the peptide. my first approach was image segmentation, but CRFs and pixel-wise random forests failed. so now, instead of saying each pixel "belongs" to one & only one amino acid (not really true), i'm assigning each pixel a relative "influence" value from nearby amino acids. this results in a $d_y$ dimensional histogram for each pixel. hence, multiple output regression! $\endgroup$
    – sergeyf
    Jul 24 '12 at 14:53
  • 1
    $\begingroup$ It might be a belated reply: in Crimisini et al. Decision Forests: A Unified Framework for Classification, Regression, Density Estimation, Manifold Learning they use RF in a way that might suits you for organs boundaries identification. $\endgroup$
    – Simone
    Jan 29 '13 at 1:29
  • 4
    $\begingroup$ This might be late as well, but might help anyone stumbling upon this post. Random Forest can easily be trained using multivariate data. Everything happens in the same way, however instead of using variance for information gain calculation, we use covariance of the multiple output variables. And more importantly, the leaves now contain N-dimensional PDFs. $\endgroup$
    – masad
    Sep 24 '14 at 14:12
  • 1
    $\begingroup$ I don't know that the RF "This ignores their correlations". Given the ensemble nature of the RF, I think they might account for the correlations. If they took univariate input and gave univariate output then they wouldn't be accounting for correlations. $\endgroup$
    – EngrStudent
    Feb 3 '15 at 12:10
3
$\begingroup$

Here's an example of a multi-output regression problem undertaken with facial recognition. It includes a coding sample as well, it should give you a start with your methodology. http://scikit-learn.org/stable/auto_examples/plot_multioutput_face_completion.html

Improve this answer
$\endgroup$
2
$\begingroup$

There is a new package specifically for that (not personally tested)

https://cran.r-project.org/package=MultivariateRandomForest

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.