1
$\begingroup$

I'm using Random Forest regression and Lasso regression for the task of predicting multiple outputs. I'm using sklearn.ensemble.RandomForestRegressor and sklearn.linear_model.Lasso in Python and they support predicting multiple outputs at once instead of creating an estimator for each output (same features for each one). The question is whether I benefit in terms of accuracy? I see for the Random Forest there is a section called multi-output problems (https://scikit-learn.org/stable/modules/tree.html#tree-multioutput) that explain that there can be an improvement in terms of accuracy when the outputs are correlated. what about the case I don't know if they are correlated? Anyway, I will glad to hear more of your knowledge about the way Random Forest solve the "multi-output problems"

For the Lasso regression, I didn't find any information about this.

$\endgroup$
1
$\begingroup$

Linear models

Lasso (and linear models in general) gets no benefit in terms of performance: any multi-output linear model will just be fitting a matrix of weights, and thus treats all the outputs as independent.

Just to check that:

from sklearn.datasets import make_regression
from sklearn.linear_model import Lasso
X, y = make_regression(n_targets=3, n_features=5)

model = Lasso()
model.fit(X, y)
print(model.coef_)

models = [Lasso()]*3
for i, m in enumerate(models):
    m.fit(X, y[:, i])
    print(m.coef_)

produces the same coefficients.

Tree models

First, imagine you have two perfectly correlated outputs. Then, if you keep all the random effects the same, one tree built on both outputs and two trees each built on one output will make exactly the same splits. So you reduce computational cost, but get absolutely no change in the resulting models.

Next, suppose the two outputs are identical except that their noise terms are independent. Now you may expect the combined model to sometimes choose a split that looks worse for $y_1$ but only because of the noise in $y_1$. In that case, the combined model may be more robust to noise, by using the other variable(s) to select better splits. I suspect this is the main source of benefit, but there won't be such a quick experiment to verify.

If the outputs are correlated, I guess a similar benefit occurs: even where the correlation isn't as strong, using the other variables can reject splits based just on noise in one variable.

If the outputs are completely uncorrelated, then probably no (consistent) gain can be had. And indeed I suspect sometimes it can hurt, because you need to make common splits for all the variables in the same tree. Your optimal hyperparameters will likely need to allow for a more-complex tree, and you are more at the mercy of the greedy splitting than if you separated into multiple models.

$\endgroup$
2
  • $\begingroup$ Regreding the Lasso models - what about the cost function that we try to optimize? Does it combine cost function? If it is, it seems strange to me that it does not affect the fitting of the matrix of weights $\endgroup$
    – Mr.O
    Feb 1 at 9:37
  • 1
    $\begingroup$ @Mr.O It doesn't matter how the loss functions are combined, because the mechanism for decreasing the loss(es) is changing the individual weights, which are independent across outputs. $\endgroup$ Feb 1 at 14:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.