1
$\begingroup$

I've seen this statement in the SciKit documentation for Regression Trees: Supported criteria are “mse” for the mean squared error, which is equal to variance reduction as feature selection criterion

Considering that (MSE = variance + bias^2), is this claim always true? is mse reduction in regression trees always equal to variance reduction? Can we safely say that without considering bias?

UPDATE: deviation of the estimator from the true parameter is MSE, and the deviation of the estimator from its expected value is the variance. They will be the same if bias is 0. I'm just trying to see what's the math behind this claim of mse reduction is equal to variance reduction? In other words, what if selecting a feature to do the split only decreases the bias (and not the variance)?

$\endgroup$

1 Answer 1

1
$\begingroup$

Here, MSE = variance / sample size. Bias is implicitly included, and trying to separate it out wouldn't be meaningful.

Note that random forests often outperform regression with this task.

The goal of a regression tree is to generate a line that best fits the data.

MSE is the average squared difference between the actual data values and where the data point would be on the proposed line. The tree runs an algorithm that finds the line that results in the smallest MSE.

Regarding variance, this method seeks to minimize variance. Variance is defined as the sum of the squared errors (or differences between the actual data points and line selected as the best solution).

R-squared is another method that is commonly used.

$\endgroup$
2
  • $\begingroup$ thanks. So, deviation of the estimator from the true parameter is MSE, and the deviation of the estimator from its expected value is the variance. They will be the same if bias is 0. I'm just trying to see if there is an actual math formula behind the claim of mse reduction is equal to variance reduction? In other words, what if selecting a feature to do the split only decreases the bias (and not the variance)? $\endgroup$ Commented Jul 31, 2020 at 0:14
  • $\begingroup$ The correlation between MSE reduction and variance reduction will be very, very high. They are practically equivalent in this context. As far as a proof of equality, I don't know. $\endgroup$
    – user280191
    Commented Jul 31, 2020 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.