1
$\begingroup$

I'm trying to solve a classification problem using a random forest in R. The training data is a particle's charge at 30 different time instances. However, I need to convert this 30 dimensional data into a single value. I've tried using the sum of the charge across the 30 time bins and I've tried using the variable importance to calculate a weighted sum. To clarify, I fitted a random forest to the 30 dimensional training data and then used the resulting variable importance values to assign a weighting to each time interval for the weighted sum.

Then I trained a random forest on the one dimensional data using the sum and the variable importance weighted sum values. However, the random forest performed better when I wasn't using a weighted sum.

I've tried this with the standard randomForest package and the cforest package and I get the same results.

Could anybody explain why the weighted sum doesn't perform better? In principle should using a weighted sum work better?

Improve this question
$\endgroup$
3
  • $\begingroup$ Based on the information you've provided, I'd guess that the weighted sum is essentially throwing away information, reducing classification accuracy. Variable importance is a rough measure of how much each variable contributes to the model and different methods give different relative weights. If you really want to boil down your predictors into a single variable, you might be better off with principal components or partial least squares. But even then, a single predictor is unlikely to contain all the predictive information available from your full data set. $\endgroup$
    – eipi10
    Sep 25, 2015 at 17:04
  • $\begingroup$ I've flagged this to migrate it to CV. But may have something to do with the auto-correlated nature of your time measurements. This probably impacts both the RF and the VI. See Breiman 2001 and Strobl et al 2008 $\endgroup$
    – Alex W
    Sep 25, 2015 at 17:30
  • 1
    $\begingroup$ random forest (as well as cforest) on the 1-dimentional data makes no sense by the very nature of this algorithm. $\endgroup$
    – lanenok
    Sep 25, 2015 at 18:03

3 Answers 3

Reset to default
3
$\begingroup$

I don't quite get what you're doing (or why), but by training a model on the data and then using the weighted sum to train another model on the same data you're going to overfit - if you are tuning any parameters, you'll need to hold out some of your data and then assess performance against that holdout set.

If all you're looking to do is get a single feature that represents the 30 you have, a good way to do so is PCA, and there are many different packages that allow you to do so. Since you're features are likely highly correlated, you should be able to capture a lot of the predictive power with just a few principal components.

Improve this answer
$\endgroup$
3
$\begingroup$

This would only work for generalized linear models (GLMs). With 'work' I mean that you get perfectly identical error estimates (e.g., train and test MSE) using the original predictors, and using a single predictor, computed by weighing each of the coefficients by their respective regression coefficients.

Random forests are not linear models. Any deviations from linearity will be captured by the random forest, yielding differences between the predictions of the model with the original predictors as inputs, and the predictions of the model with the weighted sum of the predictors as input.

Furthermore, variable importance measures approximate the amount of variance explained. Regression coefficients are on the scale of the standard deviation, not on the scale of the variance, so that will also contribute discrepancies.

Improve this answer
$\endgroup$
1
$\begingroup$

If I understand your question correctly, you are looking for a one dimensional feature computed from a set of 30 features in such a way that the classes are best distinguishable.

A weighted sum is a linear combination of the features, and, for linear combinations, the optimal weights can be found with Fisher's Linear Discriminant Analysis (function lda in the MASS package):

> library(MASS)
> result <- lda(iris[,-5], iris$Species)
> weights <- result$scaling[,1]
> weights
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
   0.8293776    1.5344731   -2.2012117   -2.8104603 
> weighted <- data.frame(weighted.sum=as.matrix(iris[,-5]) %*% weights,
                         Species=iris$Species)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.