3
$\begingroup$

I have three time series say (Stock price open, Stock price high, Stock price low) and one output (Stock price close) and I need to know which of the 3 inputs has a greater effect on my output. R's Random Forests' (importance) ' %IncMSE yields the importance for this case.

Are there any other algorithms apart from Random Forest, to measure the importance of a input variable?

$\endgroup$
  • $\begingroup$ Gradient Boosting Machines with the "gbm" R package is other algorithm with similar performance than RF, and showing also the importance of the predictors, but the ranking should be very similar $\endgroup$ – Jesus Herranz Valera Feb 17 '16 at 11:44
  • 1
    $\begingroup$ Maybe Boruta algorithm will help you: cran.r-project.org/web/packages/Boruta/Boruta.pdf ? $\endgroup$ – Maju116 Feb 17 '16 at 11:50
  • 2
    $\begingroup$ Ordinary regression coefficients will tell you "importance" in a specific sense. What kind of importance are you interested in? $\endgroup$ – Sycorax Feb 17 '16 at 14:39
  • 2
    $\begingroup$ @user777 Actually and unless the predictors have been standardized (never a good practice), your statement is not true since regression coefficients are expressed in the unit of the predictor. As such they are not scale invariant and sensitive to variations in the moments. A better, quick and dirty heuristic is to rank the standardized metrics such as F-statistics, t-values and/or chi-squares associated with the parameters, as appropriate. For the "state-of-the-art" in deriving relative importance, see Ulrike Groemping's papers on 'relaimpo'...a multivariate approach to relative rankings $\endgroup$ – Mike Hunter Feb 17 '16 at 14:52
  • $\begingroup$ @Maju116 I just now saw Random uniform forest package in R . I think it does the best with this use case of finding the input variable affecting the response variable . Also if the input is categorical, say region(A,B,C) it also shows the importance of each region . $\endgroup$ – Sudharsan Apr 26 '16 at 11:32
4
$\begingroup$

Edit i added a code example below of how to compute importance for any learner in R. I think the caret package can also provide importance for any learner

Any supervised regression/classification model, that I can think of, could be bootstrap aggregated (bagged) and therefore variable importance could be computed. It would just be a little slow to train e.g. svm 50 times compared to growing 50 trees.

"I need to know which of the 3 inputs has a greater effect on my output"

I would abstain from causal interpretation of importance and at most see it as a source of inspiration. Importance only describe usefulness of features to predict, given all other features and one specific model. Two highly redundant features will roughly share the same fixed amount of variable importance. Two features can be complimentary and have a higher variable importance, than if one of them were never included in the training. This happens if an interaction between two features is useful to predict the output.

So importance is not an universal metric, and the answer will depend on your model and and the included features. You may want to ask instead:

"Which overall relationship between input and output has e.g. an RF/SVM/NN model captured?" -You could use some fancy plots exploring the high dimensional model structure. But I can reveal that the effective RF, SVM or NN model will be something very boring like $close_t = (open_t+high_t+low_t)/3$

"Is this relationship trivial or inspiring?" - In this case, quite trivial, as the future absolute price is highly dependent on the past price. If some asset were 10€ yesterday, its probably gonna be priced 9€ or 12€ today, not 1 cent or 5000€.

Try use rolling window to predict the change of price, that is in contrary more challenging. If you were to succeed better than others, then the effective structure of a well performing empirical model could be very inspiring to form new hypotheses.

rm(list=ls())
nvar = 6
nobs = 500
X = data.frame(replicate(nvar,rnorm(nobs),simplify = FALSE))
names(X) = paste0("X",1:nvar)
y = with(X,X1 + X2 + X3 * X4)

library(randomForest)

rf = randomForest(X,y,importance = T)
imp.internal = importance(rf,type = 1,scale = FALSE)


learner = randomForest
nbag = 50
trainPar = list(ntree=50)
sampsize = NULL
replace=T
trainPar = alist(x = Xbootstrap, y = ybootstrap, ntree = 50) #unevaluated list
bagLearner = function(learner,trainPar = alist(), X,y, nbag =50,sampsize=NULL,replace=T) {
  if(is.null(sampsize)) sampsize = nrow(X)
  out.list = lapply(1:nbag, function(i) {
    bootstrap.ind = sample(nrow(X),replace = replace,size = sampsize)
    oob.bool= !(1:nrow(X) %in% bootstrap.ind) #is the nth element/observation out-of-bag? true/false
    Xbootstrap = X[bootstrap.ind,]
    ybootstrap = y[bootstrap.ind]
    model = do.call(what=learner,trainPar)
    out = list()
    out$model = model
    out$oob.bool = oob.bool
    return(out)
  })
  return(out.list)
}


rf.bag = bagLearner(learner = randomForest,X=X,y=y,
                    trainPar = alist(x = Xbootstrap, y = ybootstrap, ntree = 50)
)

model.bag = rf.bag
metric.explVariance = function(y,yhat){1 - sum((y-yhat)^2) / sum(y^2)}
metric.mse = function(y,yhat) {mean((y-yhat)^2)}

computeImportance = function(model.bag,X,y, metric) {
  oob.pred.m = do.call(cbind,lapply(model.bag,function(mb) {
    oob.pred.1 =  rep(NA,nrow(X))
    oob.pred.1[mb$oob.bool] = predict(mb$model,X[mb$oob.bool,])
    return(oob.pred.1)
  }))
  oob.pred = apply(oob.pred.m,1,mean,na.rm=T)
  baseScore = metric(y,oob.pred)
  permScores = sapply(1:ncol(X), function(j) {
    Xperm = X
    Xperm[,j] = sample(Xperm[,j])
    oob.pred.perm.m = do.call(cbind,lapply(model.bag,function(mb) {
      oob.pred.perm.1 =  rep(NA,nrow(X))
      oob.pred.perm.1[mb$oob.bool] = predict(mb$model,Xperm[mb$oob.bool,])
      return(oob.pred.perm.1)
    }))
    oob.pred.perm = apply(oob.pred.perm.m,1,mean,na.rm=T) 
    return(metric(y,oob.pred.perm ))
  })
  scores = permScores - baseScore
  names(scores) = names(X)
  return(scores)
}

imp.external = computeImportance(rf.bag,X = X,y=y,metric = metric.mse)

#check if results are kind of the same
plot(imp.external,imp.internal)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.