Is Random Forest the only algorithm to measure the importance of input variables ....?

Question

I have three time series say (Stock price open, Stock price high, Stock price low) and one output (Stock price close) and I need to know which of the 3 inputs has a greater effect on my output. R's Random Forests' (importance) ' %IncMSE yields the importance for this case.

Are there any other algorithms apart from Random Forest, to measure the importance of a input variable?

Answer 1

Edit i added a code example below of how to compute importance for any learner in R. I think the caret package can also provide importance for any learner

Any supervised regression/classification model, that I can think of, could be bootstrap aggregated (bagged) and therefore variable importance could be computed. It would just be a little slow to train e.g. svm 50 times compared to growing 50 trees.

"I need to know which of the 3 inputs has a greater effect on my output"

I would abstain from causal interpretation of importance and at most see it as a source of inspiration. Importance only describe usefulness of features to predict, given all other features and one specific model. Two highly redundant features will roughly share the same fixed amount of variable importance. Two features can be complimentary and have a higher variable importance, than if one of them were never included in the training. This happens if an interaction between two features is useful to predict the output.

So importance is not an universal metric, and the answer will depend on your model and and the included features. You may want to ask instead:

"Which overall relationship between input and output has e.g. an RF/SVM/NN model captured?" -You could use some fancy plots exploring the high dimensional model structure. But I can reveal that the effective RF, SVM or NN model will be something very boring like $close_t = (open_t+high_t+low_t)/3$

"Is this relationship trivial or inspiring?" - In this case, quite trivial, as the future absolute price is highly dependent on the past price. If some asset were 10€ yesterday, its probably gonna be priced 9€ or 12€ today, not 1 cent or 5000€.

Try use rolling window to predict the change of price, that is in contrary more challenging. If you were to succeed better than others, then the effective structure of a well performing empirical model could be very inspiring to form new hypotheses.

rm(list=ls())
nvar = 6
nobs = 500
X = data.frame(replicate(nvar,rnorm(nobs),simplify = FALSE))
names(X) = paste0("X",1:nvar)
y = with(X,X1 + X2 + X3 * X4)

library(randomForest)

rf = randomForest(X,y,importance = T)
imp.internal = importance(rf,type = 1,scale = FALSE)


learner = randomForest
nbag = 50
trainPar = list(ntree=50)
sampsize = NULL
replace=T
trainPar = alist(x = Xbootstrap, y = ybootstrap, ntree = 50) #unevaluated list
bagLearner = function(learner,trainPar = alist(), X,y, nbag =50,sampsize=NULL,replace=T) {
  if(is.null(sampsize)) sampsize = nrow(X)
  out.list = lapply(1:nbag, function(i) {
    bootstrap.ind = sample(nrow(X),replace = replace,size = sampsize)
    oob.bool= !(1:nrow(X) %in% bootstrap.ind) #is the nth element/observation out-of-bag? true/false
    Xbootstrap = X[bootstrap.ind,]
    ybootstrap = y[bootstrap.ind]
    model = do.call(what=learner,trainPar)
    out = list()
    out$model = model
    out$oob.bool = oob.bool
    return(out)
  })
  return(out.list)
}


rf.bag = bagLearner(learner = randomForest,X=X,y=y,
                    trainPar = alist(x = Xbootstrap, y = ybootstrap, ntree = 50)
)

model.bag = rf.bag
metric.explVariance = function(y,yhat){1 - sum((y-yhat)^2) / sum(y^2)}
metric.mse = function(y,yhat) {mean((y-yhat)^2)}

computeImportance = function(model.bag,X,y, metric) {
  oob.pred.m = do.call(cbind,lapply(model.bag,function(mb) {
    oob.pred.1 =  rep(NA,nrow(X))
    oob.pred.1[mb$oob.bool] = predict(mb$model,X[mb$oob.bool,])
    return(oob.pred.1)
  }))
  oob.pred = apply(oob.pred.m,1,mean,na.rm=T)
  baseScore = metric(y,oob.pred)
  permScores = sapply(1:ncol(X), function(j) {
    Xperm = X
    Xperm[,j] = sample(Xperm[,j])
    oob.pred.perm.m = do.call(cbind,lapply(model.bag,function(mb) {
      oob.pred.perm.1 =  rep(NA,nrow(X))
      oob.pred.perm.1[mb$oob.bool] = predict(mb$model,Xperm[mb$oob.bool,])
      return(oob.pred.perm.1)
    }))
    oob.pred.perm = apply(oob.pred.perm.m,1,mean,na.rm=T) 
    return(metric(y,oob.pred.perm ))
  })
  scores = permScores - baseScore
  names(scores) = names(X)
  return(scores)
}

imp.external = computeImportance(rf.bag,X = X,y=y,metric = metric.mse)

#check if results are kind of the same
plot(imp.external,imp.internal)