9
$\begingroup$

I am building an interactive forecast tool (in python) as an aid to forecasting that is done in my organisation. To date the forecast process has been largely human driven, with forecasters assimilating the data in their natural neural networks and using their learned gut feel to make predictions. From a long term forecast verification and predictive modelling study I've done I've found what you might expect; different forecasters exhibit different biases, the effects of some predictors seem to be overstated and other important ones seem to be ignored and in general the forecast performance is mediocre compared with relatively simple empirical models.

The forecasts will continue to be manual, but I am trying to build a useful tool to provide the forecasters with a better quantification of the relative effects of predictors. There are also important effects such as seasonal influences that are often overlooked that I would like the tool to highlight to the user. I am expecting a degree of backlash and scepticism about the modelling process from some of the more 'experienced' forecasters (many of whom have little formal knowledge of statistics), so the communication is at least as important and the model performance itself in terms of achieving a measurable improvement in forecast accuracy.

The models I'm developing have a strong auto-regressive component that is at times modified significantly by events which show up as measured values in some predictors that are, during non-event times, close to zero. This accords with the mental model that forecasters use. The key part is being able to demonstrate which of the 'event' measurements are most influential in driving the prediction away from the auto-regressive value for any given forecast. I imaging the process in this way; the forecaster divines their best guess value, the model suggests a different one and the forecaster asks why. The model replies something like "see here, this value of this predictor increases the forecast value in Summer. If it was Winter, it would move the other way. I know there are these other measurements, but they have much less effect than this one".

Now, imagine the model was a simple linear regression. One could imagine displaying the relative 'effect' of event based predictors by multiplying the value by the model co-efficient and displaying as a simple bar chart. All the bars from the different predictors add up to the total deviation from the AR value, and this succinctly and clearly shows the ones that are, in this instance, having a strong influence.

The problem is that the process being forecast displays a high degree of non-linearity in the predictors, or at least, I have had much more success with black-box non-linear machine learning algorithms (random forest and GBM) than with GLMs for this data-set. Ideally I would like to be able to seamlessly change the model working 'under the hood' without the user experience changing, so I need some generic way of demonstrating in a simple fashion the importance of the different measurements without using some algorithm specific approach. My current approach will be to quasi-linearise the effects by setting all values to zero except for one predictor, record the predicted deviation and then repeat for all predictors, displaying the results in the bar chart mentioned above. In the presence of strong non-linearity, this may not work so well. Are there any known approaches for achieving in a clear way what I am trying to do here?

$\endgroup$
1
  • 1
    $\begingroup$ What did you end up with -- could you put up a picture or two ? Also, "setting all values to zero except for one predictor" -- don't you want the gradient around the current best values, not around all 0 ? $\endgroup$
    – denis
    Jun 26 '13 at 8:25
4
+50
$\begingroup$

One way that you can assess predictor influence on forecasts is to estimate the gradient of the output with respect to the predictors. This can be done by estimating the partial derivatives of the non-linear prediction function with respect to each of the predictors by finite differences.

Ideally you will do this on the actually observed test inputs. For example, you may average the absolute values of the estimated gradients at all the test inputs in the previous 2 days. The magnitude of this average gradient can be used to sort the predictors' importance. (You will need to careful with the gradient estimation to use appropriate units by z-scoring or some such method.) You can save these estimated gradients by season for comparative analysis.

See "How to Explain Individual Classification Decisions", by David Baehrens et. al. in JMLR for more on this idea. The paper deals with classification but easily generalizes to regression as well.

$\endgroup$
1
  • $\begingroup$ That's fantastic! A very useful reference that will be helpful for this issue I have and elsewhere. $\endgroup$ Feb 19 '13 at 10:59
2
$\begingroup$

Have you tried scikit-learn module in python.

You can "computer_importance" for the features of its randomForestClassifier

$\endgroup$
5
  • 1
    $\begingroup$ First I also thought that calculating feature importance might be helpful, but in the end it is a comparably poor approach when one shall explain the predicted value for a specific instance. Feature importance delivers only vague hints to human experts. $\endgroup$
    – mlwida
    Feb 19 '13 at 8:21
  • $\begingroup$ Aside, the OP asked for a model independent approach ... $\endgroup$
    – mlwida
    Feb 19 '13 at 8:24
  • $\begingroup$ The problem with variable importance measures is that they apply to the whole dataset on average, rather than telling you what was actually important in any one particular case. $\endgroup$ Feb 19 '13 at 11:01
  • $\begingroup$ Actually I think this is a model independent approach, you might indeed apply it to other classifiers than random forests. In Breiman's website there is a subtle remark about how you could compute the variable importance for a single case. stat.berkeley.edu/~breiman/RandomForests/cc_home.htm#varimp (last sentence) I think that this hasn't been extensively studied yet, or at least extensively tested. The mean variable importance is not always what you want. For example it is not when you want to help a practitioner to take a decision on one case. This is a really interesting topic. $\endgroup$
    – Simone
    Feb 21 '13 at 3:40
  • $\begingroup$ There is an interesting paper where Breiman discusses a bit about this method on logistic regression as well: "Statistical Modeling: The Two Cultures". A nice read. The sentence that I like the most is: "My definition of variable importance is based on prediction. A variable might be considered important if deleting it seriously affects prediction accuracy." This statement applies to any classifier you might use. $\endgroup$
    – Simone
    Feb 21 '13 at 3:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.