2
$\begingroup$

I'm working with the ALE implementation provided by the iml package in R. This package is accompanied by the usual documentation, a vignette and even a very nice book.

I've studied all three of them trying to figure out the exact interpretation of the resulting ALE values in a classification task. I do have a high level understanding which is: Increasing ALE values when moving from one feature value to a neighbouring one mean that the probability of the model predicting a specific class has increased.

What I cannot figure out is: what is the exact ALE value? The closest thing I find is around figure 8.17 in the book where it says "For the age feature, the ALE plot shows that the predicted cancer probability is low on average up to age 40 and increases after that.". Is it really a probability such that a value of 0.075 for an age of ~82 means the probability for cancer="yes" at that age is 7.5% higher compared to the average prediction probability for that class (cancer="yes") across all feature values for age?

Here is an example I have copied straight from the documentation:

library(rpart)
library(iml)

rf <- rpart(Species ~ ., data = iris)
mod <- Predictor$new(rf, data = iris, type = "prob")
plot(FeatureEffect$new(mod, feature = "Petal.Width"))

The result looks like so: ALE Petal Width

And what is the ALE value when you apply it to a classification model which does not return probabilities but just predicted classes? E.g. I'm using a caret trained random forest w/o specifically extracting the prediction probabilities and iml's ALE still produces values which seem to make sense. What are these?

I guess while qualitatively my high level understanding applies to all implementations of ALE I would be interested in the exact meaning of the ALE values produced by the iml package for a classification task.

$\endgroup$
10
  • $\begingroup$ Please edit your question with sample code or at least sample output images. I will try my best to answer you, but as your questions is framed, it is too general. You can copy the images from the book you refer to (with appropriate citation, of course), but please make the question more concrete. $\endgroup$
    – Tripartio
    Oct 17, 2023 at 12:55
  • 1
    $\begingroup$ @Tripartio, sorry for not being concrete enough, pls see edits above, is this what you were looking for? $\endgroup$
    – MarkH
    Oct 19, 2023 at 15:26
  • $\begingroup$ You seem to be new to Statistics SE, so let me clarify that here, you should only ask a single question and then respond to answers to that question. I asked you to edit your original question because it was not clear, which you did and then I tried to answer it. If my answer was appropriate for your original question, then you should click "accept" to mark it as the accepted answer. But now, you have edited your original question with a different question. In that case, you should start a completely new question and ask that separately. $\endgroup$
    – Tripartio
    Oct 21, 2023 at 9:41
  • $\begingroup$ Could you please revert your edits so that the question shown is the question that was answered below? Otherwise, it looks now as if I did not try to properly answer your latest question, which is very confusing to anyone who would read this thread. You may then start a completely new question with your further inquiries--that is the proper way to go about this. We have a strict policy of one question per post. If you want to attract my attention to the new question, you can add a comment to the question and tag me @Tripartio so that I will receive a notification. Be sure to use the "ale" tag $\endgroup$
    – Tripartio
    Oct 21, 2023 at 9:43
  • $\begingroup$ All that said, it is also important to realize that Statistics SE answers questions about statistics concepts, not questions that are specific to how a particular software (e.g., the "iml" package) implements its statistics. So, your original question was appropriate since I could answer about ALE in general. However, if your question is about a specific implementation, then it should be asked on Stack Overflow. So, please decide what the nature of your second question is and which site is more appropriate. $\endgroup$
    – Tripartio
    Oct 21, 2023 at 9:47

1 Answer 1

1
$\begingroup$

The Y axis for ALE is always on the scale of the predicted values of the Y outcome variable. For numeric outcome variables, this is the same original scale as the Y outcome variable. So, if you were predicting Sepal.Length, for example, then the unit of Y would be centimetres. The 0 point means the average value of predictions for the ALE dataset. Positive values are above average and negative values are below average.

In the case of binary or categorical variables, as in your example, the unit of the ALE Y axis is still the scale of the predicted values of the Y outcome variable. In your example above, the key is type = "prob" Predictor$new(rf, data = iris, type = "prob"). So, the Y outcome in your case is the probability of Species. The zero value means the average probability of the predictions of the decision tree rf. Since these are probabilities, the range from the lowest Y to the highest Y value cannot be greater than 1.0, which is the case here, ranging from approximately -0.5 to 0.5.

(As an aside, I'm not sure why the result of rpart is called rf; dt would make more sense to me.)

The results can be interpreted as:

  • Petal.Width has no measured effect on the probabilities of setosa, which is no doubt because it is the reference class against which the other two classes are compared.
  • The model assigns Petal.Width from 0 to 1.6 cm or so higher than average probability of being versicolor; Petal.Width from 1.6 to 2.5 cm has below average probability of being versicolor.
  • The model assigns Petal.Width from 0 to 1.6 cm or so lower than average probability of being virginica; Petal.Width from 1.6 to 2.5 cm has above average probability of being virginica.
$\endgroup$
1
  • $\begingroup$ Thanks, that is in good alignment with what I described as my "high level understanding". The devil's in the detail, i.e. the bonus question about what happens when not extracting classProbs e.g. in a caret model or when not specifying type="prob" for the iml's Predictor object. I'll modify my question once more to be more specific on that part and add a few more examples of code and resulting images. $\endgroup$
    – MarkH
    Oct 21, 2023 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.