I know machine learning algorithms are good at determining/predicting what an instance is (classification and such) or how instances are related. Are there any algorithms that, given classified training data and classified test data, can describe (in some way, maybe pointing out attributes) why an test instance is classified a particular way.

An example might help as well. Say we're dealing with a binary-class dataset:

    Class    A    B    C    D    ...
1   T        0.5  0.7  0.3  0.9  ...
2   T        0.6  0.2  0.4  0.8  ...
3   T        0.5  0.6  0.6  0.7  ...
4   F        0.9  0.2  0.2  0.5  ...
5   F        0.9  0.8  0.4  0.5  ...

Now we want to train a model, given training data of this type, such that we can give a test instance and ask, "Why, or why not, is this instance class=T? What variables have the most significant impact on the classification?

So I could then give a test instance with a classification given:

    Class    A    B    C    D    ...
X   T        0.4  0.9  0.8  0.9  ...

The algorithm could then respond, maybe indicating specific attributes which seem to be indicators of distinction between the two classes or something along those lines. For example: "Instance X would tend to be classified as T because value A tends to be around 0.5 and D tends to be around 0.9." Or alternatively there may be no good reason for X to be classified as T, then the algorithm would either be inconclusive or have low confidence measure on its results.

I'm not sure if anything like this exists, but have searched around and thought it might be useful to ask here. Any help (for good or worse) is greatly appreciated. Thank you.


closed as unclear what you're asking by Michael Chernick, kjetil b halvorsen, Peter Flom Jun 3 '17 at 12:17

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ You wouldn't have another algorithm to tell you why your classifier produced this output. You would just study your classifier itself. $\endgroup$ – gung Jun 6 '17 at 18:39
  • $\begingroup$ @gung in this scenario, I don't have access to the classifier. I just have it's results. $\endgroup$ – ryan Jun 6 '17 at 20:34
  • $\begingroup$ Do you know anything about the classifier that was used (eg, random forest, neural network, etc)? How it was trained? Etc. $\endgroup$ – gung Jun 6 '17 at 22:19
  • $\begingroup$ @gung I should clarify, having access to the classifier is unnecessary or at least not helpful. Take this scenario, we have a dataset of restaurants. I now say for each restaurant, if average_price is above $10, class = expensive, else, class = inexpensive. Now remove the attribute average_price from all data. Train a model to figure out why some restaurants are expensive, regardless of average price. Basically trying to fit a model to an artificially created classification. Does this example help more? $\endgroup$ – ryan Jun 6 '17 at 23:40

Decision tree learning produces models that are very easy to interpret. You can just trace an input's path down the tree and see exactly why it ended up in the class it landed in. That direct interpretability is actually a key strength of decision trees. Similarly, random forests, which are made of decision trees, have a similar benefit. But interpretation is less direct because there are many trees to consider for a single input.

More complex, nonlinear, or highly dimensional models (like neural networks) still have rules that govern input to output relationships. Those rules are usually even deterministic, so you can trace the inputs in the same way as with decision trees. The problem arises from interpreting that information in a meaningful way.

A toy example about house price prediction might help demonstrate what I mean.

  • A decision tree can directly tell you that a house ended up in the 500K-750K bucket because it has 2 bathrooms, 2 bedrooms, between 1200 and 1400 square feet, and is newer than 5 years old.

  • A neural net making the same prediction will directly tell you that 100 inputs to 50 tanh functions across 3 layers gave a softmax output that corresponds to 500K-750K.

The latter is much more difficult to interpret, but not a lost cause.

Look up model/prediction interpretation for the type of model you're interested in to see how others have approached the problem.


You are talking about variable or feature selection, of which the literature is vast.

Various methods exist, which can be broadly characterized in three categories:

  • Filter methods: these methods typically use univariate statistics to filter out or select features in function of your target variable.
  • Wrapper methods: these methods incrementally add/remove features which are best/worst performing.
  • Embedded methods: these methods instantly give you a score along with your prediction. As the answer above, Random Forests for example, will be able to give you along with your trained model an output which is called variable importances.

An overview of review papers concerning the topic of variable selection can be found here.


Not the answer you're looking for? Browse other questions tagged or ask your own question.