How to describe most important features of ensemble model as list?

Question

I have created 3 different models and output of them is a class probability in binary classification problem. Models are bit different, showing importance from different features. I have of course one data matrix as a source for this exercise where 70% of data is used as training sample.

How one can summarize importance of different feature values to the final class prob estimate if only data matrix and list of features used is know besides this class probability estimate?

Individual models can be of course explained by different methods, but how one can explain avg ensemble predictions?

EDIT:

I have an data matrix containing all features and their values from different models plus of course combined ensemble probability estimate. How can one summarize how globally different features affect ensemble prob?

EDIT 2:

Can feature importances from different models combined somehow if different models use different features and variable value codings might be different?

Answer 1

Here are three intuitive ways to solve the problem:

• First normalize the feature importance of the features for each model to belong to 0-1 and then average the normalized feature importance values across the three models.
• Do the same as above, but instead of averaging perform weighted averaging of the feature importance. The weights in this case can be the performance of the models on your hold-out set. That way, you put more weight on your better performing models.
• In case you are interested in just ranking the features and you are not interested in their relative importance you can rank the features for each model and then average (or even weight-average) the corresponding ranks. For instance, the most important features has rank 1, the second most important feature rank 2 etc.. You do this across the three models and then you average the ranks. Of course, lower values suggest higher feature importance.

Answer 2

A straightforward way is to remove each feature and see how the ensemble model performs without it.

Answer 3

How to describe most important features of ensemble model as list?

To make best use of a probability forecasts, users must choose a probability threshold which gives the correct balance of alerts and false alarms for their particular application. You can assign fixed values or calculate posterior probability.

You can use a range of values or percentages on the input data, mid-point calculations and/or the end result. If you find an outlier you might alter your output to specify the percentage of outliers included or removed. If the data has a narrow or flat confidence you might want to include that disclaimer in the output, omitting it otherwise.

• "Using ensemble forecasts in probability and decision-making"

• "Ensemble Confidence Estimates Posterior Probability"

• "Ensemble Methods and Classifiers - Bagging, Boosting and Stacking" (.PDF)

• "Ensemble Methods for Classifiers"

Your output can be presented using a pyramid chart, Funnel Chart, or PowerPoint, other graphic/text based, or sophisticated 'Mind Map' style output. That should flexibly cover every situation from simple to complex. Calculations made with respect to the confidence, alerts, outliers, etc. can be used to choose one preferred chart style over another while you debug.

Your program ought to be formulated so it is always able to provide output that is suitable as a final result even in 'debug mode', (over what you originally had in mind). You don't want 'debug output' to provide an important result but have it be presentable to no one but yourself. The ability to save settings of Rules and have presets would make your program simple and flexible, and enable reproducible tests. Much as you would keep a copy of the input and output, keep a copy of the settings so they can be reloaded.

You want your program to work year after year, regardless of the input, providing useful output without having to tweak for each dataset if you want consistent results. You probably don't want it to appear as though the work was done by a different person every time.

Once you are finished the tool should 'just work', consistently. Here are examples of pyramid charts that can be automatically modified by the above techniques and the source code links provided below.

Examples for creating a Pyramid Chart in Python:

• ChartDirector 6.0 (Python Edition) - Simple Pyramid Chart

• How to Plot a Population Pyramid in Python

• PyGal Chart Library - pyramid

• Combination Pie/Pyramid Chart

Also consider JavaScript output to create a web-based dynamic chart.

EDraw - Free Mind Map Software