What are the ways to use expert opinion to refine or adjust a classification model?

Question

I already have a classification model that performs pretty accurately. I received some feedback from an expert regarding the importance of features as he foresees. I need to somehow use this feedback to refine the model. The goal is to test if the expert view can help improve the model or not.

For example, there were features about users' page activities. In the table below, each cell represents one feature (e.g., number of entries in Page3). The expert was asked to rank these features based on their importance and relevance for the prediction task.

                               Page1   Page2   Page3   Page4
number of visits
number of entries
number of distinct visit-days

The ranking was 0 [not important] to 5 [very important]. Here are example results:

                               Page1   Page2   Page3   Page4
number of visits                   4       3       5       1
number of entries                  3       2       4       5
number of distinct visit-days      1       3       2       1

Similarly, the expert rank the importance and relevance of each page with regard to the prediction task:

Rank 1: Page2
Rank 2: Page4
Rank 3: Page1
Rank 4: Page3

I need to incorporate these information into my model to test the effectiveness of expert opinion in improving the model. One option could be configuring the weights of features in the model based on the ranking provided by the expert. I know that this is not recommended as the algorithm automatically finds the perfect weights. However, my goal is more experiment-oriented rather than improving the model. I wonder how you would approach this task. Any recommendations? (Note that it could be any classifier and that I prefer a solution with scikit-learn).

Answer 1

I think it depends on what you are using as a classifier. It would be straight forward to incorporate these directly in a Bayesian model (and there is a rich literature on incorporating expert opinion as informative priors), but if you're using something more black-boxy (e.g. random forest) it will be less straight forward.

This post (http://andrewgelman.com/2017/09/20/using-black-box-machine-learning-predictions-inputs-bayesian-analysis/) from Andrew Gelman comes to mind as a possible starting point.

Answer 2

Obtaining features from an expert is a great way to incorporate knowledge. I would have take it one step further and ask the expert to propose a model. A small decision tree is a very "human friendly" model to build. Usually you find out that the model performs quite well (since the expert is indeed such), but not too good.

Then you can use the expert model and analyze with him the errors - examples of false positives and false negatives. In this phase it is common to hear "Sure, I forgot that..."

Once you have a model you can use it as the initial classifier and improve it performance using boosting, making your ML focus on the ares of lower performance.

I think that a ranking of the features is less useful (e.g., a feature is the fifth most useful. Does it make it useful or not?) I'd go back to my expert and try the process above.