I already have a classification model that performs pretty accurately. I received some feedback from an expert regarding the importance of features as he foresees. I need to somehow use this feedback to refine the model. The goal is to test if the expert view can help improve the model or not.
For example, there were features about users' page activities. In the table below, each cell represents one feature (e.g., number of entries in Page3). The expert was asked to rank these features based on their importance and relevance for the prediction task.
Page1 Page2 Page3 Page4 number of visits number of entries number of distinct visit-days
The ranking was 0 [not important] to 5 [very important]. Here are example results:
Page1 Page2 Page3 Page4 number of visits 4 3 5 1 number of entries 3 2 4 5 number of distinct visit-days 1 3 2 1
Similarly, the expert rank the importance and relevance of each page with regard to the prediction task:
Rank 1: Page2 Rank 2: Page4 Rank 3: Page1 Rank 4: Page3
I need to incorporate these information into my model to test the effectiveness of expert opinion in improving the model. One option could be configuring the weights of features in the model based on the ranking provided by the expert. I know that this is not recommended as the algorithm automatically finds the perfect weights. However, my goal is more experiment-oriented rather than improving the model. I wonder how you would approach this task. Any recommendations? (Note that it could be any classifier and that I prefer a solution with scikit-learn).