Goal: I would like to find a way to compare the feature importance score (e.g.,"coef" values for SVM in scikit learn) for a feature (say "weight") for dataset A with the feature importance score for the same feature (i.e. "weight") for dataset B. For the below feature importance scores, for example, I want to be able to say that "weight" is significantly more important in dataset A than in dataset B. And along similar lines, "education" has similar importance in dataset A and dataset B. Dataset A could be from City A, and Dataset B could be from City B.

features    Dataset A    Dataset B
weight      99.9         11.1
height      22.2         88.8
education   33.3         44.4

Note that I have used the same model for both datasets, i.e. same algorithm (e.g., SVM with linear kernel) and same hyper-parameters.

Question 1: I suppose that first and foremost - can the feature importance scores between two datasets be compared and how can I check that? I assumed that since I used the same model for both datasets that they can, but not actually sure.

Question 2: If the scores can indeed be compared, how should I do it? What I have in mind is to normalise the scores across both datasets to values between 0 and 1, and then compare these normalised values. Using the above example, 99.9 would become 1 and 11.1 would become 0 after normalisation. I could then do this with different algorithms (or multiple times with same algorithm?) and do a t-test to see if the difference is significant...?

Sorry if my questions are convoluted. I hope that I have made them clear enough. Thank you very much for any help or advice!

Note: I read somewhere that AUC can be used to judge if two datasets are comparable (https://cran.r-project.org/web/packages/datarobot/vignettes/ComparingSubsets.html), not sure if anyone can provide any insight on this?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.