0
$\begingroup$

Goal: I would like to find a way to compare the feature importance score (e.g.,"coef" values for SVM in scikit learn) for a feature (say "weight") for dataset A with the feature importance score for the same feature (i.e. "weight") for dataset B. For the below feature importance scores, for example, I want to be able to say that "weight" is significantly more important in dataset A than in dataset B. And along similar lines, "education" has similar importance in dataset A and dataset B. Dataset A could be from City A, and Dataset B could be from City B.

features    Dataset A    Dataset B
weight      99.9         11.1
height      22.2         88.8
education   33.3         44.4

Note that I have used the same model for both datasets, i.e. same algorithm (e.g., SVM with linear kernel) and same hyper-parameters.

Question 1: I suppose that first and foremost - can the feature importance scores between two datasets be compared and how can I check that? I assumed that since I used the same model for both datasets that they can, but not actually sure.

Question 2: If the scores can indeed be compared, how should I do it? What I have in mind is to normalise the scores across both datasets to values between 0 and 1, and then compare these normalised values. Using the above example, 99.9 would become 1 and 11.1 would become 0 after normalisation. I could then do this with different algorithms (or multiple times with same algorithm?) and do a t-test to see if the difference is significant...?

Sorry if my questions are convoluted. I hope that I have made them clear enough. Thank you very much for any help or advice!

Note: I read somewhere that AUC can be used to judge if two datasets are comparable (https://cran.r-project.org/web/packages/datarobot/vignettes/ComparingSubsets.html), not sure if anyone can provide any insight on this?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.