Goal: I would like to find a way to compare the feature importance score (e.g.,"coef" values for SVM in scikit learn) for a feature (say "weight") for dataset A with the feature importance score for the same feature (i.e. "weight") for dataset B. For the below feature importance scores, for example, I want to be able to say that "weight" is significantly more important in dataset A than in dataset B. And along similar lines, "education" has similar importance in dataset A and dataset B. Dataset A could be from City A, and Dataset B could be from City B.
features Dataset A Dataset B weight 99.9 11.1 height 22.2 88.8 education 33.3 44.4
Note that I have used the same model for both datasets, i.e. same algorithm (e.g., SVM with linear kernel) and same hyper-parameters.
Question 1: I suppose that first and foremost - can the feature importance scores between two datasets be compared and how can I check that? I assumed that since I used the same model for both datasets that they can, but not actually sure.
Question 2: If the scores can indeed be compared, how should I do it? What I have in mind is to normalise the scores across both datasets to values between 0 and 1, and then compare these normalised values. Using the above example, 99.9 would become 1 and 11.1 would become 0 after normalisation. I could then do this with different algorithms (or multiple times with same algorithm?) and do a t-test to see if the difference is significant...?
Sorry if my questions are convoluted. I hope that I have made them clear enough. Thank you very much for any help or advice!
Note: I read somewhere that AUC can be used to judge if two datasets are comparable (https://cran.r-project.org/web/packages/datarobot/vignettes/ComparingSubsets.html), not sure if anyone can provide any insight on this?