1
$\begingroup$

I'm looking at feature importance. I have a set of point estimates for features I would like to rank from two different models. I also have their confidence intervals.

What is the best way to rank these point estimates while also accounting for the confidence intervals?

I don't think it makes sense to rank point estimates by averaging their magnitude without accounting for the confidence intervals.

EDIT:

Following something similar to this (in the feature importance section where they compare different model feature importance outputs): https://www.r-bloggers.com/iml-and-h2o-machine-learning-model-interpretability-and-feature-explanation/

$\endgroup$
2
  • $\begingroup$ There might be some insights to be gained from meta-analysis which accounts for precision of individual estimates but I do not know how that fits into the machine learning world. $\endgroup$
    – mdewey
    Aug 5, 2020 at 14:31
  • $\begingroup$ It sounds similar to the problems of ranking sets of ratings where the ratings are uncertain (e.g. consider ratings out of 5 where one item has two ratings both of 5 stars while another has 100 ratings but an average of 4.8 stars (a few of the ratings were 4 stars, all the rest were 5 stars). Should the item with only two ratings really be regarded as the better of the two? If thats relevant to you, there are several questions on site related to that sort of problem and a search should turn some if them up. $\endgroup$
    – Glen_b
    Aug 5, 2020 at 17:54

2 Answers 2

0
$\begingroup$

Have you considered ranking them on the absolute value of their $t$-statistics? Or is there something deeper that I missed?

$\endgroup$
4
  • $\begingroup$ t statistics for what tests? $\endgroup$
    – BruceET
    Aug 5, 2020 at 4:56
  • $\begingroup$ $t$-statistics for your point estimates. Your confidence intervals are usually the point estimate $\pm$ some number of standard errors of the estimates. The $t$-statistics are the points estimates divided by those standard errors. $\endgroup$
    – kurtosis
    Aug 5, 2020 at 6:51
  • 1
    $\begingroup$ Let's say I do a permutation feature importance for two different models. I get their point estimates and confidence intervals for both models. Averaging their point estimates make sense to rank. But does it make sense to average the confidence intervals as well? $\endgroup$
    – Eisen
    Aug 5, 2020 at 12:28
  • $\begingroup$ I don't think you can average those CIs because they are not from the same model and multicollinearity issues could cause them to behave oddly when that amount of uncertainty would not be present in a model with all features. Also... I would never jump to using machine learning concepts before trying something simple from statistics that has been studied in far more depth. $\endgroup$
    – kurtosis
    Aug 5, 2020 at 19:27
0
$\begingroup$

If the goal is to rank the estimates according to the magnitude of the point estimates, that seems simple. It is unclear from what you say how or why the the confidence intervals (CIs) should be involved in the ranking.

If the concern is that some CIs are longer than others, so that the order of ranking of the true population means might be different, you can make a graph to show the lengths of the CIs as well as the point estimates to make it clear that there are some overlaps.

Here are five datasets along with their point and interval estimates. The five population standard deviations are all the same, but the sample sizes are different, so the lengths of the CIs also differ. [Sampling, computations, and graphics in R.]

# sample data
set.seed(804)
x1 = rnorm(30, 50, 5)
x2 = rnorm(25, 55, 5)
x3 = rnorm(35, 60, 5)
x4 = rnorm(30, 65, 5)
x5 = rnorm(35, 70, 5)

# compute means
a1 = mean(x1)
a2 = mean(x2)
a3 = mean(x3)
a4 = mean(x4)
a5 = mean(x5)
a = c(a1,a2,a3,a4,a5);  a
[1] 49.90262 55.30521 58.99290 65.29689 70.16361

# use t test to compute t confidence intervals
CI1 = t.test(x1)$conf.int
CI2 = t.test(x2)$conf.int
CI3 = t.test(x3)$conf.int
CI4 = t.test(x4)$conf.int
CI5 = t.test(x5)$conf.int
CI = cbind(CI1, CI2, CI3, CI4, CI5);  CI
          CI1      CI2      CI3      CI4      CI5
[1,] 47.57335 53.78918 57.36025 63.37359 68.36907
[2,] 52.23188 56.82125 60.62555 67.22020 71.95815

# make plot
mn = min(CI[1,]); mx = max(CI[2,])
plot(1:5, a, ylim=c(mn,mx), pch=19)
for (i in 1:5) {
 lines(c(i,i), CI[,i], col="red", lwd=2) }

enter image description here

There is some slight overlapping of CIs among the samples, but the ranking of the point estimates seems clear. If you use this style of graphic, be sure to make it clear that the red lines are CIs. [In some fields similar plots are used to display standard deviations, standard errors, or other descriptive statistics.]

In R, the boxplot procedure will show boxplots of a number of samples with 'notches' in the sides of the boxes showing nonparametric confidence intervals for population medians. [These CIs are calibrated for comparing two samples at a time for significant differences.]

boxplot(x1,x2,x3,x4,x5, col="skyblue2", pch=19, notch=T)

enter image description here

$\endgroup$
2
  • $\begingroup$ I've edited my question to make it more clear. I think the problem is that I have two models which I want to use to estimate feature importances for the data. I'm averaging the point estimates from both models, but I don't think it makes sense to not consider the confidence intervals. For example one feature could have a large confidence interval and their is high uncertainty for how it should be rank. I want to consider those in rankings. $\endgroup$
    – Eisen
    Aug 5, 2020 at 12:46
  • $\begingroup$ You have changed your question--but, for me, not clarified it. You still have two potentially conflicting attributes, sizes of point estimates and lengths of CIs, with no clear criterion how to reconcile them. Seems wrong not to show both both point estimate and CI in a graph or chart. // Maybe rank tops (or bottoms) of CIs. What is the purpose of making a ranking? // How would you rank what kind of animal makes the best pet (maybe among: cat, dog, hamster, iguana, parakeet, parrot)? $\endgroup$
    – BruceET
    Aug 5, 2020 at 15:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.