Average Point Estimates but also considering confidence intervals I'm looking at feature importance. I have a set of point estimates for features I would like to rank from two different models. I also have their confidence intervals.
What is the best way to rank these point estimates while also accounting for the confidence intervals?
I don't think it makes sense to rank point estimates by averaging their magnitude without accounting for the confidence intervals.
EDIT:
Following something similar to this (in the feature importance section where they compare different model feature importance outputs):
https://www.r-bloggers.com/iml-and-h2o-machine-learning-model-interpretability-and-feature-explanation/
 A: Have you considered ranking them on the absolute value of their $t$-statistics? Or is there something deeper that I missed?
A: If the goal is to rank the estimates according to the magnitude of the point
estimates, that seems simple. It is unclear from what you say how or why the the confidence intervals (CIs)
should be involved in the ranking.
If the concern is that some CIs are longer
than others, so that the order of ranking of the true population means
might be different, you can make a graph to show the lengths of the CIs as
well as the point estimates to make it clear that there are some overlaps.
Here are five datasets along with their point and interval estimates.
The five population standard deviations are all the same, but the
sample sizes are different, so the lengths of the CIs also differ.
[Sampling, computations, and graphics in R.]
# sample data
set.seed(804)
x1 = rnorm(30, 50, 5)
x2 = rnorm(25, 55, 5)
x3 = rnorm(35, 60, 5)
x4 = rnorm(30, 65, 5)
x5 = rnorm(35, 70, 5)

# compute means
a1 = mean(x1)
a2 = mean(x2)
a3 = mean(x3)
a4 = mean(x4)
a5 = mean(x5)
a = c(a1,a2,a3,a4,a5);  a
[1] 49.90262 55.30521 58.99290 65.29689 70.16361

# use t test to compute t confidence intervals
CI1 = t.test(x1)$conf.int
CI2 = t.test(x2)$conf.int
CI3 = t.test(x3)$conf.int
CI4 = t.test(x4)$conf.int
CI5 = t.test(x5)$conf.int
CI = cbind(CI1, CI2, CI3, CI4, CI5);  CI
          CI1      CI2      CI3      CI4      CI5
[1,] 47.57335 53.78918 57.36025 63.37359 68.36907
[2,] 52.23188 56.82125 60.62555 67.22020 71.95815

# make plot
mn = min(CI[1,]); mx = max(CI[2,])
plot(1:5, a, ylim=c(mn,mx), pch=19)
for (i in 1:5) {
 lines(c(i,i), CI[,i], col="red", lwd=2) }


There is some slight overlapping of CIs among the samples, but
the ranking of the point estimates seems clear. If you use this style of
graphic, be sure to make it clear that the red lines are CIs.
[In some fields similar plots are used to display standard deviations,
standard errors, or other descriptive statistics.]
In R, the boxplot procedure will show boxplots of a number of
samples with 'notches' in the sides of the boxes showing nonparametric
confidence intervals for population medians. [These CIs are calibrated
for comparing two samples at a time for significant differences.]
boxplot(x1,x2,x3,x4,x5, col="skyblue2", pch=19, notch=T)


