# Estimating Confidence in Feature Rankings from Multiple Experiments with Non-Normal Data

Hello dear Cross Validated Community,

I am a new doctoral student in bioinformatics, and I am working on a project involving multiple experiments, each generating a single numerical result for each of 7000 features.

The goal is to find out which are the 20 features of all 7000 features that have the highest value.

We are repeating those experiments multiple times to ensure that the results are backed up by statistics. Therefore, each feature is characterized by the mean of all numerical results of that feature for each experiment. Based on that mean, the ranking of the features is carried out, and as said above, only the 20 with the highest mean are of interest. Each experiment is expensive, so we want to minimize the number of experiments.

My focus is to determine how many such experiments have to be carried out until we have a given confidence that this ranking of the top 20 will not change anymore (so that lets say feature x stays at rank 3 and feature y at rank 4, and they will not switch places if we do more experiments).

The challenge is that the values for each feature are not normally distributed. Rather, some values obtained are almost zero (e.g. 10^(-6)) while others are much bigger (e.g. 0.03). This non-normality led me to choose non-parametric methods for my analysis (plot for one distribution at the end of the post)

## Approach Taken:

Bootstrap Method for Confidence Intervals:

To assess the reliability of the feature means, I used bootstrap resampling to calculate 99% confidence intervals.

1. Resampling: For each bootstrap iteration (n_bootstrap = 10,000), resample the data with replacement to create a new sample of the same size as the original data.

2. Mean Calculation: Calculate the mean of each resampled dataset.

3. Confidence Interval Determination: Determine the desired confidence interval (e.g. 99%) by calculating the relevant percentiles from the bootstrap means—specifically, the lower and upper bounds using the specified confidence level.

Permutation Tests for Pairwise Comparisons:

For the top 20 features, based on their mean values, I performed pairwise permutation tests to evaluate the statistical significance of the differences between their means. This was done for each pair within the top 20 (e.g., 1 vs 2, 1 vs 3, ..., 19 vs 20), resulting in p-values that help infer significant differences between the feature rankings.

The algorithm would be like this:

1. Calculate mean of both features of the pairwise comparison
2. Calculate difference of those two means​
3. Put both groups together --> make one big group​
4. Shuffle (permutate) the big group, and recreate the two individual
groups by splitting shuffled big group​
5. Calculate individual means and diff between means (with consistent
order, for example have a group 1 and 2 and give them their values, and always do group 1 mean minus group 2 mean (so these mean diffs can also be negative)​
6. Repeat 10.000 times and write down all permutation mean diffs​
7. p-value = proportion of permutation mean diffs "extremer" (bigger or smaller) than mean diff original groups (both sides)

## Questions:

Are these methods—bootstrap for confidence intervals and permutation tests for pairwise comparisons—appropriate for this problem and my specific goal of getting some metrics which tell me how "stable" the ranks of the top 20 features are (or if the ranks will shift when we do some more experiments)? Are they accepted by the statistical community (because those methods will potentially go into a publication, and they have to hold in front of reviewers).

Any insights, suggestions, or criticisms would be greatly appreciated.

I would appreciate your help very much! Thank you!

(If anything is not clear, forgive me. I will try to make it clearer then)

Here the data distribution of one exemplary feature for 20 experiments (the x-axis is just experiment 1, 2, 3, etc., sorry for bad quality).