I perform some analysis (with the goal to select some relevant variables that are related between two experiments) on a group of samples (n = 158) but I would like to know how robust are, before going to another cohort to validate it I thought of bootstrap.
What I understand from bootstrap is that the assumption is that the samples are randomly taken from the population. However in my case it isn't, the samples are not randomly selected:
- Some are from diseased people and some are from healthy people (double samples from diseased than from healthy).
- They can be from different regions (up to 7 locations) with uneven distribution (several reasons why the distribution is not even between location I can't assume anything).
- It is a treatment and I have a longitudinal data with three important time points. Although I have several time points from some patients and just one for some patients. From the healthy samples I only have one time point.
- Age is also an important factor (the healthy samples are older than the diseased) and some of the patients are followed up to 2 years.
- 70% of the samples are from females.
I don't know how to bootstrap while keeping the structure of the cohort (if I need to keep the cohort structure):
- I used the leave-one-out strategy but I received some criticism (by some professor I contacted and my thesis advisor) because then I have to few variability.
- If I use random resampling I might get too many samples of one condition (say healthy samples).
- I read about stratified bootstrapping but I don't know how to stratify in this case (if I had only one category I could but with several categories...)
My current approach is to calculate the probability to get a sample of these conditions on each category (P(Sample) = P(location)*P(status)P(sex)...) and then multiply it with the draws from uniform Dirichlet distribution (following the Bayesian bootstrap). Use these probabilities to select the samples (in the linked blogpost from 43 data points they take 1000 samples in the bootstrap).
However, I don't know how to select the bootstrap size, it could be random or simply fixed but after reading "Determining sample size necessary for bootstrap" I'm not longer sure about the it. The answer there suggest selecting the size if the samples show consistency of the estimate. However here I want to test if there is consistency on the variables selected to estimate the relation between the experiments.
How do I perform bootstrapping with not random samples?