In many statistics courses, bootstrapping (and other random sampling with replacement methods) are suggested as ways to improve the confidence level in a statistic and improve our inference. Some even say it is a "powerful" method.
However it seems intuitively incorrect...
Say we this is our population (N = 15) : 1 - 3 - 3 - 4 - 5 - 1 - 2 - 3 - 4 - 1 - 2 - 2- 3 - 4 - 9
and this is our sample (n = 5 ) : 1 - 4 - 4 - 2 - 9
if we use the bootstrap method the 9 value has a 1/5 chance of being randomly selected each time where as in our population the 9 value only represents 1/15 of all values!
Bootstrapping can make us believe that the values we have in our sample are more frequent than they really are. Therefore it feels like there is a huge bias... Or am I missing something?
I have been looking for discussion/publications on this but I haven't found any, there seems to be a consensus on the fact that it's a powerful method, but I can't help but feel like this is a biased method that will make us overconfident of our sample.
This seems so obvious and simple that I can't imagine that all the statisticians never thought of that, so I'm guessing I'm just missing something quite elemental...