I have a set of $M$ variables $\{ \lambda_N^{(i)} \}_{i=0}^{M(N)}$. The distribution of $\lambda$ depends on the $N$ parameter; for $N \rightarrow \infty$ the distribution shrinks around the mean value, but for finite $N$ the distribution is characterized by the presence of outliers which determine a fat tails decays. The presence of this outliers gradually (and slowly) disappear increasing $N$. I want to verify if the shrinking of my distribution follow the theoretical prediction. So, starting from a set $\{ \lambda_N^{(i)} \}_{i=0}^{M(N)}$ I look for an estimate of the variance, which is the variable I want to study, and a confident interval for it.
So I have to repeat it for different values of the $N$ parameter (say $n$ times) and perform a fit with the $n$ couples $(N, Var_{\lambda_N})$.
My doubt is how to associate the uncertainties with my variables $Var_{\lambda_N}$
My first attempt was this:
starting from the set $\{ \lambda_N^{(i)} \}_{i=0}^{M(N)}$ I perform the 2 following transformation:

  • $\lambda_N \rightarrow w_N \equiv \lambda_N- \overline{\lambda_N}$, with $\overline{\lambda_N} = \frac{1}{M(N)} \sum_{i=0}^{M(N)} \lambda_N^{(i)} $
  • $w_N \rightarrow z_N \equiv w_N^2$
    With the distribution of $z_N$ I estimated $Var_{\lambda_N} = \overline{z_N}$ and the standard deviation of $Var_{\lambda_N}$ with the standard deviation of $z_N$.
    The problem with this approach is that, due to the outlier's presence the standard deviation's estimate is too large.
    My second attempt was to use bootstrap method:
    Starting from the set of $M(N)$ values I perform $n_r$ resampling (with repetition), each one with $n_s$ element.
    The problem with this approach is that the estimation of the confident interval (always for the presence of outliers) is strongly dependent on the choice of $n_s$
    In fact, using a big $n_s$'s value, I increase the probability to find at least one outlier in a generic resampled set, which deeply increase the variance estimation of that set.
    So, with big $n_s$ I have a small variability between different resampled set (small confident interval), while decreasing $n_s$ I reduce the probability to extract an outlier, increasing the variability between different resampled set (bigger confident interval).
    I don't want that my confidence interval's estimate depend on the choice of $n_s$.
    Is there any reasonable reasonable basis to determine the right choice of $n_s$?
    Should I use a different method in situations like that (jack knife or others)?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.