Error bars on sample variance

Question

I'm evaluating the performance of a statistical estimator under a number of parameter settings. The estimator is unbiased for all of the parameters, so I'm reporting the sample variance as a measure of quality, that is, If I am interested in approximating: $$ \int h(x)\pi(x)dx $$ Where $h(x) : \mathbb{R}^d \rightarrow \mathbb{R}$ and $\pi$ is a normalized distribution over $\mathbb{R}^d$. And I am using importance sampling with proposal distribution $q$ which gives samples $\{x_1, ..., x_n\}$. I have the estimator: $$ \bar{h} = \frac{1}{n}\sum_{i=1}^n h(x_i)\pi(x_i)/q(x_i) $$ The variance of $\bar{h}$ is given by: $$ \frac{1}{n}\mbox{var}(h(x)\pi(x)/q(x)) $$ I can approximate this variance with a test run of $m$ samples, and compute the sample variance: $$ \bar{v} = \frac{1}{m-1}\sum_{i=1}^m (h(x_i)\pi(x_i)/q(x_i)-\bar{h})^2 $$ Gives me the approximated value for the variance of my estimator for any number of samples $n$: $$ \mathcal{V} = \frac{1}{n}\bar{v} $$ Assuming I have not done anything stupid so far, if I'm reporting $\mathcal{V}$ as a way of judging the estimator I should but error bars on it, and I'm not sure how to do it.

edit: Actually this report is very useful, they show: $$ \mbox{var}(\bar{v}) = \frac{1}{m}(\mu_4 - \mu_2^2) + \mathcal{O}(m^{-2}) $$ Where $\mu_k$ is the $k$-th moment of the RV, so I assume I can just approximate these moments and plug them into this formula.

Answer 1

An example where the uncertainty in the sample variance is taken into account is the Student t distribution compared to the standard normal. If the variance were known you would use the Z test but if the sample estimate is substituted you use the t the critical values increase for the t and the difference between the t critical value and the Z gets small and goes to 0 as the degrees of freedom tend to infinity. This is going on with your case too but you have a more general problem. If you could get the exact distribution of your test statistic was the nuisance parameter (the standard deviation) is estimated you could get the exact amount to extend the error bars just as Gosset did. Unfortunately I don't think you can do this in general.