2
$\begingroup$

I'm evaluating the performance of a statistical estimator under a number of parameter settings. The estimator is unbiased for all of the parameters, so I'm reporting the sample variance as a measure of quality, that is, If I am interested in approximating: $$ \int h(x)\pi(x)dx $$ Where $h(x) : \mathbb{R}^d \rightarrow \mathbb{R}$ and $\pi$ is a normalized distribution over $\mathbb{R}^d$. And I am using importance sampling with proposal distribution $q$ which gives samples $\{x_1, ..., x_n\}$. I have the estimator: $$ \bar{h} = \frac{1}{n}\sum_{i=1}^n h(x_i)\pi(x_i)/q(x_i) $$ The variance of $\bar{h}$ is given by: $$ \frac{1}{n}\mbox{var}(h(x)\pi(x)/q(x)) $$ I can approximate this variance with a test run of $m$ samples, and compute the sample variance: $$ \bar{v} = \frac{1}{m-1}\sum_{i=1}^m (h(x_i)\pi(x_i)/q(x_i)-\bar{h})^2 $$ Gives me the approximated value for the variance of my estimator for any number of samples $n$: $$ \mathcal{V} = \frac{1}{n}\bar{v} $$ Assuming I have not done anything stupid so far, if I'm reporting $\mathcal{V}$ as a way of judging the estimator I should but error bars on it, and I'm not sure how to do it.

edit: Actually this report is very useful, they show: $$ \mbox{var}(\bar{v}) = \frac{1}{m}(\mu_4 - \mu_2^2) + \mathcal{O}(m^{-2}) $$ Where $\mu_k$ is the $k$-th moment of the RV, so I assume I can just approximate these moments and plug them into this formula.

$\endgroup$
  • $\begingroup$ the variance of $\bar{h}$ is dependant on the number of samples you take, but it's producted with a constant quantity, which is the variance of a single sample from the estimator, which is what $\bar{v}$ approximates $\endgroup$ – fairidox May 31 '12 at 17:40
  • 1
    $\begingroup$ I still do not get the role of $m$. $\endgroup$ – user10525 May 31 '12 at 17:55
  • $\begingroup$ it's an arbitrary constant of little consequence, I mean to use it to answer the question of "If I had computational budget of $m$ samples how well does this estimator do", it ended up there because some of my methods actually have higher "cost", a detail I left out. $\endgroup$ – fairidox May 31 '12 at 18:09
  • $\begingroup$ Thanks for the clarification. I believe that if $n$ is large enough (which should be the case), the variance is going to be quite small (this follows from the expression for the variance you present). If $\mbox{var}(h(x)p(x)/q(x))$ is not reasonably small, this indicates that the importance function $q$ is not really efficient, I think. $\endgroup$ – user10525 May 31 '12 at 18:15
  • 1
    $\begingroup$ @anonymous_4322 I am quite sure Procrastinator was referrring to the variance of the sample variance estimate and not the sample variance itself. $\endgroup$ – Michael Chernick May 31 '12 at 19:56
2
$\begingroup$

An example where the uncertainty in the sample variance is taken into account is the Student t distribution compared to the standard normal. If the variance were known you would use the Z test but if the sample estimate is substituted you use the t the critical values increase for the t and the difference between the t critical value and the Z gets small and goes to 0 as the degrees of freedom tend to infinity. This is going on with your case too but you have a more general problem. If you could get the exact distribution of your test statistic was the nuisance parameter (the standard deviation) is estimated you could get the exact amount to extend the error bars just as Gosset did. Unfortunately I don't think you can do this in general.

$\endgroup$
  • 1
    $\begingroup$ yes, the distribution of the statistic is certainly not of a standard form, and as far as I know there is no central limit theorem equivalent for sample variance. I've seen derivations for the variance of the variance of a RV, what I suppose I need is a sample variance of the sample variance... $\endgroup$ – fairidox May 31 '12 at 19:26
  • $\begingroup$ The variance of a variance estimate can under normality assumptions be figured out based on the variance of a chi square random variable. It is proportional to the 4th power of the standard deviation. As the sample variance is the average of slightly dependent chi square random variables there is a central limit theorem that applies to it. It also converges in probability to the population variance (I think the convergence is even almost sure convergence). $\endgroup$ – Michael Chernick May 31 '12 at 19:50
  • $\begingroup$ ok, thanks, that is helpful and I think consistent with my edits made at the same time you commented :) $\endgroup$ – fairidox May 31 '12 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.