2
$\begingroup$

I am currently conducting a simulation study in which there is a known true function $F(t)$ which is strictly decreasing and continuous over time. I want to know if a particular estimator $\hat{F}(t)$ is unbiased over some time interval $0$ to $\tau$. The estimator $\hat{F}(t)$ produces step functions over the interval of interest.

My question is: How do I figure out if the estimator $\hat{F}(t)$ produces unbiased estimates of the true function $F(t)$ in the range of $0$ to $\tau$ using a simulation study?

My first Idea was to simply calculate the integral of the difference between the true function and the estimated function for a big number of simulation runs. That quantity would be $Bias = \int^{\tau}_{0} (F(t) - \hat{F}(t))$. In the case of a truly unbiased estimator, this quantity clearly converges to 0 over many simulation runs. The Mean-Squared-Error could be estinated in a similar fashion using $MSE = \int^{\tau}_{0} (F(t) - \hat{F}(t))^2$. I soon realized however that there are situations in which the first quantity converges to 0 even when the estimator is biased. The (very badly drawn) graph below shows what I am talking about.

enter image description here

Case A shows an unbiased estimate. The step function wiggles around the true curve. Case B shows a biased estimate. The step function is always below the true curve. Now the problem comes with Case C. Here the integral would be approximately 0, since the positive bias in the first half of the time interval cancels out the negative bias in the second half of the interval.

I would be very thankfull for any ideas.

$\endgroup$
8
  • $\begingroup$ Seems to me you want to quantify goodness of fit rather than bias. So maybe MSE should be your measure. $\endgroup$
    – Russ Lenth
    Jul 6, 2021 at 13:23
  • $\begingroup$ Well the goodness-of-fit is an important part as well, but I also want to know if the estimator is unbiased. Any ideas what kind of measure I could use to assess the bias? $\endgroup$
    – Denzo
    Jul 6, 2021 at 13:30
  • $\begingroup$ What you did is what assesses bias. But you're not happy with it, hence my comment. I suppose you could look at other aspects of it, e.g. low-order polynomial effects of the errors. The intercept would show up in case B, the linear effect would show up in case C. $\endgroup$
    – Russ Lenth
    Jul 6, 2021 at 13:34
  • $\begingroup$ The bias is $\mathbb E[\hat F(t)]-F(t)$ for a given $t$. If $\hat F(\cdot)$ is unbiased over the interval $(0,\tau)$, the quantity$$\max_{t\in (0,\tau)}|\mathbb E[\hat F(t)]-F(t)|$$should be equal to zero. I do not think there exists an unbiased estimator of $F(\cdot)$ unless the function enjoys special properties. $\endgroup$
    – Xi'an
    Jul 6, 2021 at 13:47
  • $\begingroup$ @RussLenth that does sound like a good idea and I will definitly play around with it. Thanks a lot! $\endgroup$
    – Denzo
    Jul 6, 2021 at 14:19

2 Answers 2

0
$\begingroup$

To compare a biased with an unbiased estimate of $F(\cdot)$, consider the case of a complement cdf,$$F(t)=\mathbb P(X\ge t)$$ which is strictly decreasing in $t\in(0,\tau)$ when $X$ is a continuous random variable with support $(0,\tau)$. Then, given a sample $X_1,\ldots,X_n$ associated with the cdf $1-F(\cdot)$ the step function $$\hat F(t) =\frac{1}{n}\sum_{i=1}^n \mathbb I_{x_i\ge t}$$ is unbiased, since $$\mathbb E_F[\mathbb I_{X_i\ge t}]=\mathbb P(X_i\ge t)=F(t)$$ Here is an illustration with

  • $n=10$ points in the sample
  • $\tau=1$
  • $F$ the complement cdf for the standard normal
  • $M=10^5$ repetitions
  • $100$ plots of $\hat F$
  • the graph of the average of the $M$ $\hat F$'s (red)
  • the graph of the true $F$ (blue)

enter image description here

showing they are indistinguishable, i.e. the expectation of $\hat F$ is $F$. In the opposite, if using the biased step function $$\hat F(t) =\frac{1}{n}\sum_{i=1}^n \varphi(x_i)\mathbb I_{x_i\ge t}$$ instead, the bias is clear:

enter image description here

$\endgroup$
2
  • 1
    $\begingroup$ Thank you for the detailed answer (I upvoted it but I don't have enough reputation for it to be shown). I have created similar plots as well and they indeed are very helpfull for judging the bias. The problem with this in my case however is that I have a huge amount of different simulation scenarios that I need to cover, which is the reason why I am looking for a single statistic to judge the bias (and goodness-of-fit). I am definitly gonna try the statistic you proposed in your original comment. Thank you again for the help. $\endgroup$
    – Denzo
    Jul 6, 2021 at 17:50
  • $\begingroup$ Thanks. I am surprised that you cannot vote up or down a question as the originator of the question. Not that it matters, but you can check the answer if you wish..! $\endgroup$
    – Xi'an
    Jul 10, 2021 at 17:02
0
$\begingroup$

Like I suggested in a comment, a set of bias effects, say polynomial ones), might be useful. That is, $i$th degree bias would be defined as $$ B_i = \int_0^\tau p_i(t)\cdot(\hat F(t) - F(t))dt $$ where $p_i(t)$ is a suitable basis function. [I think you should subtract in the other direction than in the OP; for example, your original bias measure would come out positive in case B, even though the estimate is negatively biased.]

To keep these measures independent of one another, you should orthogonalize them. For orthogonal polynomials, define $h_i(t) = t^i$, $i=1,2,\ldots,n$ and construct these functions: \begin{eqnarray*} p_0(t) &=& 1\\ p_i(t) &=& h_i(t) - \sum_{j=0}^{i-1} a_{ij}p_j(t) \qquad i=1,2,\ldots \end{eqnarray*} where $$ a_{ij} = \int_0^\tau h_i(t)p_j(t)dt $$ If I did this right (i.e., the Gram-Schmidt procedure), this ensures that $\int p_i(t)p_k(t)=0$ for any $i\ne k$.

You could construct another set of bases using some other set of $h_i$s. A useful idea might be for $h_i(t)$ to be a function that jumps from 0 to 1 at abscissae of odd multiples of $\tau/(2i)$, and from 1 to 0 at even multiples. Those create a kind of wavelet basis.

Or (duh!) a Fourier basis: $p_i(t) = \cos\left(\frac{2\pi}\tau it\right)$, which are already orthogonal (where $i$ is the index $i$, not $\sqrt{-1}$!)

$\endgroup$
1
  • $\begingroup$ Thank you a lot! I will definitly try it out. It sounds very promising. $\endgroup$
    – Denzo
    Jul 7, 2021 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.