9
$\begingroup$

Would it be wrong to say that a Riemann sum approximation of an integral

\begin{align} \int_a^b f(t) \mathrm{d}t \approx \sum_{k=1}^{n_\text{samples}} f(t^{\ast}_k)\Delta t, \end{align}

where $\Delta t = \left(b - a\right)/n_\text{samples}$, and where $t^{\ast}_k$ is the left or right end point or the midpoint of the sub-intervals is an unbiased estimate of the true integral?

The argument for Monte Carlo integral approximation with $N$ uniform samples in the interval $[a,b]$ seems to be that as the number of samples goes to infinity, then the approximation will be the exact integral with probability 1 (see e.g https://cs.dartmouth.edu/wjarosz/publications/dissertation/appendixA.pdf, page 153)

This same limit for the Riemann sum is (the definition of) the Riemann integral, hence I would argue that a Riemann sum is also unbiased.

According to a blog post (https://blog.evjang.com/2016/09/riemann-bias.html) which I found on Google, then the Riemann sum is biased because of the deterministic steps.

But since the argument for the Monte Carlo integration being unbiased uses that $N$ goes to infinity, I can't see why the same argument can't be used for the Riemann sum approximation.

If it indeed is wrong that the Riemann sum is unbiased, I would be happy if anyone could explain the differences in the arguments.

$\endgroup$
0
13
$\begingroup$

It seems that you are swapping two different concepts here. The concepts are unbiased and consistent, which are properties of an estimator. A sequence of estimators $(T_n)_{n=1}^\infty$ is said to be unbiased for a quantity $\theta$ if, for all $n\,\in\mathbb{N}$,

$$ E[T_n] = \theta \quad.$$

It is said to be consistent if it converges in probability to $\theta$.

These are different concepts: the first says that, for every finite sample size, the average of your estimator is $\theta$. The other states that, as the sample sizes grow, the estimator getting arbitrarily close to $\theta$ with increasing probability.

Let $I = \int_a^bf(x)dx$ be your quantity of interest (assume it exists). What the most basic Monte Carlo method does is to observe that

$$I = \int_a^bf(x)dx = (b-a)\int_a^bf(x)\frac{1}{b-a}dx = (b-a)E[f(X)] \quad.$$

In the last line, we wrote the integral as being the expectation of $f(X)$, where $X$ has a uniform distribution in $(a,b)$. Hence, if we sample i.i.d. random variables $(X_i)_{i=1}^n$ with $X_1 \sim U((a,b))$, then the estimator

$$T_n = \frac{(b-a)}{n}\sum_{i=1}^nf(X_i) \quad,$$

is easily shown to be unbiased for $I$.

When you think of Riemann sums, it is usual to take a deterministic partition. If it is deterministic, then the expected value for any fixed sample size is the value of the summation it self, which in general is not the value of the integral.

$\endgroup$
4
  • $\begingroup$ Thanks for the detailed explanation. $\endgroup$ Jul 24 at 13:13
  • 2
    $\begingroup$ "for every finite sample size" — choose a sample of size 1. The average of this sample must be $\theta$. This then implies that every element must equal $\theta$, thus an unbiased estimator is an exact computer of $\theta$. Is this really what you supposed to define? $\endgroup$
    – Ruslan
    Jul 24 at 21:50
  • 4
    $\begingroup$ @Ruslan The "sample size" $n$ here is not the set over which the average (that must equal $\theta$) is taken. The estimator is computed from the entire sample, and what must equal $\theta$ is its average (expectation) over all possible samples of size $n$. For example, an unbiased estimator of the mean from a sample of size $1$ is just the sample value. This does not always equal the true mean $\theta$, but its expectation value over samples of size $1$ equals $\theta$. $\endgroup$
    – nanoman
    Jul 25 at 0:47
  • $\begingroup$ As @nanoman has well said, the expectation is not the average over a fixed sample. The expectation is taken over the process that generates samples of size $1$. What is fixed is the sample size $n$, not the sample values. $\endgroup$ Jul 25 at 15:34
6
$\begingroup$

Any constant is a biased estimator of any different constant

Since you are using a deterministic procedure here, your Riemann sum depends only on $n$, so it is a sequence of constants. Applying the concept of statistical bias to constants is simple --- any constant is a biased estimator of any different constant and an unbiased estimator of itself. So, for example, $3$ is a biased estimator of $2$, but it is an unbiased estimator of $3$.

Your Riemann sum is generally going to be a biased estimator for the corresponding integral because you are selecting the points $t_k^*$ deterministically within the interval and so your estimator is a constant. There are exceptions, for functions where the Riemann sum happens to be exactly equal to the integral (e.g., piecewise linear functions). When the Riemann sum gives a different value to the integral it is biased (in the same way that $3$ is a biased estimator of $2$). When the Riemann sum gives the same value as the integral it is unbiased (in the same way that $3$ is an unbiased estimator of $3$). Irrespective of whether the Riemann sum is a biased estimator or not, it will still be a consistent estimator, since it converges to the true integral as $n \rightarrow \infty$; indeed, this is the essence of the Riemann integral.

Now, if you were to select the $t$ point uniformly at random within the interval, the resulting Riemann sum would be an unbiased estimator of the integral. This would be a variation of estimation by importance sampling, where you are varying things by generating your points conditionally within segments of a partition.

$\endgroup$
11
  • 2
    $\begingroup$ What definition or sense of "biased" are you invoking here?? $\endgroup$
    – whuber
    Jul 24 at 14:18
  • 1
    $\begingroup$ Sometimes it will be equal, sometimes not, depending on the integrand, the intervals, and the kind of Riemann sum. But since there's no probability model in evidence, it doesn't look like your "biased" has the same meaning it does in estimation theory. $\endgroup$
    – whuber
    Jul 25 at 14:28
  • 1
    $\begingroup$ I'm not really trying to make any particular point --- just to answer the question that was asked. My answer would be largely the same if the OP had asked whether $3$ is a biased estimator of $2$. Re the exceptions, I already conceded there are exceptions, both in the question and again in my above comment, so why continue to approach the matter as if I don't know this? When you have a mathematical proposition $P(x)$ that is true for some $x$ and false for (many more) others, and someone asks "Is it wrong to say "$P(x)$ is true in general", you tell them, yes, generally that is wrong. $\endgroup$
    – Ben
    Jul 26 at 21:53
  • 1
    $\begingroup$ Also, I reject your characterisation of "shoehorning" --- all constants are special cases of random variables (when appropriately defined as mappings from the sample space to that constant), so all statistical procedures that apply to random variables should apply to constants (even when this renders them trivial). This is an important check on working for any statistical procedure --- i.e., does it make sense for constants. In this case, any constant is a biased estimator of another different constant, so yes, it does. $\endgroup$
    – Ben
    Jul 26 at 21:57
  • 1
    $\begingroup$ Rather than leave all this in the comments, I have decided to flesh out my answer to include this material in the body. $\endgroup$
    – Ben
    Jul 26 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.