After reading this post, I still don't thoroughly understand what an estimator is. Suppose the samples $D_i={(x_1,y_1),...(x_n,y_n)}$ are drawn randomly from function $$f(x)=sin(2\pi x),$$ so my ultimate goal is to come up with an estimated function $h(x)$, which should approximate $f$ as close as possible, right? Since I don't know what the true $f$ looks like, so I might choose different models to do the approximation, here I consider the linear regression model, which is $$h(x)=\beta^Tx,$$ then I try to estimate the model parameters $\beta$ via OLS, and finally given training sample $D_i$, I have the estimates $\hat\beta_{|D_i}$, and absolutely dffierent $\hat\beta$ for different training set $D_i$.

Here is my problem,

  1. What is the estimator here? According to the post I read, I think that the process of choosing the linear regression model and estimating $\hat\beta$ via OLS are components of the estimator, right?

  2. How to check the estimator is biased or not? Since I chose the linear regression model $h(x)$, and now have the estimates $\hat\beta$, I think to check the estimator is biased or not, is to find out $$B(\hat\beta)=\beta_{true}-E[\hat\beta],$$ right? Now this is the thing confuses me badly, since we don't have this $\beta_{true}$, because the true $f$ is actually $sin(\theta x)$ with $\theta=2\pi$, we only have $\theta_{true}$, no such $\beta_{true}$, right?

Hope you can help me to get things right.


To address @whuber's comment, my question originates from this lecture note (from page 11)

1.how do you "randomly draw" values from a function?

Let's say the target function $f(x)=sin(2\pi x)$ is defined on $x\in [-1,1]$, I could randomly (via a uniform distribution over $[-1,1]$) draw $N$ points $\{x_1,...,x_N\}$, where $N$ is the size of the sample, after that I could compute the output value $y_i=f(x_i)$, no noise added. Now the sample is $\{(x_1,y_1),...,(x_N,y_N)\}$.

2.are either (or both) of the $xi$ or $yi$ measured? what is the expected statistical behavior of the measurement errors?

Here I only measure the training/test error of output value $y_i$. I mean, with different training sample, I have different $\hat\beta$, then for a fixed point $x_0$, the predicted output $\hat y_0=h(x_0)=\beta^Tx_0$ will be different, but the true value is $y_0=f(x_0)$.

3.why are you choosing a linear estimator β′x to approximate a function sin(2πx) that is a fortiori strongly nonlinear?

As what is done in the lecture note, what I have is just the sample, which looks nonlinear, maybe it's not wise to choose a linear estimator, and maybe I should choose this one $$h(x)=\beta_0+\beta_1x+\beta_2x^2$$, a polynomial estimator, but it's still far from the target $sin(2\pi x)$.

  • 1
    $\begingroup$ You are having trouble because your description lacks critical information. First, how do you "randomly draw" values from a function? Related to this is concern about how those values might depart from the correct functional values: are either (or both) of the $x_i$ or $y_i$ measured? If so, what is the expected statistical behavior of the measurement errors? Third, how do you measure "closeness" of an approximation to a function? Fourth, why are you choosing a linear estimator $\beta^\prime x$ to approximate a function $\sin(2\pi x)$ that is a fortiori strongly nonlinear? $\endgroup$
    – whuber
    Dec 30 '13 at 16:38
  • $\begingroup$ @whuber, I just update my post to address the questions you pose, ;-) $\endgroup$
    – avocado
    Dec 31 '13 at 1:08
  • $\begingroup$ Thanks (+1). There is a fundamental problem with those lecture notes: they appear to be from the middle of a long sequence and so they do not define or explain any of their terms. Although thanks to your edits (1) and (2) we now have an effective probability model, what is missing is any quantitative way to compare an estimator $h$ to the target $f$. That's essential for proceeding further. Unfortunately the lecture slides have no reliable indication of how that is done. In (3) you allude to a concept of distance between $h$ and $f$ ("far from"): do you have something quantitative in mind? $\endgroup$
    – whuber
    Dec 31 '13 at 14:35
  • $\begingroup$ @whuber, actually according to the lecture notes, it compare th $h$ to the target $f$ by computing the bias and variance of the predicted output $\hat y$, it even drew some graph in the note, and comparing the two selected Models $h1,h2$. $\endgroup$
    – avocado
    Jan 2 '14 at 1:18

I'm not 100% sure about this either, as the post you link to mentions a lot of subtleties that I haven't considered or studied much yet...but here's an attempt to answer nevertheless.

  1. As you've stated it, $h(x)$ is your estimator, because you're using it to approximate $f$. "The process of choosing the linear regression model" is the process of choosing the estimator function, and using it to estimate $\hat\beta$ is just that: using the estimator to produce an estimate. That is, no, these are not components of the estimator itself; these are the process from which the estimator originates, and the process it serves, respectively.

  2. The limited way of testing estimator bias with which I am amateurishly familiar is simulation testing. In such a special circumstance as this, we may actually know $\hat\beta_{true}$ (probably should in many such cases). Furthermore, by simulating data according to predefined parameters, we can test how different values affect the errors of our estimators. As I understand it, these systematic errors are the primary concerns in considerations of estimator bias.

    For example, one usually wants an estimator for which the accuracy depends minimally on sample size, or at least won't lose accuracy as sample sizes increase. I think this is sometimes a problem in significance testing, in that some tests will reject the null too often when the null is actually true and the sample size is very large (e.g., the Shapiro-Wilk test). Another example of estimator bias (I think...another place where I might be mistaken) might be your typical parametric test when used in conditions that violate its assumptions. Non-normal distributions can bias parametric tests that assume normally distributed data, whereas nonparametric tests are often relatively unbiased estimators.

    Sometimes biasing is more complex and even interactive. For example, I recently read that substituting polychoric correlations for Pearson's $r$ correlations in a matrix on which confirmatory factor analysis is to be performed can inflate (bias) standard errors of parameter estimates and $\chi^2$ goodness-of-fit when using maximum likelihood estimation (Babakus, 1985). The choice of estimator really starts to get hairy in latent factor modeling...

    In any case, problems like these are often discovered by simulation testing, wherein the true parameters are designated and altered systematically, random data are generated based on these settings, and estimates are found to deviate from the true values to different degrees depending on the parameters of the simulated distributions. The extent of that dependence on the distributional parameters is the estimator's sensitivity to those parameters; if the sensitivity is non-negligible, the estimator is biased when the parameters to which it is sensitive enter certain ranges. These are often not the parameters it is used to estimate! OLS multiple regression is sensitive to multicollinearity in regressors, for another example, whereas ridge regression can correct for bias somewhat when regressors are strongly related (collinear).

  • 1
    $\begingroup$ Thanks, but I still don't think you answer the 2nd question. $\endgroup$
    – avocado
    Dec 30 '13 at 13:15
  • $\begingroup$ Maybe I was a little circuitous, because I also addressed your misconception about bias (it is not usually a function of the estimand, as in your B*(*β) example)...but the answer I provided (one of many possible answers) is simulation testing. When you can define all the parameters for yourself, you can test the estimator's sensitivity to them across multiple simulated samples. If you find that errors change in relation to a certain parameter as you change it across simulated samples, you may have found bias. All of this info is in my answer or common practice in sim testing. $\endgroup$ Dec 30 '13 at 19:40
  • $\begingroup$ Hope I didn't sound overly indignant in the above response to your feedback BTW. My answer got rather curt as I ran into the character limit, and I don't see any element of your question I didn't answer initially, but if my comment added useful clarifications, I'd be happy to edit them in. If not, could you clarify what's missing? $\endgroup$ Dec 31 '13 at 6:19
  • $\begingroup$ Not at all, it's very kind of you to write this answer, and I read your answer over and over again. I'm just having a hard time figuring out that, the target is $f(x)=sin(\theta x)$, but I choose to approximate it using $h(x)=\beta_1x+\beta_0$, which definitely won't approximate $f$ well. But I still could compute $E[\hat\beta_1], E[\hat\beta_0]$ based on all the different samples I have. According to your answer, you mean I shouldn't measure the bias of $\hat\beta$? Why? $\endgroup$
    – avocado
    Dec 31 '13 at 6:52
  • 1
    $\begingroup$ @Nick Lehman is a standard intro graduate-level textbook for courses on estimation (and the first edition has long been a classic). A more accessible introduction that covers the same conceptual material but with less reliance on the mathematical details is Kiefer's Introduction to Statistical Inference. The complication in your question is that it is unclear how one ought to measure the difference between an estimator of a function and the function itself (there are many ways). In particular, the lecture slides are opaque concerning what the bias might mean in this context. $\endgroup$
    – whuber
    Dec 31 '13 at 15:12

On looking at those lecture notes, the sample size is only two - so you can hardly do better without knowing anything about f than simply averaging the two values.

The notions of "bias" and "variance" can be defined only relative to some kind of model structure. This is clearly encapsulated by the $ E_x[.] $ and $ E_D[.] $ operators in the lecture notes. Note though, that they are a tad clumsy in that $ E_x [E_D [.]] $ should really be written as $ E_x [E_{D|x}]] $ as D and x are related by the model. These basically describe "how did you choose the C values" ($ E_x $) and "given the choice of X values how did you choose the Y values" ($ E_{D|x} $). Generally speaking, the latter encspsulates the model assumptions with the former generally assumed known.

Now, it seems like you are talking about a problem with "no noise" - now if you know that $ y=f (x) $ exactly - then any estimator that doesn't interpolate the observed data is necessarily wrong. The only source of "randomness" is which particular "X" values and "Y" values are observed. This has a similar flavour to design based inference for sample surveys.

The notion of bias in this context depends on the "sample space" for the X values, the "sampling distribution" for the X values, and the function $ f(x) $. I use quotes as it is entirely reasonable to consider degenerate cases where the X values are not "random" but fixed at prespecified values (as is the case for prediction of a new Y value not used to fit the model).

Now basically you can't get any further with the bias of an estimated function $ h $ unless you impose some conditions on what the function $ f$ might look like.

In fact, without the prescence of any noise this is "merely" a transformation of a random variable. If you propose/assume a "sampling distribution " for the $ X $ values, call this cdf $ G_X(x)=Pr (X\leq x) $, then the corresponding distribution for the response is $ G_Y (y)=Pr (Y\leq y)=Pr (f (X)\leq y)=\int I\{f (x)\leq y\} dG_X (x)$. For 1-to-1 continuous, differentiable functions you can simplify this further to state that the pdf for "y" must have the form $$ g_Y (y)=g_X (f^{-1}(y))|\frac {\partial f^{-1}(y)}{\partial y}|$$ where $ f^{-1}(.)$ is the inverse transformation of $ f (.) $. So for a linear function $ f (x)=a +bx $ (inverse function of $ f^{-1}(y)=b^{-1} (y-a) $ ) combined with a uniform $[-1, 1] $ pdf for $ X $ gives

$$ g_Y (y)=\frac {I[-1 \leq b^{-1}(y-a) \leq 1]}{2|b|}$$

That is, Y is uniform $[a-b, a+b] $ if $ b> 0 $ and uniform $[a+b, a-b] $ otherwise. Now we can show that the mle for $ a, b $ is given by the same OLS "saturated" fit , namely $\hat {b}=\frac {y_1-y_2}{x_1-x_2} $ and $\hat {a}=\frac {y_2x_1-y_1x_2}{x_1-x_2}$. In fact these must be the exact values for $ a $ and $ b $ provided the linear function is correct - regardless of the sampling distribution for X. Another way of saying this is that there can be only one noiseless linear relationship between two or more X-Y pairs. This also leads to an extremely aggressive predictive distribution degenerate at $ \hat {y }=\hat {a}+\hat {b} x $ with zero margin for error (after observing x). The aggressive prediction comes from the "no noise" assumption.

As a final remark, the observed data provides no information on what that relationship is for "X-Y" pairs that are not fully observed. This can only come from other pieces of information - such as assumptions about smoothness, and continuity of $ f (.) $. This makes calculating bias impossible in a general sense, because your answer will depend on some arbitrary unknown function. You have to assume something about what it could be in order to calculate the bias of an estimator for $ f(.) $ (eg $ f (.) $ has a third order derivative, no singularities, is analytic, etc). But these choices cannot be disentangled from the choices resulting from standard model checking (eg add a quadratic term if a plot of the residuals shows curvature). This clouds the practical use of "bias" in a rigorous and completely general fashion, as the observed data set is analysed to decide model structure. Different data sets get analysed in different ways, adding a "human element" to a bias calculation (and variance too) that is difficult to both automate (making Monte Carlo infeasible) and write down a formula for what happens.

Having said that, the notion of bias is still useful as part of a check of model assumptions - but is generally better thought of in terms of complexity and stability of the model IMO. Bias is also useful as a conceptual tool to aid understanding of general model fitting issuues and the tension between explaing the observed data and predicting unobserved data.

  • $\begingroup$ +1, but I have to say your answer is math heavy for me,;-). Anyway, as I read into your answer, 1)bias is used with some random processes, 2)and it's just a guideline for us to analyze the model, since we don't or can't know what the bias is exactly, because we don't know what the true parameter is. Am I right? $\endgroup$
    – avocado
    Jan 3 '14 at 14:41
  • $\begingroup$ @loganecolss - basically yes. Bias is always with respect to a random process (what I called a model). You can know what the bias is for some parameters in some types of models (linear regression). But for vague models like $ y=f (x) $ you can only get a vague idea of bias. $\endgroup$ Jan 4 '14 at 4:40
  • $\begingroup$ Well, since $bias(\hat y)=E[\hat y]-y_{true}$, so if I can know what the bias is, that means I can know $y_{true}$, right? But in practice, what I have is just the sample with noise, so how could I know $y_{true}$? $\endgroup$
    – avocado
    Jan 4 '14 at 5:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.