Trying to understand unbiased estimator

Question

After reading this post, I still don't thoroughly understand what an estimator is. Suppose the samples $D_i={(x_1,y_1),...(x_n,y_n)}$ are drawn randomly from function $$f(x)=sin(2\pi x),$$ so my ultimate goal is to come up with an estimated function $h(x)$, which should approximate $f$ as close as possible, right? Since I don't know what the true $f$ looks like, so I might choose different models to do the approximation, here I consider the linear regression model, which is $$h(x)=\beta^Tx,$$ then I try to estimate the model parameters $\beta$ via OLS, and finally given training sample $D_i$, I have the estimates $\hat\beta_{|D_i}$, and absolutely dffierent $\hat\beta$ for different training set $D_i$.

Here is my problem,

1. What is the estimator here? According to the post I read, I think that the process of choosing the linear regression model and estimating $\hat\beta$ via OLS are components of the estimator, right?

2. How to check the estimator is biased or not? Since I chose the linear regression model $h(x)$, and now have the estimates $\hat\beta$, I think to check the estimator is biased or not, is to find out $$B(\hat\beta)=\beta_{true}-E[\hat\beta],$$ right? Now this is the thing confuses me badly, since we don't have this $\beta_{true}$, because the true $f$ is actually $sin(\theta x)$ with $\theta=2\pi$, we only have $\theta_{true}$, no such $\beta_{true}$, right?

Hope you can help me to get things right.

UPDATE

To address @whuber's comment, my question originates from this lecture note (from page 11)

1.how do you "randomly draw" values from a function?

Let's say the target function $f(x)=sin(2\pi x)$ is defined on $x\in [-1,1]$, I could randomly (via a uniform distribution over $[-1,1]$) draw $N$ points $\{x_1,...,x_N\}$, where $N$ is the size of the sample, after that I could compute the output value $y_i=f(x_i)$, no noise added. Now the sample is $\{(x_1,y_1),...,(x_N,y_N)\}$.

2.are either (or both) of the $xi$ or $yi$ measured? what is the expected statistical behavior of the measurement errors?

Here I only measure the training/test error of output value $y_i$. I mean, with different training sample, I have different $\hat\beta$, then for a fixed point $x_0$, the predicted output $\hat y_0=h(x_0)=\beta^Tx_0$ will be different, but the true value is $y_0=f(x_0)$.

3.why are you choosing a linear estimator β′x to approximate a function sin(2πx) that is a fortiori strongly nonlinear?

As what is done in the lecture note, what I have is just the sample, which looks nonlinear, maybe it's not wise to choose a linear estimator, and maybe I should choose this one $$h(x)=\beta_0+\beta_1x+\beta_2x^2$$, a polynomial estimator, but it's still far from the target $sin(2\pi x)$.

Answer 1

I'm not 100% sure about this either, as the post you link to mentions a lot of subtleties that I haven't considered or studied much yet...but here's an attempt to answer nevertheless.

1. As you've stated it, $h(x)$ is your estimator, because you're using it to approximate $f$. "The process of choosing the linear regression model" is the process of choosing the estimator function, and using it to estimate $\hat\beta$ is just that: using the estimator to produce an estimate. That is, no, these are not components of the estimator itself; these are the process from which the estimator originates, and the process it serves, respectively.

2. The limited way of testing estimator bias with which I am amateurishly familiar is simulation testing. In such a special circumstance as this, we may actually know $\hat\beta_{true}$ (probably should in many such cases). Furthermore, by simulating data according to predefined parameters, we can test how different values affect the errors of our estimators. As I understand it, these systematic errors are the primary concerns in considerations of estimator bias.

   For example, one usually wants an estimator for which the accuracy depends minimally on sample size, or at least won't lose accuracy as sample sizes increase. I think this is sometimes a problem in significance testing, in that some tests will reject the null too often when the null is actually true and the sample size is very large (e.g., the Shapiro-Wilk test). Another example of estimator bias (I think...another place where I might be mistaken) might be your typical parametric test when used in conditions that violate its assumptions. Non-normal distributions can bias parametric tests that assume normally distributed data, whereas nonparametric tests are often relatively unbiased estimators.

   Sometimes biasing is more complex and even interactive. For example, I recently read that substituting polychoric correlations for Pearson's $r$ correlations in a matrix on which confirmatory factor analysis is to be performed can inflate (bias) standard errors of parameter estimates and $\chi^2$ goodness-of-fit when using maximum likelihood estimation (Babakus, 1985). The choice of estimator really starts to get hairy in latent factor modeling...

   In any case, problems like these are often discovered by simulation testing, wherein the true parameters are designated and altered systematically, random data are generated based on these settings, and estimates are found to deviate from the true values to different degrees depending on the parameters of the simulated distributions. The extent of that dependence on the distributional parameters is the estimator's sensitivity to those parameters; if the sensitivity is non-negligible, the estimator is biased when the parameters to which it is sensitive enter certain ranges. These are often not the parameters it is used to estimate! OLS multiple regression is sensitive to multicollinearity in regressors, for another example, whereas ridge regression can correct for bias somewhat when regressors are strongly related (collinear).

Answer 2

On looking at those lecture notes, the sample size is only two - so you can hardly do better without knowing anything about f than simply averaging the two values.

The notions of "bias" and "variance" can be defined only relative to some kind of model structure. This is clearly encapsulated by the $ E_x[.] $ and $ E_D[.] $ operators in the lecture notes. Note though, that they are a tad clumsy in that $ E_x [E_D [.]] $ should really be written as $ E_x [E_{D|x}]] $ as D and x are related by the model. These basically describe "how did you choose the C values" ($ E_x $) and "given the choice of X values how did you choose the Y values" ($ E_{D|x} $). Generally speaking, the latter encspsulates the model assumptions with the former generally assumed known.

Now, it seems like you are talking about a problem with "no noise" - now if you know that $ y=f (x) $ exactly - then any estimator that doesn't interpolate the observed data is necessarily wrong. The only source of "randomness" is which particular "X" values and "Y" values are observed. This has a similar flavour to design based inference for sample surveys.

The notion of bias in this context depends on the "sample space" for the X values, the "sampling distribution" for the X values, and the function $ f(x) $. I use quotes as it is entirely reasonable to consider degenerate cases where the X values are not "random" but fixed at prespecified values (as is the case for prediction of a new Y value not used to fit the model).

Now basically you can't get any further with the bias of an estimated function $ h $ unless you impose some conditions on what the function $ f$ might look like.

In fact, without the prescence of any noise this is "merely" a transformation of a random variable. If you propose/assume a "sampling distribution " for the $ X $ values, call this cdf $ G_X(x)=Pr (X\leq x) $, then the corresponding distribution for the response is $ G_Y (y)=Pr (Y\leq y)=Pr (f (X)\leq y)=\int I\{f (x)\leq y\} dG_X (x)$. For 1-to-1 continuous, differentiable functions you can simplify this further to state that the pdf for "y" must have the form $$ g_Y (y)=g_X (f^{-1}(y))|\frac {\partial f^{-1}(y)}{\partial y}|$$ where $ f^{-1}(.)$ is the inverse transformation of $ f (.) $. So for a linear function $ f (x)=a +bx $ (inverse function of $ f^{-1}(y)=b^{-1} (y-a) $ ) combined with a uniform $[-1, 1] $ pdf for $ X $ gives

$$ g_Y (y)=\frac {I[-1 \leq b^{-1}(y-a) \leq 1]}{2|b|}$$

That is, Y is uniform $[a-b, a+b] $ if $ b> 0 $ and uniform $[a+b, a-b] $ otherwise. Now we can show that the mle for $ a, b $ is given by the same OLS "saturated" fit , namely $\hat {b}=\frac {y_1-y_2}{x_1-x_2} $ and $\hat {a}=\frac {y_2x_1-y_1x_2}{x_1-x_2}$. In fact these must be the exact values for $ a $ and $ b $ provided the linear function is correct - regardless of the sampling distribution for X. Another way of saying this is that there can be only one noiseless linear relationship between two or more X-Y pairs. This also leads to an extremely aggressive predictive distribution degenerate at $ \hat {y }=\hat {a}+\hat {b} x $ with zero margin for error (after observing x). The aggressive prediction comes from the "no noise" assumption.

As a final remark, the observed data provides no information on what that relationship is for "X-Y" pairs that are not fully observed. This can only come from other pieces of information - such as assumptions about smoothness, and continuity of $ f (.) $. This makes calculating bias impossible in a general sense, because your answer will depend on some arbitrary unknown function. You have to assume something about what it could be in order to calculate the bias of an estimator for $ f(.) $ (eg $ f (.) $ has a third order derivative, no singularities, is analytic, etc). But these choices cannot be disentangled from the choices resulting from standard model checking (eg add a quadratic term if a plot of the residuals shows curvature). This clouds the practical use of "bias" in a rigorous and completely general fashion, as the observed data set is analysed to decide model structure. Different data sets get analysed in different ways, adding a "human element" to a bias calculation (and variance too) that is difficult to both automate (making Monte Carlo infeasible) and write down a formula for what happens.

Having said that, the notion of bias is still useful as part of a check of model assumptions - but is generally better thought of in terms of complexity and stability of the model IMO. Bias is also useful as a conceptual tool to aid understanding of general model fitting issuues and the tension between explaing the observed data and predicting unobserved data.