# Can we get the conditional bias of the estimator at a generic $x$?

Consider a standard ERM problem based on quadratic loss where we solve $$\hat{f}_n\in \operatorname*{arg min}_{f\in \mathcal{F}} R_\text{tr}(f)$$ where $$R_\text{tr}(f)=\frac{1}{n}\sum_{i=1}^n (Y_i-f(X_i))^2$$, $$\{(X_i, Y_i): i=1,\dots, n\}$$ is the training dataset, and $$\mathcal{F}:=\{f_\theta: f_\theta(x)=\sum_{i=0}^9 \theta_i x^i , \theta_i \in \mathbb{R}\}$$ the class of polynomials of degree at most $$9$$. Assume that $$f:\mathbb{R}\to \mathbb{R}$$.

Assume that $$f^*(x)=\mathbb{E}[Y\mid X=x]=2x+1$$. Let $$X\sim \operatorname{Unif}(-1, 1)$$ and $$\operatorname{Var}[Y\mid X]=5$$.

Question: can we get the conditional bias of the estimator at a generic $$x$$, $$\mathbb{E}[\hat{f}_n(x)\mid x]-f^*(x) \text{?}$$

Here we take conditional bias conditioned on the training data $$\{X_i\}$$.

#### Your estimator is an OLS estimator from a linear regression model

Your estimation equation here is an OLS estimator for a "polynomial regression" (with respect to the explanatory variable), which is a special kind of linear regression (with respect to the model parameters. You then go on to look at the properties of this estimator relative to a true regression function with some known properties. Let's generalise your problem, and suppose you have a regression function which is a polynomial of degree $$m$$, given by:

$$f_\boldsymbol{\theta}(x) = \sum_{k=0}^m \theta_k x^k.$$

Polynomial functions of the explanatory variable are linear in the parameters $$\theta_1,...,\theta_m$$, so they can be handled using standard linear regression. Given $$n$$ data points, you form the design matrix and coefficient vector as follows:

$$\tilde{\mathbf{x}} = \begin{bmatrix} 1 & x_1 & x_1^2 & \cdots & x_1^m \\ 1 & x_2 & x_1^2 & \cdots & x_2^m \\ 1 & x_3 & x_3^2 & \cdots & x_3^m \\ \vdots & \vdots & \vdots & \ddots & \vdots \\ 1 & x_n & x_n^2 & \cdots & x_n^m \\ \end{bmatrix} \quad \quad \quad \quad \quad \boldsymbol{\theta} = \begin{bmatrix} \theta_0 \\ \theta_1 \\ \theta_2 \\ \vdots \\ \theta_m \end{bmatrix}.$$

You can then write the model for your estimator as a linear regression:

$$\mathbf{Y} = \tilde{\mathbf{x}} \boldsymbol{\theta} + \boldsymbol{\varepsilon}.$$

Your estimator is stipulated to minimise the squared error over the space of polynomial regression functions, so it is the standard OLS estimator for the above linear model. The estimator has the explicit form:

$$\hat{\boldsymbol{\theta}} = (\tilde{\mathbf{x}}^\text{T} \tilde{\mathbf{x}})^{-1} \tilde{\mathbf{x}}^\text{T} \mathbf{y} \quad \quad \quad \quad \quad \hat{f}_n(x) = \sum_{k=0}^m \hat{\theta}_k x^k.$$

Given the true regression function $$f_*$$ and setting $$\mathbf{A} \equiv (\tilde{\mathbf{x}}^\text{T} \tilde{\mathbf{x}})^{-1} \tilde{\mathbf{x}}^\text{T}$$ you have:

$$\hat{\boldsymbol{\theta}} = (\tilde{\mathbf{x}}^\text{T} \tilde{\mathbf{x}})^{-1} \tilde{\mathbf{x}}^\text{T} \mathbf{Y} = \mathbf{A} \mathbf{Y},$$

which gives $$\hat{\theta_k} = \sum_{i=1}^n A_{i,k} \ Y_i$$ for the individual elements of the estimator vector. Taking the expected value then gives the individual elements:

\begin{align} \mathbb{E}(\hat{\theta_k} | \mathbf{x}) &= \sum_{i=1}^n A_{i,k} \ \mathbb{E}(Y_i | \mathbf{x}) \\[6pt] &= \sum_{i=1}^n A_{i,k} \ f_*(x_i), \\[6pt] \end{align}

so you have:

\begin{align} \mathbb{E}(\hat{f}_n(x) | \mathbf{x}) - f_*(x) &= \mathbb{E} \Bigg( \sum_{k=0}^m \hat{\theta}_k x^k \Bigg| \mathbf{x} \Bigg) - f_*(x) \\[6pt] &= \sum_{k=0}^m x^k \mathbb{E} ( \hat{\theta}_k | \mathbf{x} ) - f_*(x) \\[6pt] &= \sum_{k=0}^m x^k \sum_{i=1}^n A_{i,k} \ f_*(x_i) - f_*(x). \\[6pt] \end{align}

The specific forms you have for the stipulated functions should then allow you to reduce this further to get the expected value of the estimated polynomial at an explanatory point.