How to do Ordinary Least Squares (OLS) when the observations are not linear?

Question

This question arose from this question.

Does anyone have some worked examples of an OLS question where the observations are not linear? e.g. $y_i = \alpha + \sin (x_i) + \epsilon_i$

I tried to find to find the least estimators for the coefficient of $x$ by differentiating $(y_i - \alpha - \sin(x))^2$ with respect to $x$ (I'm not even sure if that is the correct thing to do) and end up with some complicated expression involving $\sin$ and $\cos$. So, I thought I would ask how do so such questions before proceeding further.

I am interested in an answer for this because, I want to see that $E(y_i - \hat{y}_i)$ is not always equal to 0 when $y_i$ is not a linear function of $x_i$. Also, is there a more general way to show that this statement is true i.e. without actually finding the OLS estimators?

EDIT 1:

Forgot to include an error term in the example.

EDIT 2:

The model I am trying to fit is $y = \sin(x) + \epsilon_i$.

Given a certain data set, according to R I should get $\hat{y} = 0.60330 + 0.01797x$:

set.seed(1234)

 n <- 5
 df <- data.frame(x=runif(n, 1, 10))
 df$mean.y.given.x <- sin(df$x)
 df$y <- df$mean.y.given.x + rnorm(n)
 model <- lm(y ~ x, data=df)
 summary(model)

 Call:
 lm(formula = y ~ x, data = df)

 Residuals:
     1       2       3       4       5 
 0.6190 -1.1402 -0.4852 -0.2877  1.2941 

 Coefficients:
              Estimate Std. Error t value Pr(>|t|)
 (Intercept)  0.60330    1.45534   0.415    0.706
 x            0.01797    0.22460   0.080    0.941

 Residual standard error: 1.107 on 3 degrees of freedom
 Multiple R-squared:  0.00213,  Adjusted R-squared:  -0.3305 
 F-statistic: 0.006404 on 1 and 3 DF,  p-value: 0.9413

I would like to obtain $\hat{y}$ by hand now. However, all the questions I have done thus far is always of the form $y_i = \alpha + \beta x$, where $\alpha$ and $\beta$ are constants, so that $y_i$ is a linear function of $x_i$. Therefore, I am unsure of how to proceed when $y_i$ is not a linear function of $x$, hence my request for a worked examples for these kind of questions.

After that, I expect to be able to show that $E(y_i-\hat{y_i}) \neq 0$ because the plot of the residuals against the predicted value (for large $n$) is:

set.seed(1234)
n <- 1000
df <- data.frame(x=runif(n, 1, 10))
df$mean.y.given.x <- sin(df$x)
df$y <- df$mean.y.given.x + rnorm(n)
model <- lm(y ~ x, data=df)
plot(predict(model, newdata=df), residuals(model))
abline(a=0,b=0,col='blue')

Based on the above plot, $E(y_i-\hat{y_i}) \neq 0$ must be true in this case, right?

Answer 1

You don't really believe that your data ($y_i$) are exactly equal $\alpha+\sin(x_i)$ do you? Your equation is missing an important term (the error), and how you write it matters as to whether you can do least squares or not.

If your model is $y_i = \alpha + \sin(x_i)+\epsilon_i$, then it would certainly make sense to estimate $\alpha$ by ordinary least squares. The important thing for least squares is that the model is linear in the parameters, which it is.

Let $y^*_i= y_i-\sin(x_i)$, and your model becomes $y^*_i= \alpha+\epsilon_i$, and the LS estimate,

$$\hat{\alpha}= \bar{y^*} = \frac{1}{n}\sum_{i=1}^n (y_i-\sin(x_i))$$

and so

$$\hat{y_i}=\hat{\alpha}+\sin(x_i)$$

That $E(y_i-\hat{y}_i) =0$ is pretty straightforward from there.