# Linear Regression Prediction vs Extrapolation Prediction

Suppose we observe $x$ and $y$ and we want to predict at $x=5$. A naive way would be to take each observation and compute $5/(x/y)$ or similarly $5*(y/x)$ and then take the overall mean. Thi is basically rescaling each observation to the unit scale and then extrapolating to 5.

A more sophisticated approach is to perform the linear regression and then predict at $x=5$.

Is there reason to believe the linear regression approach is more accurate compared to the first method? I believe the first approach is very sample dependent.

Here is an example:

library(ggplot2)

set.seed(123)
nobs=1000
x=runif(nobs,0.1,100)
y=abs(x*.05+rnorm(nobs,0,1))
a2=data.frame(x,y)
ggplot(a2,aes(x=x,y=y))+geom_point()+geom_smooth()+geom_smooth(method='lm')

### Linear Regression Prediction
fit=lm(y~x)
print(predict(fit,newdata=data.frame(x=100),interval='confidence'))
#     fit      lwr      upr
# 1 4.844763 4.729187 4.960339

### Naive Prediction
print(mean(100/(a2$x/a2$y)))
# 8.49


Yes, there is. We are modeling using $y = \theta x$. consider a simple example: $(1,2), (3,3)$

Using naive, we are taking the average of $\frac{y}{x}$ for all data points as $\theta$ (see below for mathematical explantion). We have: $$\theta_{naive} = (1/2)(2+1) = 1.5$$

Using LR, we have $$\theta_{LR} = 1.1$$

The training SSE with naive is $(2-1.5)^2 + (3-4.5)^2 = 2.5$ while with LR it is $(2-1.1)^2 + (3- 3.3)^2 = 0.9$.

Why is this? We can compare how the two $\theta$ differ mathematically

The naive predictor:

$$\hat{y} = \frac{1}{n} \sum^n_{i=1} \frac{y_i}{x_i}x = (\sum^n_{i=1} \frac{1}{n} \frac{y_i}{x_i})x$$

$$\theta_{naive} = \sum^n_{i=1} \frac{1}{n} \frac{y_i}{x_i}$$

Linear regression:

$$\theta_{LR} = (X^T X)^{-1} X^T Y= \frac{\sum^n_{i=1} x_i y_i}{ \sum^n_{i=1} x_i^2 } = \frac{\sum^n_{i=1} x_i^2 \frac{y_i}{x_i}}{ \sum^n_{i=1} x_i^2 } = \sum^n_{i=1} \frac{x_i^2}{ \sum^n_{j=1} x_j^2} \frac{y_i}{x_i}$$

So, they differ in how much they weigh each $\frac{y}{x}$. Naive weighs all $\frac{y}{x}$ equally (using $\frac{1}{n}$), whereas linear regression weighs it relative to $x$ (using $\frac{x_i^2}{ \sum^n_{j=1} x_j^2}$). It makes sense that we should weigh the slopes of points with larger $x$ values more: consider if we had (1,1) and (1,0). $\theta = 0.5$ minimizes error. If we had (1,1) and (10000,0), $\theta$ should decrease, not stay the same, despite the slopes staying the same.

• I'm confused where you say average of the slopes as $\theta$. There are only two points and thus one slope of 0.5. Could you please elaborate here more? – Glen Apr 21 '16 at 18:34
• Sorry, my fault. That doesn't make sense. I meant the averages of $\frac{y}{x}$ – cheniel Apr 21 '16 at 18:36
• This analysis goes in the right direction. Ordinarily one would take it further to compute the standard error of prediction for both procedures and demonstrate that one of them has a uniformly smaller SEP (if that is the case). – whuber Apr 21 '16 at 19:22
• @whuber thanks -- not too familiar with this, do you mean that ordinarily one would demonstrate $\sum^n_{i=1} (y_i - x_i \theta_{LR})^2 \le \sum^n_{i=1} (y_i - x_i \theta_{naive})^2$? – cheniel Apr 21 '16 at 19:34
• You are analyzing two procedures that predict a value at $x=5$. Suppose the correct (but unknown value) is $\phi$. Let's call these predictions $\hat{f}(5)$ and $\hat{g}(5)$. Since each of the $\hat f$ and $\hat g$ is a definite (and explicitly computable) function of the data, which are modeled as random variables, each is itself a random variable. Compute and compare $\mathbb{E}((\hat{f}(5) - \phi)^2)$ and $\mathbb{E}((\hat{g}(5) - \phi)^2)$. You might want to start by computing the expectations $\mathbb{E}(\hat{f}(5))$ and $\mathbb{E}(\hat{g}(5))$. – whuber Apr 21 '16 at 19:40