Suppose we observe $x$ and $y$ and we want to predict at $x=5$. A naive way would be to take each observation and compute $5/(x/y)$ or similarly $5*(y/x)$ and then take the overall mean. Thi is basically rescaling each observation to the unit scale and then extrapolating to 5.

A more sophisticated approach is to perform the linear regression and then predict at $x=5$.

Is there reason to believe the linear regression approach is more accurate compared to the first method? I believe the first approach is very sample dependent.

Here is an example:



### Linear Regression Prediction
#     fit      lwr      upr
# 1 4.844763 4.729187 4.960339

### Naive Prediction
# 8.49

Yes, there is. We are modeling using $y = \theta x$. consider a simple example: $(1,2), (3,3)$

Using naive, we are taking the average of $\frac{y}{x}$ for all data points as $\theta$ (see below for mathematical explantion). We have: $$\theta_{naive} = (1/2)(2+1) = 1.5$$

Using LR, we have $$ \theta_{LR} = 1.1$$

The training SSE with naive is $(2-1.5)^2 + (3-4.5)^2 = 2.5$ while with LR it is $(2-1.1)^2 + (3- 3.3)^2 = 0.9$.

Why is this? We can compare how the two $\theta$ differ mathematically

The naive predictor:

$$\hat{y} = \frac{1}{n} \sum^n_{i=1} \frac{y_i}{x_i}x = (\sum^n_{i=1} \frac{1}{n} \frac{y_i}{x_i})x$$

$$ \theta_{naive} = \sum^n_{i=1} \frac{1}{n} \frac{y_i}{x_i} $$

Linear regression:

$$ \theta_{LR} = (X^T X)^{-1} X^T Y= \frac{\sum^n_{i=1} x_i y_i}{ \sum^n_{i=1} x_i^2 } = \frac{\sum^n_{i=1} x_i^2 \frac{y_i}{x_i}}{ \sum^n_{i=1} x_i^2 } = \sum^n_{i=1} \frac{x_i^2}{ \sum^n_{j=1} x_j^2} \frac{y_i}{x_i} $$

So, they differ in how much they weigh each $\frac{y}{x}$. Naive weighs all $\frac{y}{x}$ equally (using $\frac{1}{n}$), whereas linear regression weighs it relative to $x$ (using $\frac{x_i^2}{ \sum^n_{j=1} x_j^2}$). It makes sense that we should weigh the slopes of points with larger $x$ values more: consider if we had (1,1) and (1,0). $\theta = 0.5$ minimizes error. If we had (1,1) and (10000,0), $\theta$ should decrease, not stay the same, despite the slopes staying the same.

  • $\begingroup$ I'm confused where you say average of the slopes as $\theta$. There are only two points and thus one slope of 0.5. Could you please elaborate here more? $\endgroup$ – Glen Apr 21 '16 at 18:34
  • $\begingroup$ Sorry, my fault. That doesn't make sense. I meant the averages of $\frac{y}{x}$ $\endgroup$ – cheniel Apr 21 '16 at 18:36
  • $\begingroup$ This analysis goes in the right direction. Ordinarily one would take it further to compute the standard error of prediction for both procedures and demonstrate that one of them has a uniformly smaller SEP (if that is the case). $\endgroup$ – whuber Apr 21 '16 at 19:22
  • $\begingroup$ @whuber thanks -- not too familiar with this, do you mean that ordinarily one would demonstrate $\sum^n_{i=1} (y_i - x_i \theta_{LR})^2 \le \sum^n_{i=1} (y_i - x_i \theta_{naive})^2$? $\endgroup$ – cheniel Apr 21 '16 at 19:34
  • $\begingroup$ You are analyzing two procedures that predict a value at $x=5$. Suppose the correct (but unknown value) is $\phi$. Let's call these predictions $\hat{f}(5)$ and $\hat{g}(5)$. Since each of the $\hat f$ and $\hat g$ is a definite (and explicitly computable) function of the data, which are modeled as random variables, each is itself a random variable. Compute and compare $\mathbb{E}((\hat{f}(5) - \phi)^2)$ and $\mathbb{E}((\hat{g}(5) - \phi)^2)$. You might want to start by computing the expectations $\mathbb{E}(\hat{f}(5))$ and $\mathbb{E}(\hat{g}(5))$. $\endgroup$ – whuber Apr 21 '16 at 19:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.