0
$\begingroup$

Let $m$ be a list of 10 observations and $t$ a list of $10$ covariates. Assume that there's a somewhat linear relationship going on. What is the following process called, is it in any ways useful, and why does it end the way it does:

Process: We estimate our regression line for $m$ using linear regression. This give us 10 fitted values. We simulate 10 new observations from 10 normal distributions with the fitted values as the 10 different means (same variance). These then new observations are called $m$. The process is now repeated.

Ending I did this repeatedly, and after many observations, I ended up getting "perfect" fits in R. Why? Is this an obvious result? Is the process useful in any way?

Improve this question
$\endgroup$
1
  • $\begingroup$ What is the "repeatedly" here? Are the many iterations all from the original fit, or is each from the previous fit? Do I understand correctly that you have N = 10 data & p = 10 variables? If so, the simulation issue is irrelevant: the model is saturated. $\endgroup$
    – gung - Reinstate Monica
    Commented Mar 7, 2016 at 22:35

1 Answer 1

Reset to default
4
$\begingroup$

I did this repeatedly, and after many observations, I ended up getting "perfect" fits in R. Why?

Because you have as many covariates as observations. In fact if you're not counting the constant column as one of the covariates it would even happen if you had one fewer covariates.

Is this an obvious result?

Yes.

Mark two points on a piece of paper. Draw the line of closest possible fit to those two points. Is it a perfect fit?

Now consider three points in space. Would the plane of best fit go through all three points? (Stick three fingers up in the air and try to lay a flat piece of cardboard on them -- can it touch all three fingers at the same time?)

If you have as many covariates (this time counting the constant predictor as one) as you do points, then the fit goes exactly through the data since each covariate has an additional degree of freedom with which to make an exact fit to an additional point*, there are no degrees of freedom left for error.

* this only fails to happen if your set of covariates (predictors) are linearly dependent, but that won't have been the case for your simulation.

Is the process useful in any way?

Well, hopefully it leads you to learning something pretty basic about regression (or indeed, about linear algebra), and knowledge you found yourself is certainly likely to be useful -- it's more likely to stick in the mind.

But fitting all the noise in the data is not of itself typically valuable.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.