A professor in one of my graduate statistics courses once said, when briefly reviewing simple linear regression: "I would never EVER fit a line to fewer than 8-10 data points, it would make me feel...rather uncomfortable." Many of us would agree, as evidenced by "rules of thumb" suggested previously on CV here, here, and here for regression (i.e., "10 samples per covariate").

A regression line like this one ($n$ = 10) might represent this "minimum comfort level":

Simple linear regression, n=10

Naturally, our level of confidence in the relationship between Predictor 1 and Response would rise if we had 10 repeated samples at each x-value (for $n$ = 100):

Simple linear regression, n=100

Now, suppose we have a second predictor and want to fit a plane $y$ = $\beta_0$ + $\beta_1$$x_1$ + $\beta_2$$x_2$ + $e$ or even a non-linear interaction $y$ = $\beta_0$ + $\beta_1$$x_1$ + $\beta_2$$x_2$ + $\beta_3$$x_1$*$x_2$ + $e$. Like the example above, suppose we have $n$ = 100, and these 100 samples are repeated 10 times at 10 "unique" predictor values (in this case, 10 unique $x_1$-$x_2$ combinations). Therefore, we have a relatively large sample size ($n$ = 100) for a relatively small number of predictors $n$ = 2, which, given our rule of thumb, would be generally acceptable.

(Left, linear plane; Right, non-linear surface):

Linear planeNon-linear surface

My questions are:

  1. Would repeated samples ("stacking" at the unique $x_1$-$x_2$ combos) improve your confidence in the fit of a plane (as they did in the fitting of a line), given that these samples do not appear to give us more information about un-sampled locations on the plane? Why or why not?
  2. Given your response to question 1, would you "feel comfortable" fitting an interaction term $x_1$*$x_2$, as I have shown in the graph above (right)?

I look forward to discussion.

  • $\begingroup$ If you feel it makes sense with a single predictor, why wouldn't it with two? $\endgroup$ – dsaxton Jul 20 '15 at 4:09
  • $\begingroup$ @dsaxon my concern is that there are few unique values for predictors. Consider a more extreme example where we still have n=100 but only 5 "unique" predictor values instead of 10 as shown above. In my mind, it would be treacherous to predict a surface, especially a non linear one, when I have very little data forming the remainder of the surface. $\endgroup$ – Gavin M. Jones Jul 20 '15 at 4:14
  • 1
    $\begingroup$ What you remember your professor saying reflects an overly narrow understanding of regression and its uses. Three points would be just fine in many circumstances where the assumptions have been independently verified (linearity; normality, independence, and homoscedasticity of errors; accurate measurement of the independent variable). This, and your allusion to "unsampled locations," suggests you really want to explore issues of model selection and prediction rather than regression sample sizes or rules of thumb. $\endgroup$ – whuber Jul 20 '15 at 12:28
  • 1
    $\begingroup$ @GavinM.Jones About taking the mean before fitting, this isn't equivalent because the least squares fit would change in general (just imagine the samples at the different points being heavily unbalanced, for instance). But probably more importantly, there's more information in an average than a single observation, so it isn't really correct to think of this as only 10 data points. $\endgroup$ – dsaxton Jul 20 '15 at 13:26
  • 2
    $\begingroup$ @Engr Questions about the "median" or any other property of a collection of datasets are rarely relevant to the problem one currently faces. This is not a mathematical question: it's a statistical one, and it has to be answered within a context that includes whatever theory is believed to apply, any prior information, the costs of collecting data, and the objectives of the exercise. Blindly following rules of thumb, with absolutely no thought or discretion--"I would never EVER..."--will surely be a costly mistake for some people in some circumstances. $\endgroup$ – whuber Jul 20 '15 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.