1
$\begingroup$

Let's say I want to create a line of best fit to approximate the relationship between years of golf experience, and average golf score.

If I have only 4 data points, my line of best fit will have a lot of noise. Is there an equation I can use to say how good the line of best fit is based on the number of data points are used to create it?

I guess we could say quality = theNumberOfDataPoints, but it doesn't seem like a linear relationship to me... Is it maybe the square root of the number of data points?

$\endgroup$
3
$\begingroup$

You mention your line of best fit, so you are thinking graphically. You could also show the "quality" graphically.

In that case I would suggest plotting, along with the line of best fit itself, the upper and lower bounds of:

  • the 95 % confidence interval (of the mean DV for different values of IV), and
  • the 95 % prediction interval (of DVs predicted by the model from different values of IV).

There are some examples here: http://www.medcalc.org/manual/scatter_diagram_regression_line.php

...and here is a simple one for data just like yours, with

  • 95 % confidence interval bounds in red
  • 95 % prediction interval bounds in orange

graph with 95 % confidence interval bounds and 95 % prediction interval bounds plotted

R code:
(

a <- c(5,10,15,20)
score <- c(95,82,75,69)
plot(a,score)
model.lm <- lm(score ~ a)
abline(model.lm,col="grey30")
frame = data.frame(a,score)
newx <- seq(0,25)
prdConf <- predict(model.lm, newdata=data.frame(a=newx), interval = c("confidence"), level = 0.95, type="response")
prdPred <- predict(model.lm, newdata=data.frame(a=newx), interval = c("prediction"), level = 0.95, type="response")
lines(newx,prdConf[,2],col="red",lty=2)
lines(newx,prdConf[,3],col="red",lty=2)
lines(newx,prdPred[,2],col="orange",lty=2)
lines(newx,prdPred[,3],col="orange",lty=2)
# with help from [https://stat.ethz.ch/pipermail/r-help/2007-November/146285.html][3]

)

In my view plots like this should be actually be standard practice (especially the 95 % prediction interval) since it communicates the predictions made by the model so clearly, but I have only seen it rarely.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Can you define DV and IV? I'm sorry if I missed that. $\endgroup$ – sooprise Aug 9 '13 at 15:47
  • $\begingroup$ IV = independent variable (predictor variable (could refer to a manipulated variable or a covariate)); DV = dependent variable (outcome variable). I am sure I am forgetting other possible terms. Above, the IV is a (which is years of golf), and the DV is score. $\endgroup$ – A.M. Aug 9 '13 at 17:39
2
$\begingroup$

You could look at the variance in your residuals and see how long it takes for that to begin to converge:

x=NULL; y=NULL; N=1e2; v=NULL
for(i in 1:N){
  new.x=runif(1)
  x=c(x,new.x)
  y=c(y,jitter(5*new.x))

  m=lm(y~x)      
  v=c(v, var(m$residuals))
}

plot(v, type='l', xlab='sample size', ylab='var(residuals)')

enter image description here

This isn't a measure of the quality of your model as much as it is a way of evaluating if you should expect your model to improve with more data. If you're just interested in model quality, for linear models the go-to evaluation of fit is generally the $R^2$ statistic, but there are certainly others. As Justin mentioned, you can reference the F statistic and p-value.

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

When you run your model you should get an F statistic and p-value for the model overall (standard significance guidelines apply) and those also appear for each coefficient. They should take sample size into account.

Here's some more good information: http://blog.yhathq.com/posts/r-lm-summary.html

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.