3
$\begingroup$

This question is completely theoretical in the sense that I don't have any real data. Suppose I am designing a study, in which I will go measure N pairs of (X,Y), then the goal is to build a predictive model (say linear regression) between X and Y. i.e. I want good predictive accuracy, but it will also be good to have a prediction interval.

Given the limited resources, I have two choices:

  1. Increase N, but X will be measured less accurately; or
  2. Decrease N but measure X more accurately.

The question is, how can I find the balance between the 2 cases?

I understand I haven't provided real data, which is why I am not asking for specific answer. I just want to know what's the line of thought you would follow? What's a good way to approach this question? etc....

Clarification: Don't know why I put it the other way around. X in the dot points above should be Y. I am predicting Y using X. My choice is between increasing N or measuring Y more precisely.

Added: It will be good if your response considers both small sample size and adequate sample size situations.

$\endgroup$
1
  • $\begingroup$ You're predicting X from Y or Y from X? If you're predicting X from Y, the reversal of the usual direction may confuse some readers (a minor point but still worth considering and definitely worth lampshading). If you're predicting Y from X but X is measured with error, you shouldn't use ordinary linear regression for that. $\endgroup$
    – Glen_b
    Jun 11, 2014 at 7:02

1 Answer 1

3
$\begingroup$

I'll assume for the moment you're predicting X from Y and it's only the variable being predicted that's measured with error (see the warning in my comment if this is not the case). I'll interchange the roles of X and Y to be the usual way around. That is, in what follows $Y$ is the variable being predicted, and is measured with error, while $X$ is the predictor or predictors.

The calculation is reasonably straightforward.

The variance of a confidence interval is $\sigma^2\mathbf{x^*}(X^TX)^{-1}\mathbf{x^*}^T$ * where $\mathbf{x^*}$ is the row vector of predictors. If the rows of $X$ are random draws from some population of predictors, than as $n$ grows, $n(X^TX)^{-1}$ approaches a constant.

*(while that for a prediction interval is $\sigma^2(1+\mathbf{x^*}(X^TX)^{-1}\mathbf{x^*}^T)$)

That is, there's a $\sigma/\sqrt{n}$ type effect in the standard error of the mean forecast at $\mathbf{x^*}$ (just as with an average).

You need 4 times the data to make the standard error halve, but you only need to halve the average error (make measurements with twice the precision) to have the same effect.

If you can make the measurements have 90% of the error, that would be equivalent to having ($1/0.9^2 \approx 1.235$) 23.5% more observations.

Because $\sigma$ should be proportional to the average error, while $1/\sqrt{n}$ decreases as the square of the sample size, the tradeoff is pretty easy to work with.

If you're talking prediction intervals rather than confidence intervals, larger $n$ is unlikely to be worthwhile; it soon gives almost no return.

$\endgroup$
6
  • $\begingroup$ +1 for your great response. I will probably accept the answer at the end. However, the reason why I asked this is because we have a small sample size problem. N<10, if I don't try to increase it. In this case, asymptotic don't apply. I believe n(X^TX)^-1 will decrease with increasing n quite dramatically in this range of n. What would your response be in light of this? $\endgroup$
    – qoheleth
    Jun 12, 2014 at 0:13
  • $\begingroup$ The if the rows of $X$ are being drawn from a common "population" of rows (in the sense that $n(X^TX)^{-1}$ is approximately constant as $n$ increases), the variance calculation isn't asymptotic, but applies even to small samples. That is, even for $n<10$, doubling your $n$ should tend to divide the standard error for a predicted mean by $\sqrt 2$, just as it does when $n$ is large. I see no basis on which you could assert that $\mathbf{n}(X^TX)^{-1}$ will decrease rapidly as $n$ increases (note $n\times$), unless you are doing something very strange indeed. Why do you think that can happen? $\endgroup$
    – Glen_b
    Jun 12, 2014 at 0:23
  • $\begingroup$ mmm.... I take $n(X^TX)^{-1}$ to mean $1/\hat{var}(X)$ (ignoring the mean). When $n=1$ $\hat{var}(X)$ is 0 (or undefined). But when you gradually include more X, wouldn't $\hat{var}(X)$ increases, and eventually stabilizes around $var(X)$? rx<-replicate(100,{x<-rnorm(10,0,10); sapply(1:10,FUN=function(z)1/var(x[1:z]))}); plot(apply(rx,1,median,na.rm=TRUE)) $\endgroup$
    – qoheleth
    Jun 12, 2014 at 1:10
  • $\begingroup$ Well, it's an uncorrected inverse of average squares and crossproducts but if you center it, it's like the inverse of a variance, yes. However, there's two levels of approximation in there that confound that reasoning. In actuality, it's even more like an inverse of an expectation, and (under assumptions stated above) that's unbiased even at n=1. $\endgroup$
    – Glen_b
    Jun 12, 2014 at 3:25
  • $\begingroup$ can you elaborate on 'its even more like an inverse of an expectation'? $\endgroup$
    – qoheleth
    Jun 12, 2014 at 23:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.