2
$\begingroup$

Assume a multiple linear regression model with $F$ features and $N$ samples. For each feature, a standard error on the $\beta$ coefficient and a Student's T P-value can be calculated. The P-value tests the null that the $\beta$ coefficient is zero against the alternative that the $\beta$ coefficient is nonzero.

If I want to maintain a fixed standard error on the $\beta$ coefficients and a fixed power in the Student's T tests, how does the required $N$ scale with increasing $F$, in terms of big-O notation? Similarly, what would happen to my standard errors and statistical power if I doubled both $N$ and $F$?

$\endgroup$
  • 6
    $\begingroup$ That doesn't seem to me to be a sufficiently well-defined problem; for example, if you add one feature, but it's highly informative w.r.t. the residuals of the "current" model, the std. errors on the existing coefficients will go down, perhaps substantially, even with no increase in sample size; if the additional feature is sufficiently close to orthogonal to the residuals, it may increase the std. errors of the existing coefficients. $\endgroup$ – jbowman Dec 23 '11 at 23:23
5
$\begingroup$

It all depends on the question. For most situations I would not consider this from the power standpoint as was done in http://personal.health.usf.edu/ywu/logistic.pdf. Rather model reliability (e.g., calibration) is often a more appropriate consideration. For that purpose, start by considering simple cases with no competing covariates.

Suppose there were no covariates, so that the only parameter in the model were the intercept. What is the sample size required to allow the estimate of the intercept to be precise enough so that the predicted probability is within 0.1 of the true probability with 0.95 confidence, when the true intercept is in the neighborhood of zero? The answer is n=96. What if there were one covariate, and it was binary with a prevalence of $\frac{1}{2}$? One would need 96 subjects with $X=0$ and 96 with $X=1$ to have an upper bound on the margin of error for estimating $\textrm{Prob}\{Y=1|X=x\}$ not exceed 0.1 for either value of $x$. The general formula for the sample size required to achieve a margin of error of $\delta$ in estimating a true probability of $\theta$ at the 0.95 confidence level is $n = (\frac{1.96}{\delta})^{2} \times \theta(1 - \theta)$. Set $\theta = \frac{1}{2}$ (intercept=0) for the worst case.

This idea can be extended by simulation of a number of continuous covariates or by imagining the effective number of unique orthogonal "groups" there are in the covariate space for which one desires good prediction accuracy. $96\times k$ where $k$ is the number of "groups" would be an estimate of the required sample size.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ +1 this is an outstanding answer. It takes a question which I thought wasn't really answerable (I had agreed with @jbowman), and offers real insight. I wish I could upvote it more than once. $\endgroup$ – gung - Reinstate Monica Dec 28 '11 at 18:12
0
$\begingroup$

Well I guess the following paper answers your question at least partially. The paper gives the sample size and its upper bounds for Linear as well as Logistic regression. The case where there are already , say p covariates in the model and the effect of adding some more is also treated there in case of multiple linear regression on page 3. The main test presented is to check the significance of regression i.e. whether betas are significantly different from zero or not and the sample size is given for that.

http://personal.health.usf.edu/ywu/logistic.pdf

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.