2
$\begingroup$

Assume a multiple linear regression model with $F$ features and $N$ samples. For each feature, a standard error on the $\beta$ coefficient and a Student's T P-value can be calculated. The P-value tests the null that the $\beta$ coefficient is zero against the alternative that the $\beta$ coefficient is nonzero.

If I want to maintain a fixed standard error on the $\beta$ coefficients and a fixed power in the Student's T tests, how does the required $N$ scale with increasing $F$, in terms of big-O notation? Similarly, what would happen to my standard errors and statistical power if I doubled both $N$ and $F$?

Improve this question
$\endgroup$
1
  • 6
    $\begingroup$ That doesn't seem to me to be a sufficiently well-defined problem; for example, if you add one feature, but it's highly informative w.r.t. the residuals of the "current" model, the std. errors on the existing coefficients will go down, perhaps substantially, even with no increase in sample size; if the additional feature is sufficiently close to orthogonal to the residuals, it may increase the std. errors of the existing coefficients. $\endgroup$
    – jbowman
    Dec 23, 2011 at 23:23

2 Answers 2

Reset to default
5
$\begingroup$

It all depends on the question. For most situations I would not consider this from the power standpoint as was done in http://personal.health.usf.edu/ywu/logistic.pdf. Rather model reliability (e.g., calibration) is often a more appropriate consideration. For that purpose, start by considering simple cases with no competing covariates.

Suppose there were no covariates, so that the only parameter in the model were the intercept. What is the sample size required to allow the estimate of the intercept to be precise enough so that the predicted probability is within 0.1 of the true probability with 0.95 confidence, when the true intercept is in the neighborhood of zero? The answer is n=96. What if there were one covariate, and it was binary with a prevalence of $\frac{1}{2}$? One would need 96 subjects with $X=0$ and 96 with $X=1$ to have an upper bound on the margin of error for estimating $\textrm{Prob}\{Y=1|X=x\}$ not exceed 0.1 for either value of $x$. The general formula for the sample size required to achieve a margin of error of $\delta$ in estimating a true probability of $\theta$ at the 0.95 confidence level is $n = (\frac{1.96}{\delta})^{2} \times \theta(1 - \theta)$. Set $\theta = \frac{1}{2}$ (intercept=0) for the worst case.

This idea can be extended by simulation of a number of continuous covariates or by imagining the effective number of unique orthogonal "groups" there are in the covariate space for which one desires good prediction accuracy. $96\times k$ where $k$ is the number of "groups" would be an estimate of the required sample size.

Improve this answer
$\endgroup$
6
  • $\begingroup$ +1 this is an outstanding answer. It takes a question which I thought wasn't really answerable (I had agreed with @jbowman), and offers real insight. I wish I could upvote it more than once. $\endgroup$
    – gung - Reinstate Monica
    Dec 28, 2011 at 18:12
  • $\begingroup$ Hi Prof Harrell, why does $\theta = \frac{1}{2}$ imply that intercept equals 0? I approached this as follows: $log(\frac{0.5}{0.5})= 0 = \beta_0 + \beta_1 I(X=x)$. Is $\beta_1 = 0$ because prevalence is $50\%$? Thanks. $\endgroup$
    – user1916067
    Sep 9, 2022 at 1:30
  • $\begingroup$ $\theta = \frac{1}{2}$ if and only if $\beta_{0}=0$ when there are zero covariates in the model. The no-covariate case is the simplest case requiring the smallest sample size. $\endgroup$
    – Frank Harrell
    Sep 9, 2022 at 14:19
  • $\begingroup$ Got it. Thanks. $\endgroup$
    – user1916067
    Sep 9, 2022 at 20:16
  • $\begingroup$ @Prof. Harrel: I see that this estimation is based on the logit model framework. Does this extend to mlr without further considerations? Thanks $\endgroup$
    – user1916067
    Sep 10, 2022 at 9:11
0
$\begingroup$

Well I guess the following paper answers your question at least partially. The paper gives the sample size and its upper bounds for Linear as well as Logistic regression. The case where there are already , say p covariates in the model and the effect of adding some more is also treated there in case of multiple linear regression on page 3. The main test presented is to check the significance of regression i.e. whether betas are significantly different from zero or not and the sample size is given for that.

http://personal.health.usf.edu/ywu/logistic.pdf

Improve this answer
$\endgroup$
1
  • $\begingroup$ Can you share the actual title of the paper? The link is broken. $\endgroup$
    – user1916067
    Sep 10, 2022 at 8:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.