0
$\begingroup$

I am asked to determine the test sample size of some data in order to get a 95% confidence interval with width $w$ of a given loss function $L$. The formula in wikipedia is

$$ n = (\frac{4\cdot1.96 \sigma_L}{w})^2. $$

However, this question is asked before building any model for the data, so I do not have any information about the standard deviation $\sigma_L$ of $L$. I tried to use the standard deviation of the trivial model which always predict the mean of the data, but the number is too big. I know that because after building the models I got to a better CI with much fewer data. So

  • I want to know if the question itself makes sense.
  • Any ideas about how to tackle it.

Thank you in advance!

$\endgroup$

1 Answer 1

0
$\begingroup$

The question makes sense. The estimate from the trivial model likely over estimates sigma, which will in turn mean your confidence interval will be narrower than you expect. No matter, the calculation will ensure the resulting CI will have width no larger than $w$.

EDIT: It might be worth examining where this equation comes from. If you write down the length of a confidence interval and solve for $n$, you obtain the formula you've provided. In order for this to be useful, be need an estimate of sigma.

In the case that no variables are predictive of the outcome, then sigma from the trivial model should be sigma for any model which does not over fit on the data (or at the very least, should be very very close to the sigma from any model which does not over fit). Thus, using sigma from the trivial model is a sort of "worst case scenario", and in the case where variables do predict the data then sigma will be smaller and so to $n$.

However, there is no way to know what the "right" sigma, and hence what $n$, to use to get exactly the desired precision prior to fitting a model. But fitting a model requires a train test split. We're in a catch 22. That is why we use the sigma from the trivial model.

$\endgroup$
2
  • $\begingroup$ This is true, but then the question asks if I can achieve the desired CI with the given data, and based on my calculations I said no. Then it turned out it is actually possible. This means the question, in the beginning, wasn't that helpful. $\endgroup$ Feb 22, 2021 at 4:29
  • $\begingroup$ @AhmedElashry The question actually asks if you can achieve the precision prior to fitting a model to the entire data set. Sigma from the trivial model is as best you could do before fitting a model to the data, which would necessitate you split into train and test. You're likely working on an assignment I wrote. $\endgroup$ Feb 22, 2021 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.