Sample size required for a given margin of error in sample mean when population standard deviation is not known

Question

I have trained a regressor to predict the revenue generated when users click on an ad impression. Only few clicks eventually lead to positive revenue. As such, I have used the Tweedie regression in XGBoost to train the model.

My question is, how many samples do i need to train on for a given ad to have confidence in its prediction? Is there a way to compute the margin of error?

I am familiar with computing the margin of error for classification tasks (sample proportions). However, I am not sure how to do this for sample mean.

When I searched online, most texts (for example) state that one needs to know thee population standard deviation in order to compute the margin of error. But that itself is unknown here.

My idea is that, I can define a margin of error (say 1% of sample mean) and see if the given sample size gives me a margin of error less than this threshold. I can use this formulation to compute a minimum sample size.

I don't have a formal stats background. So, might be totally off here.

Answer 1

You can use the sample standard deviation to estimate the population standard deviation (using the Bessel correction, for example), and then compute a standard $ t $-statistic with this information. Under assumptions of normality, the distribution of the test statistic is well known, so you can use standard $ p $-value tables to create a hypothesis test, which will give you a confidence interval for the population mean given the sample mean.

If the assumption of normality is not met, then depending on your sample size you might have to be careful; but the estimator is asymptotically normal, so as long as the true distribution is not too far from a normal (doesn't have high kurtosis relative to the sample size and so forth) you can reliably use these $ t $-statistic tables to get confidence intervals, i.e. margins of error.