1
$\begingroup$

I have trained a regressor to predict the revenue generated when users click on an ad impression. Only few clicks eventually lead to positive revenue. As such, I have used the Tweedie regression in XGBoost to train the model.

My question is, how many samples do i need to train on for a given ad to have confidence in its prediction? Is there a way to compute the margin of error?

I am familiar with computing the margin of error for classification tasks (sample proportions). However, I am not sure how to do this for sample mean.

When I searched online, most texts (for example) state that one needs to know thee population standard deviation in order to compute the margin of error. But that itself is unknown here.

My idea is that, I can define a margin of error (say 1% of sample mean) and see if the given sample size gives me a margin of error less than this threshold. I can use this formulation to compute a minimum sample size.

I don't have a formal stats background. So, might be totally off here.

$\endgroup$
1
$\begingroup$

You can use the sample standard deviation to estimate the population standard deviation (using the Bessel correction, for example), and then compute a standard $ t $-statistic with this information. Under assumptions of normality, the distribution of the test statistic is well known, so you can use standard $ p $-value tables to create a hypothesis test, which will give you a confidence interval for the population mean given the sample mean.

If the assumption of normality is not met, then depending on your sample size you might have to be careful; but the estimator is asymptotically normal, so as long as the true distribution is not too far from a normal (doesn't have high kurtosis relative to the sample size and so forth) you can reliably use these $ t $-statistic tables to get confidence intervals, i.e. margins of error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.