I have trained a regressor to predict the revenue generated when users click on an ad impression. Only few clicks eventually lead to positive revenue. As such, I have used the Tweedie regression in XGBoost to train the model.
My question is, how many samples do i need to train on for a given ad to have confidence in its prediction? Is there a way to compute the margin of error?
I am familiar with computing the margin of error for classification tasks (sample proportions). However, I am not sure how to do this for sample mean.
When I searched online, most texts (for example) state that one needs to know thee population standard deviation in order to compute the margin of error. But that itself is unknown here.
My idea is that, I can define a margin of error (say 1% of sample mean) and see if the given sample size gives me a margin of error less than this threshold. I can use this formulation to compute a minimum sample size.
I don't have a formal stats background. So, might be totally off here.