I have recently implemented a Regression machine learning algorithm that predicts house prices. We have defined Mean absolute error as an intuitive metric to calculate how much off each prediction can be, mimicking that way confidence intervals for each prediction. I would now like to evaluate the Mean absolute percentage error of the algorithm when used in a "real world setting".
In order to do the evaluation, there is a need for manual data gathering, since this is labor-intensive I would like to keep the evaluation sample as small as possible. Thus based on a confidence level and a margin of error in my MAPE metric I would like to calculate the sample size required. For example, let's assumed that I would like to retrieve a MAPE with 95% confidence level and 5% margin of error, so I can say that when my model predicts 1000 dollars, it is with 95% confidence 1000 +- (1000Mape5%).
It seems that the Cochran’s Sample Size Formula Cochran’s Sample Size is used for binary distributions which is not the case here. Thus in order to calculate the mean absolute percentage error of the population for a certain confidence level and margin error we will need to use the below formula: However, the formula requires the standard deviation of the population as well as the mean of the population. Which are not known.
I see here two work arounds:
- We already have a sample of 20 houses. Could we boostrap this sample to calculate a proxy of the population mean and standard deviation?
- Can I use t table rather than z table?
Also could you comment on our decision to use MAPE as confidence bounds for each prediction. So if the MAPE is 10% (calculated during testing) and the prediction of the model is 500.000 then we want to say that the house price lies between 550.000 and 450.000 dollars. Are there any better ways to add confidence intervals in my regression predictions? Should we use prediction intervals instead of MAPE? and if yes then how can we estimate the appropriate size for production testing?