1
$\begingroup$

I trained a regression model and then I tested it on a sample data with 800000 points which give me a root mean square error rmse =17.

Instead of predicting the whole dataset (50 million data points) for me it is enough to get some estimated bound around my rmse. Is there a way to create a confidence bound for rmse based on sample size?

I tried to find some useful information on the internet, but I did not get an answer. Any comment will be very helpful

$\endgroup$
1
$\begingroup$

You can take a Monte-Carlo approach and apply your entire model-fitting procedure to randomly drawn subsets of your data. For each subset, you would perform every step in your process (e.g. feature selection, hyperparameter optimization, model fit) and record the RMSE. From this process, you'll end up with a distribution of RMSE values, and you can infer confidence bounds and whatever else you'd like from that distribution.

There's a good discussion on this in Elements of Stastical Learning in the chapter on cross validation.

I'm interested in a related problem, which is identifying the minimum number of samples you need to obtain a specific confidence bound on the RMSE, without the luxury of a huge dataset to draw from. If anyone comes across a good answer to this please let me know!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.