A quite simple approach is to use the training partition (your 4000 samples) for model tuning and model evaluation (to decide on your single best model), then to use the test partition (your 1000 held-out samples) once for obtaining an estimate on how well your single best model performs on new, unseen data.
Besides others, this can be done using repeated cross validation on the training set. With e.g. using 10 partitions and 20 repeats, you will train each model type and parameter set 200 times and obtain 200 performance estimates for each. The spread of those estimates gives you a hint on how likely it is to obtain a model that performs badly using this model type and parametrization. This information can be used to decide upon the best suited model type and parametrization then. After doing so, the chosen model is trained using all data in the training partition (usually done automatically by your ML software), then used to predict the target variable for the held-out test partition for a final performance estimate.
Here's a simple example on how this could look like (using
> model <- train(x = iris[,1:3], y = iris[,4], method = 'lm',
trControl = trainControl(method = 'repeatedcv', number = 10, repeats = 20))
RMSE Rsquared Resample
1 0.1485536 0.9667557 Fold01.Rep01
2 0.2404998 0.9097242 Fold02.Rep01
3 0.2173052 0.9494554 Fold03.Rep01
4 0.1733321 0.9486735 Fold04.Rep01
5 0.1871449 0.9429690 Fold05.Rep01
6 0.2347253 0.9206204 Fold06.Rep01
7 0.2419445 0.8938819 Fold07.Rep01
8 0.1856019 0.9603457 Fold08.Rep01
9 0.1195387 0.9785385 Fold09.Rep01
> boxplot(model$resample$RMSE, main = 'RMSE')
PS: this result is not a classic confidence interval (which can be calculated as well), but still employed quite frequently as it gives you a good hint about what is going on with your partitions and resamples.