In Wadoux et al. 2021 I came across two modes of statistical inference, design-based and model-based. I get the difference between the two modes and I know what that means regarding a sampling scheme (design-based relies solely on randomness while model-based relies on the model parameters). In the mentioned paper they calculated a Random Forest and validated with so called design-based validation. Furthermore they state that they did a Simple Random Sampling (SRS) and estimated a design-based RMSE of the population. The given formula is: $\widehat{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^n(\hat{z}(s_i)-z(s_i))^2}$. The lower case n indicates that the RMSE is calculated from a drawn sample. My question regarding that is what's the workflow when using design-based validation for e.g. Random Forest validation? Do I draw one SRS for calibrating the model and one SRS for validation? Or am I drawing one SRS for calibration and validation (that would be weird)?
Furthermore what exactly is the difference to Cross Validation (CV)? In standard CV the data is shuffled and then divided into k-folds (for calibration and validation). In my opinion that's very close (if not equal) to probabilistic sampling. So why is standard CV not design-based despite a random sampling design?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.