Among many methods used to detect and avoid overfitting, I am particularly interested in those two:

  1. replica method
  2. reusable holdout

My question is: what is their relation in the context of adaptive data analysis and model selection? To be more concrete, let's focus on developing a multivariate logistic classifier. We are given a dataset of $N$ $D$-dimensional points $x$ associated with binary labels $y$. $N$ and $D$ are large. We want to select which dimensions (i.e. variables) of $x$ should be used by the classifier. Is it possible to say which method of the above two would in general be better for this purpose? If yes, which one?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.