We have a POC project that is looking for rules that fit our data (eg "when a=1 and b=2 and c=3 then X=6" sort of thing). We split our data into 6 sets, and we use the first 5 sets as K-fold training sets where we say "only keep the rules where the accuracy of the rule on each of the 5 parts is within 5% of the accuracy of the other 4 combined". This gives us a bunch of rules. Then we validate each of these rules against the holdout set to see if it generalizes.
My view is that if the majority (over 90%) of the rules are failing on the holdout set then the way we are finding these rules is too overfitted and that the remaining 10% that pass validation are probably just passing by coincidence but won't perform well on future data. My colleague's view is that if we keep the 10% that passed then these are the ones that have shown to perform well on unseen data and we will have a good set of rules.
What is the standard accepted practice in this scenario? And are there any other tests one can do to say if our methodology is flawed or not?