By hyper-parameters I mean not only the machine learning algorithm hyper-parameters (learning rate, etc.), but also hyper-parameters like "what's the ideal number of data points to use" or "which features should you include or not include" or even hyper-parameters associated with how the data is prepared.
My thought was that you can just run exhaustive searches (maybe even grid searches) on all kinds of hyper-parameter values and combinations, and just use cross-validation to avoid over-training, but I'm wondering whether or not that's contrary to best practices. I suppose that if you run enough tests on something as integral as the features used, you'll end up with a features that just happen to be highly correlated with your training/validation set, but then what's the guideline on "making sure you don't run too many tests"?
For reference this is with respect to a fast training model like xgboost (though this is more of a general question), and computational time isn't really an issue for me since I'm mostly only concerned with accuracy/performance, even if it takes me weeks to run the exhaustive searches. And if it matters I'm looking at a time-series dataset with datapoints in the order of 10,000s.
note: I apologize in advance since I'm just an amateur trying to do machine learning, any thoughts, even basic ones, would be highly appreciated!