I am creating a dataset that I am going to use as the training/test sets for a supervised classification problem. The problem is to identify quotations in a large corpus. I have randomly sampled the pages of the corpus and extracted the features that I am going to use as predictors. I have a table of with many potential matches, the vast majority of which are noise. One or two predictors in particular are the most useful. One predictor ranges from 0 to 1. (It basically measures the percentage of the n-grams from the quotation that are present on the page.) If there is a high value (
> 0.1) for this predictor, then it is almost certainly a match.
I am going to take a sample of these values and verify by hand whether or not there is actually a match or not. If I do a simple random sample, then very few genuine matches will show up. So, though I know that there are thousands of potential matches that have a score above 0.25, if I take a random sample of 1000 rows from the dataset, then the maximum score in the dataset is 0.25.
I intend to do an disproportionate stratified random sample. I will define the strata as having a high proportion (
> 0.1) and a low proportion on this predictor. Then I will take an equal number of rows from each stratum.
So here is the question: Is this a valid approach when constructing a training/test set, as opposed to actually using the set to create the model?