I am creating a dataset that I am going to use as the training/test sets for a supervised classification problem. The problem is to identify quotations in a large corpus. I have randomly sampled the pages of the corpus and extracted the features that I am going to use as predictors. I have a table of with many potential matches, the vast majority of which are noise. One or two predictors in particular are the most useful. One predictor ranges from 0 to 1. (It basically measures the percentage of the n-grams from the quotation that are present on the page.) If there is a high value (> 0.1) for this predictor, then it is almost certainly a match.

I am going to take a sample of these values and verify by hand whether or not there is actually a match or not. If I do a simple random sample, then very few genuine matches will show up. So, though I know that there are thousands of potential matches that have a score above 0.25, if I take a random sample of 1000 rows from the dataset, then the maximum score in the dataset is 0.25.

I intend to do an disproportionate stratified random sample. I will define the strata as having a high proportion (> 0.1) and a low proportion on this predictor. Then I will take an equal number of rows from each stratum.

So here is the question: Is this a valid approach when constructing a training/test set, as opposed to actually using the set to create the model?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.