Assuming a 500k dataset. For statistical modeling purposes (selecting up to 10% of 500k as a sample). Should I clean the 500k dataset first before selecting a sample or select the sample first and then clean the data?

What if this goes into bigger datasets such as 1 million+ rows?

  • 3
    $\begingroup$ The answer likely depends on what you really mean by "statistical modeling purposes." Sometimes the "cleaning" is part of the modeling and sometimes not. $\endgroup$
    – whuber
    May 14 at 14:47
  • $\begingroup$ Statistical modeling purposes........means building a statistical model. If the dataset is large, I only need roughly a 10% sample. If the full dataset is dirty to begin with, do I clean it first before extracting a sample or do I extract a sample and cleanup if it randomly drew dirty data. $\endgroup$
    – Jiaming He
    May 14 at 16:04
  • 1
    $\begingroup$ "Cleaning" might or might not be part of building your model. That's the issue. $\endgroup$
    – whuber
    May 14 at 18:05
  • $\begingroup$ Assume cleaning is part of the process. $\endgroup$
    – Jiaming He
    May 17 at 1:56
  • $\begingroup$ Then include it in the process! $\endgroup$
    – whuber
    May 17 at 13:09


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.