I would like to know some general parameters that can be used to describe how "dirty" the data is.

Issues I am having are the following:

  • Lots of missing values;
  • The values are some predictors are filled in but often completely wrong;
  • I can try to extract some other variables out of a huge text string, but by doing so I create other doubts of the observations without this long text string.

Is there a general approach to formulate how dirty the data is?

I was thinking about a kind of Sankey diagram that visualises what fraction of the data can be used when taking all the predictors into account?

  • $\begingroup$ There are many problems that can afflict a dataset; you have mentioned some. I don't think that there is any magic threshold from separates data[sets] not worth using from data[sets] worth using. $\endgroup$ – Nick Cox Oct 6 '14 at 13:36
  • $\begingroup$ An after the fact measure could be time spent scrubbing the data / (#records * #fields) $\endgroup$ – JenSCDC Oct 7 '14 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.