Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

A common problem is ML is poor quality of the data: errors in feature values, misclassified instances, etc etc.

One way of addressing this problem is to manually go through the data and check, but are there other techniques? (I bet there are!)

Which ones are better and why?

share|improve this question
add comment

4 Answers

Dimensionality reduction via something like PCA would be helpful to get an idea of the number of dimensions that are critical to represent your data.

To check for misclassified instances, you can do a rudimentary k-means clustering of your data to get an idea of how well your raw data would fit your proposed categories. While not automatic, visualizing at this stage would be helpful, as your visual brain is a powerful classifier in and of itself.

In terms of data that are outright missing, statistics has numerous techniques to deal with that situation already, including imputation, taking data from the existing set or another set to fill in the gaps.

share|improve this answer
3  
Plotting data is a manual check. –  andreister Feb 16 '12 at 7:55
    
@andreister I consider checking point by point on a spreadsheet to be a manual check, but okay, I see what you are getting at. –  jonsca Feb 16 '12 at 7:57
add comment

If you know that your data is not quite good, it is always good to check for outliers as well. Most of the time there are anomalies.

If you have a lot of features, dimensionality reduction is a must. PCA is quite efficient for that.

If you have missing data, you can use imputation or interpolation, but if your needs allows it, the winning case is to use collaborative filtering.

share|improve this answer
add comment

Google Refine might be worth a look.

share|improve this answer
add comment

We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

You can't really remove a knowledgeable person from the loop and expect reasonable results. That doesn't mean that the person has to look at every single item individually, but ultimately it takes some actual knowledge to know if summaries/graphs of data are reasonable. (For example: can variable A be negative, can variable B be larger than variable A, or are there 4 or 5 choices for categorical variable C?)

Once you've had a knowledgeable human look at the data, you can probably make a series of rules that you could use to test the data automatically. The problem is, other errors can arise that you haven't thought about. (For example, a programming error in the data gathering process that duplicates variable A to variable C.)

share|improve this answer
    
Great answer. I would only add to make sure that the syntax used to clean the variables is retained in documentation, with comments if not descriptive passages about why things were changed. :) –  Michelle Feb 23 '12 at 17:07
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.