Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 10 down vote favorite
4

As the title goes, does anyone know of a good, up to date book that covers data preprocessing in general and especially outlier detection techniques?

The book doesn't need to be focusing exclusively on that, but it should deal with the aforementioned topics exhaustively - I wouldn't be happy with something that's a starting point and quotes a list of papers, explanations of the various techniques must appear in the book itself.

Techniques for dealing with missing data preferable, but not necessary...

share|improve this question
Could you tell us what kind of data (scientific field or measurement technique) you are looking at? – cbeleites Apr 12 '12 at 15:03
Data collected from web users (can't be more specific). Included are timestamps (although the data is not strictly time-related, at least intuitively), categorical attributes and continuous attributes. Outliers may be caused by countless reasons, incl. web robots, malicious users and many more sources. The data is also quite big (GBs in CSV format, several millions of entries) – emaster70 Apr 12 '12 at 15:20
For me it's specific enough: no need to bore you with preprocessing for chemical or spectroscopic data sets... – cbeleites Apr 12 '12 at 15:25

3 Answers

up vote 3 down vote

Although specific to Stata, I've found Scott Long's book, The Workflow of Data Analysis Using Stata, invaluable in the area of data management and preparation. The author gives a lot of helpful advice regarding good practices in data management, such as cleaning and archiving data, checking for outliers and dealing with missing data.

share|improve this answer
2  
I love this book too, but I am dyed-in-the-wool Stata user as far as data management is concerned. While I disagree, others on this list have argued that it's too Stata specific to be useful, so caveat emptor/lector. – Dimitriy V. Masterov Apr 11 '12 at 21:36
Very stata-ish from what I gather, and I'm neither familiar with stata, nor would it help for this very project if I were (data is too big, using different technologies) – emaster70 Apr 12 '12 at 15:21
The book is indeed very idiosyncratic. The particular data (and especially meta-data) handling techniques are Stata-specific, but the general ideas are transferable between platforms. I am surprised that with the ratio of about 20 Stata books / 100 R books on the market, there aren't any comparable books on organizing workflow in R -- is the latter impossible? The largest amount of memory I vividly recall allocating to Stata was 48Gb on a 64Gb machine -- that's whether the size matters. If you need to manipulate objects of wildly different structure, you'd want to do this in R, not in Stata. – StasK Apr 12 '12 at 20:33
up vote 0 down vote

For SAS, there is Ron Cody's Data Cleaning Techniques using SAS Software. There is a saying on SAS-L: "You can never go wrong with a book by Ron Cody"

share|improve this answer
I'm afraid SAS is not the tool of choice in my setting nor do I have familiarity with it. Besides, I'm looking for some approach, rather than for a cookbook. Let's say I'm after something that's more on the mathematical and modeling side of things. – emaster70 Apr 12 '12 at 15:22
up vote 0 down vote

If you have the basics (identifying outliers, missing values, weighting, coding) depending on the topic there's a lot more in the plain academic literature to be found. For example in survey research (which is a topic where many things can go wrong, and prone to many sources of bias) there are a lot of good articles to be found.

When preparing for regular crossectional regression, things may be less complex. Problem there may for example that you remove too many 'outliers' and thus artificially fitting your model well.

I thus also recommend you besides learning good techniques, also keep common sense in mind. Make sure you apply the techniques rightfully and not blindly. As for the software discussion in the other answers. I think SPSS is not bad for data preparation (I also heard good things about SAS) depending on your dataset size. The drop down menus are very intuitive.

But as a direct answer to your question, academic literature may or may not be a very good source for your data preparation depending on the topic and analysis.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.