My supervisor once told me that before I run any classifier or do anything with a dataset I must fully understand the data and make sure that the data is clean and correct.

My questions:

What are the best practices to understand a dataset (high dimensional with numerical and nominal attributes)?

Practices to make sure the dataset is clean?

Practices to make sure the dataset doesn't have wrong values or so?


Pre-processing is a key step in any machine learning exercise, and an important part of pre-processing is getting familiar with the data.

The obvious starting point is talking to experts and getting a grasp of the domain, be it clinical, internet, survey data, etc. It does obviously help to understand the semantics of the variables.

Focusing on subsets and producing plots would definitely help (e.g. how does a particular variable affect the outcome?) What happens when you look at two variables and the outcome? Obviously, you will not be able to visualise the entire joint probability distribution (due to high dimensionality) but even scratching the surface is still useful.

If there is missingness, you shold talk to the people who collected the data on issues like: is it safe to assume missing at random (MAR) or even missing completely at random (MCAR)?

As you spend more time looking at the data, you will most likely discover inconsistencies. You will need to make asusmptions to address/correct these. Again flagging these up, investigating why they are so, and speaking to the collector of the data (if it is not you) is very important.

Finally, before going the full length to try whatever complicated ML algorithm you want to try (e.g. a Bayesian network), it is good throw the data into a fast and relatively simple punching bag algorithm (e.g. logistic regression or naive bayes - although there are rather complicated versions of the former) as a first step. This is like your rehersal. It will let you know your reference points in terms of model performance and the process will inevitably get you more familiar with the data.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.