Is exploratory data analysis important when doing purely predictive modeling?

Question

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your model(s)? How are descriptive statistics used in EDA important?

Answer 1

Not long ago, I had an interview task for a data science position. I was given a data set and asked to build a predictive model to predict a certain binary variable given the others, with a time limit of a few hours.

I went through each of the variables in turn, graphing them, calculating summary statistics etc. I also calculated correlations between the numerical variables.

Among the things I found were:

• One categorical variable almost perfectly matched the target.
• Two or three variables had over half of their values missing.
• A couple of variables had extreme outliers.
• Two of the numerical variables were perfectly correlated.
• etc.

My point is that these were things which had been put in deliberately to see whether people would notice them before trying to build a model. The company put them in because they are the sort of thing which can happen in real life, and drastically affect model performance.

So yes, EDA is important when doing machine learning!

Answer 2

Obviously, yes.

The data analysis could lead you to many points that would hurt your predictive model :

Incomplete data

Assuming we are talking about quantitative data, you'll have to decide whether you want to ignore the column (if there's too much data missing) or figure out what will be your "default" value (Mean, Mode, Etc). You can't do this without exploring your data first.

Abnormal data

Say you have data that is pretty strongly correlated but there is a 2% of your data that is way off this correlation. You might want to remove this data altogether to help your predictive model

Remove columns with too much correlation

Ok this contradicts a little bit my previous point but english isn't my main language so I hope you'll understand.

I'll take a dumb example, say you analysis a football's stadium dataset and you have Width, Length, Area as parameters. Well, we can easily imagine that these three parameters will be strongly correlated. Having too much correlation between your column leads the predictive model in a wrong direction. You might decide to flush one or more of the parameters.

Find new features

I'll take the example of the small Titanic Kaggle "Competition". When looking at the folks' names, you could figure out that you can extract a feature that is the Title of the person. This feature turns out to be pretty important when it comes to modeling, but you would have missed it if you didn't analyse your data first.

You might decide to bin your continuous data because it feels more appropriate or turn a continuous feature into a categorical one.

Find what kind of algorithm to use

I can't draw plots right now, but let's make this a simple example.

Imagine that you have a small model with one feature column and one binary (0 or 1 only) "result" column. You want to create a predictive classifying model for this dataset.

If, once again as an example, you were to plot it (soo, analyse your data), you might realise that the plot forms a perfect circle around your 1 value. In such a scenario, if would be pretty obvious that you could use a polynomial classifier to have a great model instead of jumping straight to the DNN. (Obviously, considering there's only two columns in my example, it doesn't make for a excellent example, but you get the point)

Overall, you can't expect a predictive model to perform well if you don't look at the data first.

Answer 3

One important thing done by EDA is finding data entry errors and other anomalous points.

Another is that the distribution of variables can influence the models you try to fit.

Answer 4

We used to have a phrase in chemistry:

"Two weeks spent in the lab can save you two hours on Scifinder".

I'm sure the same applies to machine learning:

"Two weeks spent training a neuralnet can save you 2 hours looking at the input data".

These are the things I'd go through before starting any ML process.

• Plot out the density of every (continuous) variable. How are the numbers skewed? Do I need a log transform to make the data make sense? How far away are the outliers? Are there any values that do not make physical or logical sense?
• Keep an eye out for NAs. Usually, you can just discard them, but if there are a lot of them, or if they represent a crucial aspect to the behaviour of the system, you might have to find a way of recreating the data. This could be a project in and of itself.
• Plot every variable against the response variable. How much sense can you make out of it just by eyeballing it? Are there obvious curves that can be fitted with functions?
• Assess whether or not you need a complicated ML model in the first place. Sometimes linear regression is all you really need. Even if it isn't, it provides a good baseline fit for your ML model to improve upon.

Beyond those basic steps, I wouldn't spend much additional time looking at the data before applying ML processes to it. If you already have a large number of variables, complicated nonlinear combinations of them get increasingly difficult not only to find, but to plot and understand. This is the sort of stuff best handled by the computer.

Answer 5

Statistical perspective:

Leaving aside errors in the modelling stage, there are three likely outcomes from attempting prediction without first doing EDA:

1. Prediction gives obvious nonsense results, because your input data violated the assumptions of your prediction method. You now have to go back and check your inputs to find out where the problem lies, then fix the issue and redo the analysis. Depending on the nature of the issue, you may even need to change your prediction methods. (What do you mean, this is a categorical variable?)
2. Prediction gives results that are bad but not obviously bad, because your data violated assumptions in a slightly less obvious way. Either you go back and check those assumptions anyway (in which case, see #1 above) or you accept bad results.
3. By good fortune, your input data is exactly what you expected it to be (I understand this does occasionally happen) and the prediction gives good results... which would be great, except that you can't tell the difference between this and #2 above.

Project-management perspective:

Resolving data issues can take a significant amount of time and effort. For instance:

• The data is dirty and you need to spend time developing processes to clean it. (For example: the time I had to code an autocorrect for all the people who keep writing the wrong year in January, and the people who enter the date in the year field, and the system that was parsing dates as MM/DD/YYYY instead of DD/MM/YYYY.)
• You need to ask questions about what the data means, and only Joan can answer them. Joan is going on a six-month holiday, starting two weeks after your project begins.
• Data limitations prevent you from delivering everything you had intended to deliver (cf. Bernhard's example of being unable to produce analysis by sex/gender because the data set only had one woman) and you/your clients need to figure out what to do about that.

The earlier you can identify such issues, the better your chances of keeping your project on the rails, finishing on time, and making your clients happy.