How to do exploratory data analysis to choose appropriate machine learning algorithm

We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case which algorithm is better, and when it does, it doesn't say how to tell which case I'm in.

For example, for the choice of kernel, I've been told to do exploratory data analysis to gauge how complex my data is. In simple 2 dimensional data, I can plot and see whether a linear or radial kernel is appropriate. But what to do in higher dimension?

More generally, what do people mean when they say "get to know your data" before choosing an algorithm? Right now I can only distinguish classification vs regression algorithm, and linear vs non-linear algorithm (which I can't check).

EDIT: Even though my original question is about universal rule of thumb, I've been asked to provide more info on my particular problem.

Data: A panel with each row being a country-month (~30,000 rows total, covering ~165 countries over ~15 years).

Response: 5 binary variables of interest (i.e. whether protest / coup / crisis, etc. happen in that month).

Features: ~ 400 variables (a mix of continuous, categorical, binary) detailing a bunch of characteristic of the 2 previous country-months (longer lag can be created). We only use lagged variable since the goal is prediction.

Examples include, exchange rate, GDP growth (continuous), level of free press (categorical), democracy, whether neighbor having conflict (binary). Note that a lot of these 400 features are lagged variables.

This is a broad question without a simple answer. At CMU I taught a 3-month course on this topic. It covered issues such as:

1. Using projections to understand correlation between variables and overall distributional structure.
2. How to build up a regression model by successively modelling residuals.
3. Determining when to add nonlinear interaction terms to a linear model.
4. How to decide between knn vs. a decision tree vs. a logistic classifier. I went through a number of UCI datasets and showed how you could tell which classifier would win before running them.

Sadly, there is no video or textbook for the course, but I gave a talk that summarizes the main points from the class. I'm not aware of any textbook that covers the same ground.

• I will take a day or two to digest these helpful materials, but while I have your attention: Why don't we have a textbook / resource covering this topic? Isn't it important since whenever someone engages in a project they have to think about this question? Oct 17 '14 at 16:59
• Nice question (+1) and answer (+1). @Heisenberg: I agree with Tom in not having seen a specific textbook on the topic. However, in addition to his resources, I would suggest two online resources (despite them not being focused on ML applications): 1) the EDA section of the NIST Engineering Statistics Handbook; 2) an interesting paper by Prof. Andrew Gelman on EDA for complex models. Mar 28 '15 at 13:29

There are some things that you can check in your data.

1 - correlation between variables
2 - categorical variables or continuous variables?
3 - relation between number of samples and number of variables
4 - are the samples independent or is it a time series?

According to these points and to the kind of information you want to extract from your data you can decide what algorithm to use.

• Could you elaborate how each of these 4 information influence my algorithm choice? I only know that 2 will decide classification vs regression. What about the other 3? (especially #4 -- I have panel data of 165 countries over 10 years) Oct 17 '14 at 0:05
• In 2- I was thinking about categorical variables as input. The final decision on the algorithm depends on the problem that you are trying to solve. There is now way to know that before. In 2- maybe a decision tree can help you. In 3 you have to be careful about overfitting . In 4- you have to decide how to evaluate your performance. Only if you explain a particular problem we can help you to decide what algorithm to use. Oct 17 '14 at 0:10
• I have edited my question for more details about my particular problem. Oct 17 '14 at 0:22