Guidelines for discovering new knowledge in data

Question

I plot something to make a point to myself or someone else. Usually, a question starts this process, and often the person asking hopes for a particular answer.

How can I learn interesting things about the data in a less biased way?

Right now I'm roughly following this method:

1. Summary statistics.
2. Stripchart.
3. Scatter plot.
4. Maybe repeat with an interesting subset of data.

But that doesn't seem methodical or scientific enough.

Are there guidelines or procedures to follow that reveal things about the data I wouldn't think to ask? How do I know when I have done an adequate analysis?

Answer 1

There's a whole field of exploratory data analysis (EDA), and an excellent book on this subject called Exploratory Data Analysis, by John W. Tukey.

I like that you are using graphs - there are many other graphs that can be useful, depending on your data - how many variables? What nature are the variables (Categorical? Numeric? Continuous? Counts? Ordinal?)

One graph that is often useful for data with multiple variables is a scatterplot matrix.

You can look for various types of outliers, which are often interesting points.

But I don't think this whole process can be made really methodical and scientific - exploration is what comes BEFORE the methodical and scientific approaches can be brought in. Here, I think the key aspect is playfulness.

Answer 2

If you have chronological data i.e.time series data then there are "knowns" and waiting to be discovered are the "unknowns" . For example if you have a sequence of data points for 10 periods such as 1,9,1,9,1,5,1,9,1,9 then based upon this sample one can reasonably expect 1,9,1,9,... to arise in the future. What data analysis reveals is that there is an "unusual" reading at period 6 even though it is well within +-3 sigma limits suggesting that the DGF did not hold. Unmasking the Inlier/Outlier allows us to reveal things about the data. We also note that the Mean Value is not the Expected Value. This idea easily extends to detecting Mean Shifts and/or Local Time Trends that may have been unknown before the data was analyzed ( Hypothesis Generation ). Now it is quite possible that the next 10 readings are also 1,9,1,9,1,5,1,9,1,9 suggesting that the "5" is not necessarily untoward. If we observe an error process from a suitable model that exhibits provable non-constant variance we might be revealing one of the following states of nature: 1) the parameters might have changed at a particular point in time ; 2. There may be a need for Weighted Analysis (GLS) ; 3. There may be a need to transform the data via a power transform; 4. There may be a need to actually model the variance of the errors. If you have daily data good analysis might reveal that there is a window of response (lead,contemporaneous and lag structure) around each Holiday reflecting consistent/predictable behavior. You might also be able to reveal that certain days of the month have a significant effect or that Fridays before a Monday holiday have exceptional activity.

Answer 3

Datamining could be broken down into two categories. If you are interested in measuring the effect of a data set/variables on a specific variable then this would be considered supervised learning. For deep and exploratory learning with no objective you are undergoing unsupervised learning.

Graphing and statistical analysis of the data (understanding distributions and gaining intuition) are the first steps.