Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
86
votes
24answers
9k views

Rules of thumb for “modern” statistics

I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common ...
73
votes
6answers
23k views

Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?

In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms. I therefore ...
53
votes
8answers
9k views

Modern successor to Exploratory Data Analysis by Tukey?

I've been reading Tukey's book "Exploratory Data Analysis". Being written in 1977, the book emphasizes paper/pencil methods. Is there a more 'modern' successor which takes into account that we can ...
39
votes
8answers
30k views

Graphical data overview (summary) function in R

I'm sure I've come across a function like this in an R package before, but after extensive Googling I can't seem to find it anywhere. The function I'm thinking of produced a graphical summary for a ...
32
votes
5answers
3k views

Data “exploration” vs data “snooping”/“torturing”?

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem. On ...
29
votes
4answers
876 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
26
votes
2answers
2k views

How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
23
votes
5answers
4k views

Is exploratory data analysis important when doing purely predictive modeling?

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your ...
23
votes
5answers
8k views

What to learn after Casella & Berger?

I am a pure math grad student with little background in applied mathematics. Since last fall I have been taking classes on Casella & Berger's book, and I have finished hundreds (230+) of pages of ...
23
votes
6answers
694 views

Texas sharpshooter fallacy in exploratory data analysis

I was reading this article in Nature in which some fallacies are explained in the context of data analysis. I noticed that the Texas sharpshooter fallacy was particularly difficult to avoid: A ...
22
votes
5answers
4k views

How to keep exploratory analyses of large datasets in check?

When I start an exploratory analysis on a large data set (many samples, many variables), I often find myself with hundreds of derived variables, and tonnes of different plots, and no real way to keep ...
20
votes
8answers
5k views

Ideas for “lab notebook” software?

So this is an odd fit, though really I think it's an odd fit for any site, so I thought I'd try it here, among my data-crunching brethren. I came to epidemiology and biostatistics from biology, and ...
20
votes
1answer
9k views

What is the “horseshoe effect” and/or the “arch effect” in PCA / correspondence analysis?

There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
16
votes
2answers
6k views

How to do exploratory data analysis to choose appropriate machine learning algorithm

We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case ...
15
votes
5answers
3k views

A good way to show lots of data graphically

I'm working on a project that involves 14 variables and 345,000 observations for housing data (things like year built, square footage, price sold, county of residence, etc). I'm concerned with trying ...
15
votes
5answers
4k views

Is it better to do exploratory data analysis on the training dataset only?

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable. The question is: Should I do the EDA on my training dataset only? Or ...
14
votes
2answers
738 views

Are there differences in Bayesian and frequentist approaches to EDA?

Very simply put: Are there any differences in Bayesian and Frequentist approaches to Exploratory Data Analysis? I know of no inherent biases in EDA methods as a histogram is a histogram, a ...
14
votes
3answers
13k views

Flowcharts to help selecting the proper analysis technique and test

As someone who needs statistical knowledge but is not a formally trained statistician, I'd find it helpful to have a flowchart (or some kind of decision tree) to help me choose the correct approach to ...
13
votes
6answers
13k views

R package for identifying relationships between variables [closed]

Is there an R package that I can use to explore whether there exist relationships between variables? Typically when I am looking for patterns I look at correlations, and then a facet plot. Then I ...
13
votes
4answers
6k views

Best ways to aggregate and analyze data

Having just recently started teaching myself Machine Learning and Data Analysis I'm finding myself hitting a brick wall on the need for creating and querying large sets of data. I would like to take ...
12
votes
2answers
22k views

Difference between exploratory and confirmatory factor analysis in determining construct independence

Researchers often use two measures that have very similar items and argue that they measure different things (e.g., "I always worry when I am around cars"; "I am fearful of cars"). Lets call the ...
11
votes
2answers
18k views

If my histogram shows a bell-shaped curve, can I say my data is normally distributed?

I created a histogram for Respondent Age and managed to get a very nice bell-shaped curve, from which I concluded that the distribution is normal. Then I ran the normality test in SPSS, with n = 169. ...
11
votes
1answer
10k views

How to interpret notched box plots

While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor. The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
11
votes
2answers
2k views

What are some good exploratory analysis and diagnostic plots for count data? [duplicate]

Does anyone know of good reference material on exploratory analysis and diagnostic plots for count data?
10
votes
3answers
1k views

First quick glance at a dataset

Please pardon my ignorance, but... I keep finding myself in a situation, where I'm faced with a bunch of new data I managed to find. This data usually looks something like this: ...
10
votes
4answers
1k views

Tips and tricks to get started with statistical modeling?

I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work that focuses on Bayesian paradigms for learning and mining, which ...
10
votes
2answers
1k views

What's intended by “Let the data speak for itself”?

In reading the following paper, I came across the following statement: As mentioned, it is often presented without any reference to probabilistic models, in line with Benzecri [1973]’s idea to “let ...
9
votes
4answers
2k views

Why doesn't the fact that 1 median is lower than another median, mean that most in group 1 are less than most in group 2?

I believed that the boxplots below could be interpreted as "most men are faster than most women" (in this dataset), primarily because the median men's time was lower than the median women's time. But ...
9
votes
2answers
4k views

Are data transformations on non-normal data necessary for an exploratory factor analysis when using the principal axis factoring extraction method?

I am developing a questionnaire to measure four factors which constitute spirituality, and I would like to ask the following question: Are data transformations on non-normal data necessary for an ...
9
votes
3answers
307 views

Guidelines for discovering new knowledge in data

I plot something to make a point to myself or someone else. Usually, a question starts this process, and often the person asking hopes for a particular answer. How can I learn interesting things ...
8
votes
5answers
197 views

Preserving comments on graphs for exploratory data analysis

In performing exploratory data analysis, I will often print out the graphs and write out comments/annotations etc. Do people have suggestions for a better electronic methodology? I am especially ...
8
votes
1answer
5k views

OLS vs. logistic regression for exploratory analysis with a binary outcome

In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
7
votes
2answers
1k views

Is confirmatory vs exploratory statistics “induction vs deduction”?

This webpage says: Inferential Statistics - Deductive Approach Descriptive Statistics - Inductive Approach But I doubt it. If I understand correctly, Inferential Statistics is "given some ...
7
votes
1answer
494 views

Variables lack correlation, but have pattern

Below is the graph of two variables, X and Y, each representing count data. N=348. Note the scales of the axes: Y is very approximately lognormal, but X has no decent fit (including Poisson, negative ...
7
votes
1answer
612 views

Is exploratory data analysis (EDA) actually needed / useful

There are many guides prevalent on the internet about EDA and how everyone should do it and how useful it is however I rarely see it in practice and often times (in said tutorials) it sticks to very ...
7
votes
2answers
19k views

What are some good examples of exploratory data analysis today?

Are there some papers published which illustrate EDA used to tackle substantial data problems? I am particularly looking for actual (current) data examples, where plots have been made and statistics ...
6
votes
1answer
149 views

Is there a branch of statistics that tries to explain “why” the dataset has certain statistical properties?

Suppose I have a big dataset and I compute some statistical summary of it - e.g., the correlation of one dimension with another. I think a reasonable question to ask would be "what data points ...
6
votes
1answer
743 views

Data mining algorithm suggestion

I would like to use data mining to try to find a good workout schemes. The input dataset will contain the parameters of a set of workouts with dates and different performance and medical measures. The ...
6
votes
1answer
936 views

Is multiple comparisons a problem for exploratory analyses?

I have been struggling with the idea of multiple comparisons for while. Correcting for multiple comparisons is easy to understand and to correct for when conducting for example an ANOVA between ...
5
votes
2answers
65 views

Exploratory vs. Descriptive Statistical Analysis

Descriptive statistics definition is pretty clear to say that it summarizes data using statistical methods like mean, mode, median, and spread. However, I came across the term 'exploratory' today ...
5
votes
1answer
2k views

Determining probability distribution for datasets with missing values

As a part of my exploratory data analysis (EDA) prior to further analysis, I'm trying to determine a probability distribution of my pilot dataset's variables. A particular feature of this dataset is a ...
5
votes
1answer
193 views

When do I need a model?

Considering the scenarios of exploring data, predicting (in the range of predictors), extrapolating and explaining- for which would one need a model? When can one do without one? [Edit] By "model" I ...
5
votes
1answer
505 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
5
votes
0answers
110 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
4
votes
2answers
359 views

Is analysis of existing data always exploratory, or can it be used for hypothesis testing?

This is a question about the rhetoric of describing analysis done using a public data set or any other pre-existing data set. Here is the hypothetical situation: A researcher reports that they have a ...
4
votes
4answers
143 views

Does this present a linear relationship?

I have created the following graph which contains a great number of datapoints, I can see that there is a very linear scattering of the points, but am not sure since it displays a lot of datapoints ...
4
votes
2answers
165 views

Analysis of binary variables

I have a data set consisting of about a quarter-million objects, each of which may have any of 30 particular features. So I might have Object 1: feature 3, feature 7 Object 2: feature 3, feature 29, ...
4
votes
2answers
397 views

What to do with a small (27) medical dataset?

I'm working with a lot of data that was collected by obstetricians regarding the health of infants (birth weight, gestational age at delivery, mother's BMI), and I am trying to connect this data with ...
4
votes
2answers
2k views

Exploratory data analysis vs null hypothesis testing

Why would exploratory data analysis be important to undertake before null-hypothesis tests?
4
votes
1answer
136 views

Can complex sampling be ignored for exploratory patient-centered analyses?

Hello fellow StackExchange users, Preamble: I have been tasked with performing a cluster analysis (or possibly a latent class analysis, as I am pondering) to find non-overlapping groups of like ...

1
2 3 4 5