Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
73
votes
6answers
23k views

Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?

In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms. I therefore ...
11
votes
2answers
18k views

If my histogram shows a bell-shaped curve, can I say my data is normally distributed?

I created a histogram for Respondent Age and managed to get a very nice bell-shaped curve, from which I concluded that the distribution is normal. Then I ran the normality test in SPSS, with n = 169. ...
1
vote
2answers
4k views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
10
votes
3answers
1k views

First quick glance at a dataset

Please pardon my ignorance, but... I keep finding myself in a situation, where I'm faced with a bunch of new data I managed to find. This data usually looks something like this: ...
9
votes
2answers
4k views

Are data transformations on non-normal data necessary for an exploratory factor analysis when using the principal axis factoring extraction method?

I am developing a questionnaire to measure four factors which constitute spirituality, and I would like to ask the following question: Are data transformations on non-normal data necessary for an ...
23
votes
5answers
8k views

What to learn after Casella & Berger?

I am a pure math grad student with little background in applied mathematics. Since last fall I have been taking classes on Casella & Berger's book, and I have finished hundreds (230+) of pages of ...
53
votes
8answers
9k views

Modern successor to Exploratory Data Analysis by Tukey?

I've been reading Tukey's book "Exploratory Data Analysis". Being written in 1977, the book emphasizes paper/pencil methods. Is there a more 'modern' successor which takes into account that we can ...
26
votes
2answers
2k views

How to cope with exploratory data analysis and data dredging in small-sample studies?

Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
29
votes
4answers
876 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
20
votes
1answer
9k views

What is the “horseshoe effect” and/or the “arch effect” in PCA / correspondence analysis?

There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
10
votes
4answers
1k views

Tips and tricks to get started with statistical modeling?

I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work that focuses on Bayesian paradigms for learning and mining, which ...
8
votes
1answer
5k views

OLS vs. logistic regression for exploratory analysis with a binary outcome

In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
5
votes
1answer
2k views

Determining probability distribution for datasets with missing values

As a part of my exploratory data analysis (EDA) prior to further analysis, I'm trying to determine a probability distribution of my pilot dataset's variables. A particular feature of this dataset is a ...
5
votes
0answers
110 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
3
votes
1answer
409 views

Does the normal probability plot systematically underestimate the mean?

A normal probability plot is defined as a plot of $n$ pairs: ($[100(i-0.5)/n]$ th $z$ percentile, $i$th observation). Theoretically the points should fall close to a straight line with slope $\...
86
votes
24answers
9k views

Rules of thumb for “modern” statistics

I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common ...
39
votes
8answers
30k views

Graphical data overview (summary) function in R

I'm sure I've come across a function like this in an R package before, but after extensive Googling I can't seem to find it anywhere. The function I'm thinking of produced a graphical summary for a ...
16
votes
2answers
6k views

How to do exploratory data analysis to choose appropriate machine learning algorithm

We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case ...
32
votes
5answers
3k views

Data “exploration” vs data “snooping”/“torturing”?

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem. On ...
13
votes
6answers
13k views

R package for identifying relationships between variables [closed]

Is there an R package that I can use to explore whether there exist relationships between variables? Typically when I am looking for patterns I look at correlations, and then a facet plot. Then I ...
14
votes
3answers
13k views

Flowcharts to help selecting the proper analysis technique and test

As someone who needs statistical knowledge but is not a formally trained statistician, I'd find it helpful to have a flowchart (or some kind of decision tree) to help me choose the correct approach to ...
3
votes
3answers
5k views

Any necessary EDA before logistic?

I wanted to know if we do EDA before logistic regression. Sure, I will look at the variables and their distributions, but is there anything specific to logistic?
11
votes
1answer
10k views

How to interpret notched box plots

While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor. The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
12
votes
2answers
22k views

Difference between exploratory and confirmatory factor analysis in determining construct independence

Researchers often use two measures that have very similar items and argue that they measure different things (e.g., "I always worry when I am around cars"; "I am fearful of cars"). Lets call the ...
3
votes
2answers
140 views

Websites to learn data analysis and visualization?

I took "business statistics" in college and survived, but I find data analysis and visualization fascinating. I have a decent grasps on the basics (probability, distributions, etc.). What are some ...
1
vote
1answer
1k views

How to draw a side-by-side plot mentioned in “Graphical Display as an Aid to Analysis”

Emerson, J. D. (1991) Graphical Display as an Aid to Analysis, in Fundamentals of Exploratory Analysis of Variance (eds D. C. Hoaglin, F. Mosteller and J. W. Tukey), John Wiley & Sons, Inc., ...
15
votes
5answers
5k views

Is it better to do exploratory data analysis on the training dataset only?

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable. The question is: Should I do the EDA on my training dataset only? Or ...
7
votes
2answers
1k views

Is confirmatory vs exploratory statistics “induction vs deduction”?

This webpage says: Inferential Statistics - Deductive Approach Descriptive Statistics - Inductive Approach But I doubt it. If I understand correctly, Inferential Statistics is "given some ...
5
votes
1answer
193 views

When do I need a model?

Considering the scenarios of exploring data, predicting (in the range of predictors), extrapolating and explaining- for which would one need a model? When can one do without one? [Edit] By "model" I ...
3
votes
2answers
747 views

Is PCA appropriate for comparing subsets of panel data?

I have a large panel (5000+ subjects, 4 variables over 182 periods), and I've identified particular Granger-causal relationship in a large subset of those subjects (30% or so). I would like to somehow ...
1
vote
1answer
109 views

Statistical analysis for replication in very small experimental datasets

A colleague does replication on a quite coslty experiments. There are four different conditions, each one duplicated. The outcome with $4\times 2 = 8$ points is illustrated below: The analysis is ...
1
vote
3answers
3k views

Do data transformations before factor analysis need to be consistent across different variables?

(This question continues the previous one) I am creating a questionnaire, and I have identified 3 questions which are skewed (2 positively skewed & 1 negatively skewed). I successfully ...
0
votes
0answers
47 views

Should items with low intercorrelation be removed before EFA to avoid many items with low communalities?

I am attempting Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. Background: 221 participants indicated the extent of their agreement on 61 statements, ...
0
votes
0answers
28 views

Checking Normality of Numerical and Categorical Data [duplicate]

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
2answers
78 views

Checking Normality of Numerical, and Categorical Data

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...