Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
47
questions
83
votes
6
answers
33k
views
Is there any good reason to use PCA instead of EFA? Also, can PCA be a substitute for factor analysis?
In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms.
I therefore ...
- 839
13
votes
2
answers
28k
views
If my histogram shows a bell-shaped curve, can I say my data is normally distributed?
I created a histogram for Respondent Age and managed to get a very nice bell-shaped curve, from which I concluded that the distribution is normal.
Then I ran the normality test in SPSS, with n = 169. ...
- 305
6
votes
3
answers
39k
views
What type of data are dates?
According to Yale:
Categorical variables represent types of data which may be divided into groups
(Lacey M, 1997)
To me, dates do not fit this definition. They are ordinal, as one date is ...
- 161
10
votes
3
answers
1k
views
First quick glance at a dataset
Please pardon my ignorance, but...
I keep finding myself in a situation, where I'm faced with a bunch of new data I managed to find. This data usually looks something like this:
...
- 203
24
votes
1
answer
16k
views
What is the "horseshoe effect" and/or the "arch effect" in PCA / correspondence analysis?
There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
9
votes
2
answers
5k
views
Are data transformations on non-normal data necessary for an exploratory factor analysis when using the principal axis factoring extraction method?
I am developing a questionnaire to measure four factors which constitute spirituality, and I would like to ask the following question:
Are data transformations on non-normal data necessary for an ...
- 401
29
votes
2
answers
3k
views
How to cope with exploratory data analysis and data dredging in small-sample studies?
Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
- 53.8k
26
votes
5
answers
12k
views
What to learn after Casella & Berger?
I am a pure math grad student with little background in applied mathematics. Since last fall I have been taking classes on Casella & Berger's book, and I have finished hundreds (230+) of pages of ...
56
votes
9
answers
10k
views
Modern successor to Exploratory Data Analysis by Tukey?
I've been reading Tukey's book "Exploratory Data Analysis". Being written in 1977, the book emphasizes paper/pencil methods. Is there a more 'modern' successor which takes into account that we can ...
30
votes
4
answers
957
views
Has the journal Science endorsed the Garden of Forking Pathes Analyses?
The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...
- 21.1k
35
votes
5
answers
3k
views
Data "exploration" vs data "snooping"/"torturing"?
Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem.
On ...
- 1,967
21
votes
5
answers
10k
views
Is it better to do exploratory data analysis on the training dataset only?
I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable.
The question is:
Should I do the EDA on my training dataset only? Or ...
- 313
10
votes
4
answers
2k
views
Tips and tricks to get started with statistical modeling?
I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work that focuses on Bayesian paradigms for learning and mining, which ...
9
votes
3
answers
8k
views
Should exploratory data analysis include validation set?
I know that EDA should be performed on the training set but not on the test set.
But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
- 91
8
votes
1
answer
6k
views
OLS vs. logistic regression for exploratory analysis with a binary outcome
In the idealized logistic model, we obtain an S-shaped curve linking each continuous IV to the DV. But in practice this S-shape infrequently occurs, making the logistic approach seem a little less ...
- 12.5k
5
votes
1
answer
2k
views
Determining probability distribution for datasets with missing values
As a part of my exploratory data analysis (EDA) prior to further analysis, I'm trying to determine a probability distribution of my pilot dataset's variables. A particular feature of this dataset is a ...
- 8,666
5
votes
0
answers
149
views
Censored logit transform for (ad hoc) exploratory data analysis
In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
- 13k
4
votes
3
answers
8k
views
Any necessary EDA before logistic?
I wanted to know if we do EDA before logistic regression.
Sure, I will look at the variables and their distributions, but is there anything specific to logistic?
- 659
3
votes
1
answer
646
views
Does the normal probability plot systematically underestimate the mean?
A normal probability plot is defined as a plot of $n$ pairs:
($[100(i-0.5)/n]$ th $z$ percentile, $i$th observation).
Theoretically the points should fall close to a straight line with slope $\...
- 2,818
89
votes
24
answers
11k
views
Rules of thumb for "modern" statistics
I like G van Belle's book on Statistical Rules of Thumb, and to a lesser extent Common Errors in Statistics (and How to Avoid Them) from Phillip I Good and James W. Hardin. They address common ...
39
votes
8
answers
34k
views
Graphical data overview (summary) function in R
I'm sure I've come across a function like this in an R package before, but after extensive Googling I can't seem to find it anywhere. The function I'm thinking of produced a graphical summary for a ...
- 2,117
20
votes
1
answer
17k
views
How to interpret notched box plots
While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor.
The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
- 535
19
votes
2
answers
8k
views
How to do exploratory data analysis to choose appropriate machine learning algorithm
We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case ...
- 4,600
15
votes
3
answers
15k
views
Flowcharts to help selecting the proper analysis technique and test
As someone who needs statistical knowledge but is not a formally trained statistician, I'd find it helpful to have a flowchart (or some kind of decision tree) to help me choose the correct approach to ...
13
votes
6
answers
16k
views
R package for identifying relationships between variables [closed]
Is there an R package that I can use to explore whether there exist relationships between variables?
Typically when I am looking for patterns I look at correlations, and then a facet plot. Then I ...
- 1,512
12
votes
2
answers
23k
views
Difference between exploratory and confirmatory factor analysis in determining construct independence
Researchers often use two measures that have very similar items and argue that they measure different things (e.g., "I always worry when I am around cars"; "I am fearful of cars"). Lets call the ...
- 5,064
11
votes
2
answers
4k
views
What are some good exploratory analysis and diagnostic plots for count data? [duplicate]
Does anyone know of good reference material on exploratory analysis and diagnostic plots for count data?
9
votes
2
answers
3k
views
Is confirmatory vs exploratory statistics "induction vs deduction"?
This webpage says:
Inferential Statistics - Deductive Approach
Descriptive Statistics - Inductive Approach
But I doubt it. If I understand correctly,
Inferential Statistics is "given some ...
- 19.5k
9
votes
3
answers
377
views
Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?
In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
- 631
5
votes
1
answer
210
views
When do I need a model?
Considering the scenarios of exploring data, predicting (in the range of predictors), extrapolating and explaining- for which would one need a model? When can one do without one?
[Edit] By "model" I ...
- 5,684
5
votes
2
answers
2k
views
How can I know If LASSO logistic regression model is good enough to be feature selection tool?
It is known that LASSO can be used for feature selection.
How can I know if the model is reliable for that purpose?
In general the model's accuracy, R squared and etc, don't bother me because I don't ...
- 57
4
votes
2
answers
1k
views
Is PCA appropriate for comparing subsets of panel data?
I have a large panel (5000+ subjects, 4 variables over 182 periods), and I've identified particular Granger-causal relationship in a large subset of those subjects (30% or so). I would like to somehow ...
- 12.6k
4
votes
2
answers
468
views
What to do with a small (27) medical dataset?
I'm working with a lot of data that was collected by obstetricians regarding the health of infants (birth weight, gestational age at delivery, mother's BMI), and I am trying to connect this data with ...
- 145
3
votes
1
answer
16k
views
Interpreting non-significant regression coefficients
Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
- 337
3
votes
2
answers
180
views
Websites to learn data analysis and visualization?
I took "business statistics" in college and survived, but I find data analysis and visualization fascinating. I have a decent grasps on the basics (probability, distributions, etc.).
What are some ...
2
votes
0
answers
45
views
Variable Selection for Longitudinal Data with a Binary Outomce
I have a large longitudinal dataset (100,000 observations) with firm IDs and Years with about 1000 features (most numeric and ...
2
votes
1
answer
99
views
Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?
Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the ...
- 631
2
votes
1
answer
2k
views
How to draw a side-by-side plot mentioned in "Graphical Display as an Aid to Analysis"
Emerson, J. D. (1991) Graphical Display as an Aid to Analysis, in Fundamentals of Exploratory Analysis of Variance (eds D. C. Hoaglin, F. Mosteller and J. W. Tukey), John Wiley & Sons, Inc., ...
- 1,231
1
vote
3
answers
4k
views
Do data transformations before factor analysis need to be consistent across different variables?
(This question continues the previous one)
I am creating a questionnaire, and I have identified 3 questions which are skewed (2 positively skewed & 1 negatively skewed). I successfully ...
- 401
1
vote
0
answers
64
views
Differences in the factor - variable relationship in EFA and component - variable relationship in PCA?
When I read about exploratory factor analyses, I saw equations showing that each variable is a linear composite of different factors - with loadings correspond to the coefficient in front of each ...
- 189
1
vote
0
answers
650
views
How do you code missing values if 0 is meaningful?
Per this deep learning book I am reading:
In general, with neural networks, it’s safe to input missing values as 0, with the condition that 0 isn’t already a meaningful value. The network will ...
- 3,253
1
vote
1
answer
2k
views
How to find the upper outlier threshold in a right skewed distribution?
Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers?
One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
- 85
1
vote
1
answer
255
views
Statistical analysis for replication in very small experimental datasets
A colleague does replication on a quite coslty experiments. There are four different conditions, each one duplicated. The outcome with $4\times 2 = 8$ points is illustrated below:
The analysis is ...
- 2,207
0
votes
2
answers
5k
views
Checking Normality of Numerical, and Categorical Data
I have come across 3 questions on the title subject.
Why is it necessary to do a normality test? To check if data is imbalanced or not?
Are these 4 methods of checking if the data follows normal ...
- 17
0
votes
0
answers
44
views
Checking Normality of Numerical and Categorical Data [duplicate]
I have come across 3 questions on the title subject.
Why is it necessary to do a normality test? To check if data is imbalanced or not?
Are these 4 methods of checking if the data follows normal ...
- 17
0
votes
0
answers
137
views
PCA: is linearity important?
I have a 6 dimensions dataset where I want to apply PCA to remove one dimension.
I did a small analysis to check for relationships in my data and concluded that there is very low linear correlation ...
- 16
0
votes
0
answers
245
views
Is jittering variables beneficaial before building a scatter plot?
I am given a dataset with features X and Y and need to learn to classify objects into 2 classes. The corresponding targets for the objects from the dataset are denoted as y:
Top left plot shows X vs ...
