EDA stands for "Exploratory data analysis". Developed by Tukey to contrast w/ *confirmatory* DA (the formal testing of hypotheses), EDA is typically concerned w/ describing data numerically & graphically to make the data easier to understand & to yield new insights.

learn more… | top users | synonyms (1)

9
votes
1answer
132 views

What is the “horseshoe effect” and/or the “arch effect” in PCA / correspondence analysis?

There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
0
votes
1answer
27 views

Introduction to Explorative Sequential Data Analysis

Short version I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc. Long version Currently I am facing data ...
3
votes
1answer
43 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
3
votes
1answer
91 views

How to interpret notched box plots

While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor. The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
0
votes
0answers
22 views

How can we observe interaction?

Are there graphical methods to determine whether there are some sort of interaction between two variables towards the response variable? Thanks!
0
votes
3answers
79 views

Exploratory data analysis for a dataset with continous and categorical variables

I have a data set which has DV and around 40 IVs. I want to select best variables out of the existing ones. I can use correlation, but it requires only numeric variables. I would like to see relation ...
2
votes
2answers
106 views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
1
vote
0answers
34 views

(LDA) Topic Modeling: eliminate Junktopics through normalization

Question: Is it reasonable to normalize topics to eliminate junk-topics and get a better distinction of document-relations? I used the MALLET-LDA Java-library to estimate a ParallelTopicModel with ...
1
vote
0answers
21 views

Topic Model inference for subselection

Question: Does the inference of a trained topic model (LDA) used on a subselection of a text corpus result in more accurate document-document-relations? I used the MALLET-LDA Java-library to estimate ...
1
vote
0answers
8 views

What is the discrete equivalent to the 4-plot?

Quoting NIST, when analyzing a continuous variable, one needs to validate four assumptions that typically underlie all measurement processes; namely, that the data at hand "behave like": random ...
0
votes
0answers
34 views

Reduction of species variables in vegetative analysis

Edited following helpful feedback. I have vegetation species data for a number of grassland habitat sites, and am preparing to begin Exploratory Data Analysis. Data was collected in 100 quadrats over ...
0
votes
0answers
49 views

What statistical technique could I use to see if gender effect exists in this scenario?

Suppose I give 6 short stories (of word length varying from 200 to 400),and ask N=100 students about which one (e.g., story #1, #2,..., #6) is their favorite story (see below table). Some data are ...
3
votes
2answers
860 views

If my histogram shows a bell-shaped curve, can I say my data is normally distributed?

I created a histogram for Respondent Age and managed to get a very nice bell-shaped curve, from which I concluded that the distribution is normal. Then I ran the normality test in SPSS, with n = 169. ...
2
votes
1answer
51 views

Quantitative Location Shifting Detection for Run-Sequence Plot

I am trying to complement a Run-Sequence Plot by some quantitative metric to validate that a dataset has a fixed location. Since the Run-Sequence Plot will be used in the early phases of Exploratory ...
2
votes
1answer
134 views

Minimum cumulative variance to extract in exploratory factor analysis to ensure a good fit

As a part of my exploratory factor analysis, I would like to report the cumulative variance % (eigenvalues). I wonder if there are guidelines on the minimum percentage in order to have a good model ...
3
votes
1answer
49 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
1
vote
1answer
180 views

Quick Exploratory Analysis of Categorical Data

Does anyone know of a tool (preferably free) that does quick analysis of exploratory data mainly categorical with date. Using R and Python I can create time series and histograms, perform tests such ...
35
votes
6answers
3k views

Is there any good reason to use PCA instead of EFA?

In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms. I therefore ...
2
votes
2answers
120 views

Which is correct way for regression line?

I have a set of data (some Frequencies per month, Var1 is the month): ...
6
votes
2answers
554 views

How to do exploratory data analysis to choose appropriate machine learning algorithm

We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case ...
0
votes
1answer
34 views

How to determine moving window size?

I am using moving window technique for data analysis... For example I compute the mean, the standard deviation and etc. for a given window. And I wonder if there's any good criterion to determine ...
4
votes
2answers
450 views

What are some good examples of exploratory data analysis today?

Are there some papers published which illustrate EDA used to tackle substantial data problems? I am particularly looking for actual (current) data examples, where plots have been made and statistics ...
1
vote
0answers
89 views

Can factor analysis improve the fit of a predictive regression model?

My company is working with a client who have built a logistic regression model to predict whether kids with psychiatric disorders will successfully complete a State intervention program (Yes or No). ...
3
votes
1answer
72 views

Is there a way to explain and generalise the decision made by random forest?

I saw a similar question was asked a few years ago, maybe there are updates on that. I would like to have a way to explain the decisions generated by random forest, possibly in a single tree. I ...
1
vote
3answers
105 views

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct? also

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct for correlation between variables? also is chi-square used only for hypotheses ...
4
votes
1answer
405 views

Determining probability distribution for datasets with missing values

As a part of my exploratory data analysis (EDA) prior to further analysis, I'm trying to determine a probability distribution of my pilot dataset's variables. A particular feature of this dataset is a ...
0
votes
2answers
37 views

Cross Vaidation in Factor Analysis

I am trying to conduct an EFA with a sample size of 150 respondents. I would also like to use cross-validation but my professor says that the sample is not big enough for that. Is that true?
1
vote
2answers
164 views

Is PCA appropriate for comparing subsets of panel data?

I have a large panel (5000+ subjects, 4 variables over 182 periods), and I've identified particular Granger-causal relationship in a large subset of those subjects (30% or so). I would like to somehow ...
2
votes
1answer
853 views

R scatterplot matrix with nonparametric density

I normally use MATLAB, or JMP but right now am working with R. I have ~150 dimensional data with a few hundred thousand rows. Some of the columns are non-informative, they only have one value. This ...
0
votes
0answers
55 views

Exploring properties of groups in R

I have a data frame with the following structure: ...
0
votes
1answer
48 views

factor analysis: can irrelevant factors be identified by tweaking FA options?

So I ran a factor analysis (principal components method) on a dataset, first using the correlation matrix and then using the covariance matrix, both with varimax rotations. The results of both factor ...
2
votes
0answers
48 views

What to do when exploratory factor analysis results are different for complete-cases and imputed data?

I have a hundred items that I'm performing EFA on, with around 370 complete cases. Using parallel analysis to determine the number of factors to extract, EFA gave 9 factors, all of which make ...
0
votes
2answers
221 views

factor analysis problem in structural equation model

I am using an Extended Technology Acceptance Model and have adopted an instrument from a research paper. After performing factor analysis in SPSS, most questions related to different constructs ...
0
votes
2answers
401 views

should i take this variable which consist only 1 question in factor analysis? (all other variables are based on multiple questions)

I'm conducting a study on internet shopping adoption. I'm using a structural model from a research paper. My supervisor has asked me to conduct a factor analysis using SPSS. I'm quite weak in stats ...
2
votes
0answers
95 views

The (statistically unsound) exploratory study (no particular hypothesis)

It seems like a proposed study gets penalised if it doesn't also predict the direction of an effect. For instance, if I want to see the effect of a certain medication on happiness ratings, but do not ...
2
votes
1answer
128 views

Exploratory Analysis - finding the most important factor

I have a dataset of 113 variables. In exploratory analysis the first thing I want to know is what are the most important factors on a single variable (revenue). I learned that naive Bayes would ...
15
votes
5answers
327 views

How to keep exploratory analyses of large datasets in check?

When I start an exploratory analysis on a large data set (many samples, many variables), I often find myself with hundreds of derived variables, and tonnes of different plots, and no real way to keep ...
2
votes
2answers
53 views

What are some differences between confirmatory analysis and exploratory analysis?

In confirmatory analysis do you basically just test hypotheses? Then in exploratory analysis you try to generate hypotheses? In general, I know that you can first do exploratory analysis to form ...
1
vote
1answer
104 views

Exploratory factor analysis

I am running exploratory factor analysis and have extracted 3 factors, with items clearly loading on each one. However, it appears to me that the items that have loaded onto the first factor could be ...
0
votes
0answers
140 views

Does confirmatory = inferential, and exploratory = descriptive analysis / statistics?

Reading this webpage, I wonder: Are confirmatory analysis / statistics and inferential analysis / statistics the same concept? Are exploratory analysis / statistics and descriptive analysis / ...
6
votes
2answers
428 views

Is confirmatory vs exploratory statistics “induction vs deduction”?

This webpage says: Inferential Statistics - Deductive Approach Descriptive Statistics - Inductive Approach But I doubt it. If I understand correctly, Inferential Statistics is "given ...
5
votes
1answer
97 views

Is there a branch of statistics that tries to explain “why” the dataset has certain statistical properties?

Suppose I have a big dataset and I compute some statistical summary of it - e.g., the correlation of one dimension with another. I think a reasonable question to ask would be "what data points ...
4
votes
2answers
412 views

Exploratory data analysis vs null hypothesis testing

Why would exploratory data analysis be important to undertake before null-hypothesis tests?
6
votes
1answer
320 views

Variables lack correlation, but have pattern

Below is the graph of two variables, X and Y, each representing count data. N=348. Note the scales of the axes: Y is very approximately lognormal, but X has no decent fit (including Poisson, ...
27
votes
5answers
980 views

Data “exploration” vs data “snooping”/“torturing”?

Many times I have come across informal warnings against "data snooping" (here's one amusing example), and I think I have an intuitive idea of roughly what that means, and why it may be a problem. On ...
2
votes
1answer
304 views

Combining Exploratory Factor Analysis with Random Forest for Binary Logistic Regression Feature Selection

For those of you familiar with Exploratory Factor Analysis (EFA) and Random Forest (RF), I have recently had an idea of combining these two methods to reduce the number of potential predictor ...
4
votes
2answers
157 views

Is analysis of existing data always exploratory, or can it be used for hypothesis testing?

This is a question about the rhetoric of describing analysis done using a public data set or any other pre-existing data set. Here is the hypothetical situation: A researcher reports that they have a ...
1
vote
2answers
125 views

How do I perform a multi-state decomposition with interaction effects?

I am trying to perform a decomposition with interaction effects. This paper provides a solution for n-factors where each factor has a binary state (see section 2). I have a problem with 2 factors, ...
1
vote
2answers
104 views

Implications of trying multiple tests for an analysis

Let's say I have a fixed set of data containing one variable. I'm interested in whether the variable shows a difference between two conditions. Now I perform a test and determine a p-value somewhere ...
0
votes
2answers
145 views

Unexpected factor structure in different samples

I have administered a questionnaire to respondents asking about customers' adoption of technology. I want to do an exploratory principal components factor analysis in Stata to form constructs from a ...