EDA stands for "Exploratory data analysis". Developed by Tukey to contrast w/ *confirmatory* DA (the formal testing of hypotheses), EDA is typically concerned w/ describing data numerically & graphically to make the data easier to understand & to yield new insights.

learn more… | top users | synonyms (1)

0
votes
1answer
22 views

Statistical analysis for replication in very small experimental datasets

A colleague does replication on a quite coslty experiments. There are four different conditions, each one duplicated. The outcome with $4\times 2 = 8$ points is illustrated below: The analysis is ...
0
votes
1answer
27 views

The meaning of Classification Accuracy

I'm working on San Francisco Crime dataset, and only get about 20% classification accuracy. I used Random Forest Method. So how I can Interpret the result? I did EDA firstly, but how can I use EDA to ...
0
votes
0answers
10 views

Reading Centroid extracted factor matrix into SPSS for rotation, analysis [migrated]

I've been struggling with this for the last 4 days. I have concluded that I'm not smart enough to sort this out on my own, and I want to do this right. Desired Outcome: I want to instruct SPSS to ...
1
vote
0answers
11 views

Label reduction on dataset

This question related to this other one, for which I have devised a strategy and now want some feedback on it. My data consists of 434042 rows, each corresponding to an observation tagged with 1 of ...
0
votes
0answers
29 views

Which Method of Factor Extraction is Preferable With Communality Greater than 1.0?

I am trying to perform Exploratory Factor Analysis using SAS's proc factor with priors=smc, but am not sure which factor ...
0
votes
0answers
12 views

What kind of multivariate data is this judging from the QQ plot and Scatterplot Matrix?

This dataset is from R "cluster.datasets" package, named life.expectancy.1971. I would like to run Maximin Likelihood EFA(Exploratory Factor Analysis ) and therefore I checked the Multivariate ...
1
vote
1answer
28 views

Relationships of missing values for exploratory analysis

I have a survey with 30 questions on a seven item likert scale Not all of the questions were answered. I can use a heat map to visualize the missing answers but what i would like to know is if there ...
0
votes
0answers
26 views

What's the best strategy for testing 100+ Poisson distributions on seasonality and trend in Excel?

I have pooled data which can be divided into 4 categories, and those can be divided in 6-50 groups each. I tested the pooled data and the categories with time series plots. Now I'd like to test all of ...
1
vote
2answers
112 views

Any necessary EDA before logistic?

I wanted to know if we do EDA before logistic regression. Sure, I will look at the variables and their distributions, but is there anything specific to logistic?
0
votes
0answers
8 views

Finding feature values most specific for a subset

I have a set of items with lots of categorical features. I also have a small subset of this set. I want to find out what's different about this subset compared to general population (i.e. what feature ...
1
vote
2answers
101 views

How to assess seasonality effect influence on time series

Suppose I collected a time-series data (e.g. drug prescription on every month for 12 years). I have no reason to believe that my data is influenced by a seasonal factor (e.g. drug consumption is not ...
7
votes
3answers
166 views

Is it better to do exploratory data analysis on the training dataset only?

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable. The question is: Should I do the EDA on my training dataset only? Or ...
1
vote
1answer
27 views

Which method can I use to pinpoint features that separates a sub-group from a group

Let's say I have a group (G) of 1000 individuals. I have complete knowledge of 200 demographic properties of each of them, from categorical (favorite drink) to numerical (age). Let's say I invite ...
2
votes
2answers
97 views

Choosing predictors in regression analysis and multicollinearity

I would like to run a linear regression analysis and I'm uncertain about including predictors. I have three predictor variables available. One is based on a lot of previous research. Therefore I am ...
0
votes
0answers
10 views

Setting a Target or Benchmark based on Historic Process data

We have 15 Work processes for a departments. There is the historic data for the dept with all the information. Data defines the which process is done by whom, minutes of work, start date and finish ...
0
votes
1answer
30 views

Exploratory Data Analysis for financial data and validating the IID assumption

I am trying to model financial data and I need a little help. As I understand it, EDA is the first step, basically looking at the data. What tools (plots or tests) can be of help to me? Second, the ...
0
votes
0answers
15 views

Determining “n” in moving averages

I'm monitoring seven variables (flow into tank, flow out of tank, volume in tank, concentration of the flow in, concentration of the flow out, concentration in tank and weight inside tank). The weight ...
17
votes
5answers
322 views

Texas sharpshooter fallacy in exploratory data analysis

I was reading this article in Nature in which some fallacies are explained in the context of data analysis. I noticed that the Texas sharpshooter fallacy was particularly difficult to avoid: A ...
0
votes
0answers
23 views

Analysis of charts

Let $X$ a variable that represents the performance of a student in high school and a variable $Y$ representing the performance of this same student at the university. Consider the regression ...
0
votes
0answers
16 views

Best Method for Finding Cause of Change Between two Datasets

There are two data sets with exactly the same fields, taken from two different years. Two of the fields are numerical, thought of as numerator and denominator. All the remaining fields are ...
1
vote
0answers
61 views

Suggestions for Exploratory Data Analysis for high-dimensional regression data?

I have a moderately high-dim dataset (around 20 predictors) that I plan to analyse using Bayesian variable selection techniques. This will be a logistic regression analysis since the output is a ...
0
votes
0answers
17 views

Can factor analysis identify factor structure if only 5-15% of my sample give positive responses?

Looking to do EFA and CFA with different versions of the same data; 3 to 4 thousand participants, 50 symptoms with a 4-point Likert scale. I would expect most of the participants to endorse 5-10 out ...
1
vote
1answer
62 views

How to approach a regression problem? [closed]

I have to solve a regression problem involving 302 variables. How do we select appropriate models or ensemble of models to work well ? Does this decision of choosing models come from the ideas ...
1
vote
0answers
42 views

Rule of thumb for exploratory data

I frequently perform 2k factorial design of experiments and fit the data to a linear regression to control/optimize processes in a manufacturing environment. I have noticed recently that a large ...
0
votes
1answer
192 views

Python/R Exploratory Data Analysis for Classification [closed]

Are there preexisting functions in Python/R that create exploratory data analysis plots like the following:
0
votes
0answers
53 views

Conceptual question: how is a factor created in exploratory factor analysis?

As a conceptual question: in exploratory factor analysis, how is a factor created? I would like to know your simple answer to this simple question. Imagine, my academic field does not dependent on ...
10
votes
1answer
1k views

What is the “horseshoe effect” and/or the “arch effect” in PCA / correspondence analysis?

There are many techniques in ecological statistics for exploratory data analysis of multidimensional data. These are called 'ordination' techniques. Many are the same or closely related to common ...
0
votes
1answer
54 views

Introduction to Explorative Sequential Data Analysis

Short version I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc. Long version Currently I am facing data ...
3
votes
1answer
87 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
3
votes
1answer
1k views

How to interpret notched box plots

While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor. The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
0
votes
3answers
560 views

Exploratory data analysis for a dataset with continuous and categorical variables

I have a data set which has DV and around 40 IVs. I want to select best variables out of the existing ones. I can use correlation, but it requires only numeric variables. I would like to see relation ...
2
votes
2answers
343 views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
1
vote
0answers
69 views

(LDA) Topic Modeling: eliminate Junktopics through normalization

Question: Is it reasonable to normalize topics to eliminate junk-topics and get a better distinction of document-relations? I used the MALLET-LDA Java-library to estimate a ParallelTopicModel with ...
1
vote
0answers
24 views

Topic Model inference for subselection

Question: Does the inference of a trained topic model (LDA) used on a subselection of a text corpus result in more accurate document-document-relations? I used the MALLET-LDA Java-library to estimate ...
1
vote
0answers
10 views

What is the discrete equivalent to the 4-plot?

Quoting NIST, when analyzing a continuous variable, one needs to validate four assumptions that typically underlie all measurement processes; namely, that the data at hand "behave like": random ...
0
votes
0answers
39 views

Reduction of species variables in vegetative analysis

Edited following helpful feedback. I have vegetation species data for a number of grassland habitat sites, and am preparing to begin Exploratory Data Analysis. Data was collected in 100 quadrats over ...
0
votes
0answers
93 views

What statistical technique could I use to see if gender effect exists in this scenario?

Suppose I give 6 short stories (of word length varying from 200 to 400),and ask N=100 students about which one (e.g., story #1, #2,..., #6) is their favorite story (see below table). Some data are ...
8
votes
2answers
3k views

If my histogram shows a bell-shaped curve, can I say my data is normally distributed?

I created a histogram for Respondent Age and managed to get a very nice bell-shaped curve, from which I concluded that the distribution is normal. Then I ran the normality test in SPSS, with n = 169. ...
2
votes
1answer
68 views

Quantitative Location Shifting Detection for Run-Sequence Plot

I am trying to complement a Run-Sequence Plot by some quantitative metric to validate that a dataset has a fixed location. Since the Run-Sequence Plot will be used in the early phases of Exploratory ...
2
votes
1answer
327 views

Minimum cumulative variance to extract in exploratory factor analysis to ensure a good fit

As a part of my exploratory factor analysis, I would like to report the cumulative variance % (eigenvalues). I wonder if there are guidelines on the minimum percentage in order to have a good model ...
3
votes
1answer
71 views

Is there any rule of thumb to delete a variable in a large data set?

I'm working with a large set as a project for the business analytic course with $10^5$ observations and 170+ variables, some of which come with a missing value proportion of larger than 20%, even more ...
2
votes
1answer
431 views

Quick Exploratory Analysis of Categorical Data

Does anyone know of a tool (preferably free) that does quick analysis of exploratory data mainly categorical with date. Using R and Python I can create time series and histograms, perform tests such ...
40
votes
6answers
6k views

Is there any good reason to use PCA instead of EFA?

In some disciplines, PCA (principal component analysis) is systematically used without any justification, and PCA and EFA (exploratory factor analysis) are considered as synonyms. I therefore ...
2
votes
2answers
177 views

Which is correct way for regression line?

I have a set of data (some Frequencies per month, Var1 is the month): ...
7
votes
2answers
1k views

How to do exploratory data analysis to choose appropriate machine learning algorithm

We are studying machine learning via Machine Learning: A Probabilistic Perspective (Kevin Murphy). While the text explains the theoretical foundation of each algorithm, it rarely says in which case ...
0
votes
2answers
170 views

How to determine moving window size?

I am using moving window technique for data analysis... For example I compute the mean, the standard deviation and etc. for a given window. And I wonder if there's any good criterion to determine ...
4
votes
2answers
3k views

What are some good examples of exploratory data analysis today?

Are there some papers published which illustrate EDA used to tackle substantial data problems? I am particularly looking for actual (current) data examples, where plots have been made and statistics ...
1
vote
0answers
138 views

Can factor analysis improve the fit of a predictive regression model?

My company is working with a client who have built a logistic regression model to predict whether kids with psychiatric disorders will successfully complete a State intervention program (Yes or No). ...
3
votes
1answer
99 views

Is there a way to explain and generalise the decision made by random forest?

I saw a similar question was asked a few years ago, maybe there are updates on that. I would like to have a way to explain the decisions generated by random forest, possibly in a single tree. I ...
1
vote
3answers
108 views

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct? also

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct for correlation between variables? also is chi-square used only for hypotheses ...