# Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

215 questions
Filter by
Sorted by
Tagged with
120 views

### I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct? also

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct for correlation between variables? also is chi-square used only for hypotheses ...
27 views

### PCA explained variance that increases linearly

I'm wondering what it says about the data when, instead of dropping off dramatically the variance explained by the number of components continues increasing in a reasonably linear manner. For ...
3k views

### Do data transformations before factor analysis need to be consistent across different variables?

(This question continues the previous one) I am creating a questionnaire, and I have identified 3 questions which are skewed (2 positively skewed & 1 negatively skewed). I successfully ...
78 views

### What's the meaning of “frequency represents area of bars” in histogram?

In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers: [1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10] The ...
601 views

### Should exploratory data analysis include validation set?

I know that EDA should be performed on the training set but not on the test set. But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
3k views

### How to analyze correlation on high dimensional data?

I have a dataset with over 100 features from where I want to know if there is a high correlation between some of those. I'm doing: corr = features_final.corr() ...
2k views

### should i take this variable which consist only 1 question in factor analysis? (all other variables are based on multiple questions)

I'm conducting a study on internet shopping adoption. I'm using a structural model from a research paper. My supervisor has asked me to conduct a factor analysis using SPSS. I'm quite weak in stats ...
3k views

### R package for visualizing and exploring large datasets

I recently read about a package, which I believe had a name like "tabstat" or "tablestats" which produced a really useful plot of univariate distributions of multiple variables for large datasets. I ...
40 views

### How to find the upper outlier threshold in a right skewed distribution?

Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers? One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
54 views

I ran a PCA on survey data asking whether respondents trust different levels of government, where they rank their level of trust from 0-7. Whilst I can interpret PC1 as a weighted average of ...
19 views

### Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
842 views

### EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
172 views

### Exploratory data analysis techniques to percentage variable

I'm running an analysis about a dataset with three variables: Income, number of residents per residence and percentage of income spent on rent (response variable). What descriptive statistics should I ...
3k views

### When to do the split training and test set

I want to split my sample into a training and a testing set in order to cross-validate my findings. However, I am not sure in which point in the process I have to do the split. Here is the order I ...
204 views

### Introduction to Explorative Sequential Data Analysis

Short version I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc. Long version Currently I am facing data ...
1k views

### How to draw a side-by-side plot mentioned in “Graphical Display as an Aid to Analysis”

Emerson, J. D. (1991) Graphical Display as an Aid to Analysis, in Fundamentals of Exploratory Analysis of Variance (eds D. C. Hoaglin, F. Mosteller and J. W. Tukey), John Wiley & Sons, Inc., ...
25 views

### The use of Zero Value as Missing Values and Outliers

I am not sure what is the best title for this question. Sorry for that. Let's say I analyzing a dataset of prices of houses. And one of the columns (features) is ...
464 views

### Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
234 views

### appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
306 views

### Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
111 views

### Statistical analysis for replication in very small experimental datasets

A colleague does replication on a quite coslty experiments. There are four different conditions, each one duplicated. The outcome with $4\times 2 = 8$ points is illustrated below: The analysis is ...
137 views

### Relationships of missing values for exploratory analysis

I have a survey with 30 questions on a seven item likert scale Not all of the questions were answered. I can use a heat map to visualize the missing answers but what i would like to know is if there ...
324 views

### Exploratory factor analysis

I am running exploratory factor analysis and have extracted 3 factors, with items clearly loading on each one. However, it appears to me that the items that have loaded onto the first factor could be ...
236 views

### How do I perform a multi-state decomposition with interaction effects?

I am trying to perform a decomposition with interaction effects. This paper provides a solution for n-factors where each factor has a binary state (see section 2). I have a problem with 2 factors, ...
378 views

### Exploratory data analysis for discrete data

I am using a probit and a logit model for obtaining the choice probabilities of some data. What kind of plots can be useful to conduct a exploratory data analysis for these data? Here is the ...
11 views

### How do you cope with the risk of false-positives in exploratory analysis?

Let's say that I'm running exploratory analysis on a dataset. For instance, let's say that the dataset consists of several features and two groups and I want to see which features are significantly ...
10 views

### Sources on simulations of the null hypothesis [closed]

I'm looking for two types of sources. Studies on any topic in any field that simulate fake data that are supposed to represent some null distribution to test the validity of some model.I'm just ...
25 views

### Finding Patterns In a 3D Plot

I am starting to use 3D graphs to visualize relationships in my data. The reason I am doing this is to try and maximize my predictor variables. Since I am working with financial data the relationships ...
35 views

### Approaches to analyzing data from forum

Working with self written example data that mimic a chat-forum with different forums (i.e. cars, politics, sports) and different threads (i.e. "why is porsche popular?", "thoughts on andrew yang?") in ...
38 views

### Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...
56 views

### MCA in FactoMineR: all variables are the same on Dim 1?

My data is ordinal (1, 2, 3, 4, 5, NA from a likert scale) and doesn't have any obvious pattern looking at the raw data. My code looks like this: library(FactoMineR) df <- data.frame(Var1 ...
211 views

### How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
54 views

### Data exploration methods for noisy time series

I have time series data from NanoPore sequencing (Attached is a illustrative figure and short explanation) which I'd ultimately like to use in order to find various methylation patterns in the input ...
98 views

### How to Use Exploratory Data Analysis To Criticize Pre-Existing Hypotheses

Obviously, exploratory data analysis is a useful tool that can be used to support the hypothesis generation process in the social sciences. Especially in the age of big data. But, my coursework never ...
94 views

### Data exploration to determine if GAMs are appropriate [duplicate]

How to decide, based on data exploration and data visualisation, between a GAM and a GLM. Linear Models (LM) assume that the relationship between the response and the predictor are linear. ...
18 views

### Exploratory Factor Analysis for Smoking cessation RCT?

So I'm planning a RCT where I'm evaluating a smoking cessation application intervention vs usual care among adolescent smokers. For my statistical analysis I'm looking at the point prevalence of ...
113 views

### Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
281 views

### How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
37 views

### Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller ā say, only twenty or thirty ...
90 views

### Histograms for severely skewed data [duplicate]

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,...
780 views

### Can EDA be used for asking a new question?

Data Analyst gets inspiration to ask a new question about the data, so the analyst begins the EDA on the same data set and goes through the 5 phases - Question; EDA; Modeling; Interpretation and ...
26 views

### Are there any intuitive/visual ways of detecting interactions between response variables?

Say I have a dataset with ~30 variables and I'm trying to make the best lm or glm. What's the best way to do exploratory data analysis to see if any of the variables would produce a good two-way ...
22 views

### Label reduction on dataset

This question related to this other one, for which I have devised a strategy and now want some feedback on it. My data consists of 434042 rows, each corresponding to an observation tagged with 1 of ...
39 views

### How to identify crucial features that lead to a strong sports performance

I have a dataset (500 samples) that contains information about sportsmen. It contains about 30 features that describe: age body composition like weight and size amount of daily training ...
236 views

### Suggestions for Exploratory Data Analysis for high-dimensional regression data?

I have a moderately high-dim dataset (around 20 predictors) that I plan to analyse using Bayesian variable selection techniques. This will be a logistic regression analysis since the output is a ...
235 views

### How to approach a regression problem? [closed]

I have to solve a regression problem involving 302 variables. How do we select appropriate models or ensemble of models to work well ? Does this decision of choosing models come from the ideas ...
67 views

### Rule of thumb for exploratory data

I frequently perform 2k factorial design of experiments and fit the data to a linear regression to control/optimize processes in a manufacturing environment. I have noticed recently that a large ...
122 views

### Conceptual question: how is a factor created in exploratory factor analysis?

As a conceptual question: in exploratory factor analysis, how is a factor created? I would like to know your simple answer to this simple question. Imagine, my academic field does not dependent on ...