Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
1
vote
3answers
120 views

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct? also

I do not have hypotheses for my research and I have got only categorical data. what kind of data analysis can i conduct for correlation between variables? also is chi-square used only for hypotheses ...
1
vote
1answer
27 views

PCA explained variance that increases linearly

I'm wondering what it says about the data when, instead of dropping off dramatically the variance explained by the number of components continues increasing in a reasonably linear manner. For ...
1
vote
3answers
3k views

Do data transformations before factor analysis need to be consistent across different variables?

(This question continues the previous one) I am creating a questionnaire, and I have identified 3 questions which are skewed (2 positively skewed & 1 negatively skewed). I successfully ...
1
vote
1answer
78 views

What's the meaning of “frequency represents area of bars” in histogram?

In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers: [1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10] The ...
1
vote
1answer
601 views

Should exploratory data analysis include validation set?

I know that EDA should be performed on the training set but not on the test set. But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
1
vote
1answer
3k views

How to analyze correlation on high dimensional data?

I have a dataset with over 100 features from where I want to know if there is a high correlation between some of those. I'm doing: corr = features_final.corr() ...
1
vote
2answers
2k views

should i take this variable which consist only 1 question in factor analysis? (all other variables are based on multiple questions)

I'm conducting a study on internet shopping adoption. I'm using a structural model from a research paper. My supervisor has asked me to conduct a factor analysis using SPSS. I'm quite weak in stats ...
1
vote
2answers
3k views

R package for visualizing and exploring large datasets

I recently read about a package, which I believe had a name like "tabstat" or "tablestats" which produced a really useful plot of univariate distributions of multiple variables for large datasets. I ...
1
vote
1answer
40 views

How to find the upper outlier threshold in a right skewed distribution?

Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers? One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
1
vote
1answer
54 views

PCA Interpreting strong negative and positive loadings

I ran a PCA on survey data asking whether respondents trust different levels of government, where they rank their level of trust from 0-7. Whilst I can interpret PC1 as a weighted average of ...
1
vote
1answer
19 views

Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
1
vote
1answer
842 views

EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
1
vote
1answer
172 views

Exploratory data analysis techniques to percentage variable

I'm running an analysis about a dataset with three variables: Income, number of residents per residence and percentage of income spent on rent (response variable). What descriptive statistics should I ...
1
vote
1answer
3k views

When to do the split training and test set

I want to split my sample into a training and a testing set in order to cross-validate my findings. However, I am not sure in which point in the process I have to do the split. Here is the order I ...
1
vote
1answer
204 views

Introduction to Explorative Sequential Data Analysis

Short version I am looking for introductory material to explorative data analysis of complex sequential data like website activity, hospital visits etc. Long version Currently I am facing data ...
1
vote
1answer
1k views

How to draw a side-by-side plot mentioned in “Graphical Display as an Aid to Analysis”

Emerson, J. D. (1991) Graphical Display as an Aid to Analysis, in Fundamentals of Exploratory Analysis of Variance (eds D. C. Hoaglin, F. Mosteller and J. W. Tukey), John Wiley & Sons, Inc., ...
1
vote
1answer
25 views

The use of Zero Value as Missing Values and Outliers

I am not sure what is the best title for this question. Sorry for that. Let's say I analyzing a dataset of prices of houses. And one of the columns (features) is ...
1
vote
1answer
464 views

Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
1
vote
1answer
234 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
1
vote
1answer
306 views

Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
1
vote
1answer
111 views

Statistical analysis for replication in very small experimental datasets

A colleague does replication on a quite coslty experiments. There are four different conditions, each one duplicated. The outcome with $4\times 2 = 8$ points is illustrated below: The analysis is ...
1
vote
1answer
137 views

Relationships of missing values for exploratory analysis

I have a survey with 30 questions on a seven item likert scale Not all of the questions were answered. I can use a heat map to visualize the missing answers but what i would like to know is if there ...
1
vote
1answer
324 views

Exploratory factor analysis

I am running exploratory factor analysis and have extracted 3 factors, with items clearly loading on each one. However, it appears to me that the items that have loaded onto the first factor could be ...
1
vote
2answers
236 views

How do I perform a multi-state decomposition with interaction effects?

I am trying to perform a decomposition with interaction effects. This paper provides a solution for n-factors where each factor has a binary state (see section 2). I have a problem with 2 factors, ...
1
vote
1answer
378 views

Exploratory data analysis for discrete data

I am using a probit and a logit model for obtaining the choice probabilities of some data. What kind of plots can be useful to conduct a exploratory data analysis for these data? Here is the ...
1
vote
0answers
11 views

How do you cope with the risk of false-positives in exploratory analysis?

Let's say that I'm running exploratory analysis on a dataset. For instance, let's say that the dataset consists of several features and two groups and I want to see which features are significantly ...
1
vote
0answers
10 views

Sources on simulations of the null hypothesis [closed]

I'm looking for two types of sources. Studies on any topic in any field that simulate fake data that are supposed to represent some null distribution to test the validity of some model.I'm just ...
1
vote
0answers
25 views

Finding Patterns In a 3D Plot

I am starting to use 3D graphs to visualize relationships in my data. The reason I am doing this is to try and maximize my predictor variables. Since I am working with financial data the relationships ...
1
vote
0answers
35 views

Approaches to analyzing data from forum

Working with self written example data that mimic a chat-forum with different forums (i.e. cars, politics, sports) and different threads (i.e. "why is porsche popular?", "thoughts on andrew yang?") in ...
1
vote
0answers
38 views

Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...
1
vote
0answers
56 views

MCA in FactoMineR: all variables are the same on Dim 1?

My data is ordinal (1, 2, 3, 4, 5, NA from a likert scale) and doesn't have any obvious pattern looking at the raw data. My code looks like this: library(FactoMineR) df <- data.frame(Var1 ...
1
vote
0answers
211 views

How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
1
vote
0answers
54 views

Data exploration methods for noisy time series

I have time series data from NanoPore sequencing (Attached is a illustrative figure and short explanation) which I'd ultimately like to use in order to find various methylation patterns in the input ...
1
vote
0answers
98 views

How to Use Exploratory Data Analysis To Criticize Pre-Existing Hypotheses

Obviously, exploratory data analysis is a useful tool that can be used to support the hypothesis generation process in the social sciences. Especially in the age of big data. But, my coursework never ...
1
vote
0answers
94 views

Data exploration to determine if GAMs are appropriate [duplicate]

How to decide, based on data exploration and data visualisation, between a GAM and a GLM. Linear Models (LM) assume that the relationship between the response and the predictor are linear. ...
1
vote
0answers
18 views

Exploratory Factor Analysis for Smoking cessation RCT?

So I'm planning a RCT where I'm evaluating a smoking cessation application intervention vs usual care among adolescent smokers. For my statistical analysis I'm looking at the point prevalence of ...
1
vote
0answers
113 views

Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
1
vote
0answers
281 views

How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
1
vote
1answer
37 views

Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty ...
1
vote
0answers
90 views

Histograms for severely skewed data [duplicate]

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,...
1
vote
0answers
780 views

Can EDA be used for asking a new question?

Data Analyst gets inspiration to ask a new question about the data, so the analyst begins the EDA on the same data set and goes through the 5 phases - Question; EDA; Modeling; Interpretation and ...
1
vote
0answers
26 views

Are there any intuitive/visual ways of detecting interactions between response variables?

Say I have a dataset with ~30 variables and I'm trying to make the best lm or glm. What's the best way to do exploratory data analysis to see if any of the variables would produce a good two-way ...
1
vote
0answers
22 views

Label reduction on dataset

This question related to this other one, for which I have devised a strategy and now want some feedback on it. My data consists of 434042 rows, each corresponding to an observation tagged with 1 of ...
1
vote
0answers
39 views

How to identify crucial features that lead to a strong sports performance

I have a dataset (500 samples) that contains information about sportsmen. It contains about 30 features that describe: age body composition like weight and size amount of daily training ...
1
vote
0answers
236 views

Suggestions for Exploratory Data Analysis for high-dimensional regression data?

I have a moderately high-dim dataset (around 20 predictors) that I plan to analyse using Bayesian variable selection techniques. This will be a logistic regression analysis since the output is a ...
1
vote
1answer
235 views

How to approach a regression problem? [closed]

I have to solve a regression problem involving 302 variables. How do we select appropriate models or ensemble of models to work well ? Does this decision of choosing models come from the ideas ...
1
vote
0answers
67 views

Rule of thumb for exploratory data

I frequently perform 2k factorial design of experiments and fit the data to a linear regression to control/optimize processes in a manufacturing environment. I have noticed recently that a large ...
1
vote
0answers
122 views

Conceptual question: how is a factor created in exploratory factor analysis?

As a conceptual question: in exploratory factor analysis, how is a factor created? I would like to know your simple answer to this simple question. Imagine, my academic field does not dependent on ...
1
vote
0answers
278 views

(LDA) Topic Modeling: eliminate Junktopics through normalization

Question: Is it reasonable to normalize topics to eliminate junk-topics and get a better distinction of document-relations? I used the MALLET-LDA Java-library to estimate a ParallelTopicModel with ...
1
vote
0answers
649 views

Statistical method to prove inverse relationship between class size and student achievement

A hypothesis states that there is an inverse relationship between class size and student achievement. With data records about class sizes and grades of its students at a certain school, we would like ...