Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
325
questions
0
votes
1
answer
544
views
A variable that affects the dependent variable and is affected by another independent variable
I was analyzing happiness records data. One of the variables was trust in government and I hypothesized that this variable does not have an actual impact on the happiness score. I believe that this ...
CommunityBot
- 1
1
vote
1
answer
291
views
Remove dependency from a dependent variable
I am tasked with doing some exploratory data analysis on the Diamond dataset from the library(Ecdat) in R. The assignment states:...
CommunityBot
- 1
0
votes
1
answer
57
views
The interpretation of Chi-Square Independence Test
I'm working with UFO Sightings data set and my goal is to verify if the Shape of UFO ("circle", "cone"...) depends on the Duration of Sight (in seconds). To do that, I decided to perform a Chi-Square ...
- 78.1k
0
votes
0
answers
17
views
Vocabulary in EDA
I've found myself straggling to find appropriate vocabulary to describe a scatterplot as part of exploratory data analysis.
I've found this paper about the graph theory and although it's interesting I'...
- 479
1
vote
1
answer
455
views
What is the difference between EDA (Explorative Data Analysis) and Data Profiling?
I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same??
Can someone clarify? Thanks
CommunityBot
- 1
1
vote
0
answers
36
views
Valid forms of exploratory data analysis for time series that don't assume stationarity?
Lets say we are given a time series sample and want to try to create a model to forecast future values of said time series
When trying to build a model to forecast time series data, many statistics ...
- 111
3
votes
1
answer
86
views
Exploratory analysis to find out characteristics of low scorers
I'm currently looking at three specific questions of a feedback survey and have been tasked with finding out the characteristics of the lowest scorers, to see if there are any patterns or common ...
- 2,332
2
votes
0
answers
46
views
Variable Selection for Longitudinal Data with a Binary Outomce
I have a large longitudinal dataset (100,000 observations) with firm IDs and Years with about 1000 features (most numeric and ...
- 78.1k
0
votes
0
answers
44
views
Outliers in EDA - With or without?
I'm trying to carry out my first EDA on a Student Performance dataset. The dataset has 395 samples and consists of 33 attributes. After drawing the boxplots and doing some tests I detected outliers in ...
1
vote
1
answer
88
views
Principal component vs. Exploratory Factor Analysis [duplicate]
Problem: Which method should I use to observe whether there is one common factor/component explaining six different measures (i.e., face and objects)?
Study: In this study, we have four measures of ...
- 2,631
2
votes
1
answer
82
views
EDA and Model Selection for Forecasting while avoiding Data Leakage
How to do EDA and model selection for time series forecasting without data leakage?
Im assuming just checking for missing values is ok. But is graphing the entire time series considered data leakage?
...
- 124k
0
votes
0
answers
15
views
Calculate when a time series will reach s specific value
I have a time series at day level granularity:
Date. Value
2023-10-01 78945
2023-10-02 78990
2023-10-03 79005
2023-10-04 78999
...
While there are some ...
- 101
0
votes
0
answers
17
views
multivariate analysis for seasonal patterns among rivers
I sampled three different rivers over one year during the four seasons. In addition, in the autumn and winter seasons I sampled both during an event of high (drought period) and low water (flood ...
1
vote
0
answers
27
views
What kind of machine learning model could I use on this dataset?
I am a beginner to data science. I found this dataset that covers natural disaster incidents in Afghanistan from 2016 - present. Here are the 13 columns: REGION (South West, North, etc), PROV_CODE (...
- 11
0
votes
0
answers
18
views
Finding functional forms and state space in a state-space model
Let's say that we have a time series:
$$\{y_i\}_{i=0}^{n-1}, y_i \in \mathcal{Y} \subseteq \mathbb{R}^m, m \in \mathbb{N}_1$$
that we would like to model with some sort of state-space model:
$$x_i = f(...
- 389
0
votes
0
answers
30
views
How do i prioritize which features to use in my machine learning model before the feature engineering stage?
I am encountering a probably fairly common problem where I have too many features, lets say 500 possible features.
I only want to pick the top 10-50 features that would be the most predictive of y, or ...
- 911
0
votes
0
answers
96
views
exploratory data analysis on a large dataset with hundreds of variables
I'm working on longitudinal data with repeated measures for each subject and hundreds of variables. I would like to use linear mixed model to look at the mean response of each dependent variable at ...
- 78.1k
0
votes
0
answers
32
views
Best way to perform a reliability growth analysis and answering some questions like "the reliability is getting better?"
everyone!
I'm a reliability engineer and a project was given to me two weeks ago. I need to answer some questions about the reliability of a specific item in a train. There are 189 items, one for each ...
1
vote
1
answer
27
views
How can I test whether the oblimin exploratory factor analysis function in r has produced the most simple structure?
would anyone know what command could be used to test whether the oblimin exploratory factor analysis function in r has produced the most simple structure? (as my understanding is oblimin does not ...
- 120k
0
votes
0
answers
33
views
Questions related to running an exploratory factor analysis on skewed data
I want to test overall skewness / normality in a large data set of ordinal data from survey questions.
(I couldn’t use Shapiro wilk as I received an error saying the dataset is too large as it has ...
- 17.9k
0
votes
0
answers
109
views
What is the relationship between two variables that Chi-Squared Test for Independence is testing for? [duplicate]
I am trying to understand the application of Chi-Squared test for independence between predictor and response variables as it applies to feature selection in machine learning and exploratory data ...
- 78.1k
2
votes
2
answers
182
views
Which set to use for confirmatory data analysis
I am exploring data analysis and want to perform confirmatory data analysis CDA on a dataset which I have splitted into ...
- 120k
1
vote
0
answers
237
views
Why would the mean and standard deviation of the first and second derivatives of a signal (e.g. EDA) be useful?
When analysing a signal, e.g. EDA, I intuitively understand why one would want to determine the mean and standard deviation of the signal. The mean would tell us the average value of the EDA signal, ...
- 18.2k
0
votes
0
answers
16
views
Fixed effects, random effects and mixed effect modelling
I performed an incubation study to evaluate the effect of 4 fertilizer types on available P content in soil. I applied the fertilizers in 3 different soil types. I also had 4 replicates for each ...
- 1
0
votes
0
answers
31
views
Is there a way to estimate dependence between an observed variable and an unobserved variable?
I'm attempting to do some open-ended exploratory analysis/model building on real-valued time series data.
I am explicitly not assuming that all elements of my class of models are linear in the lagged ...
- 389
0
votes
0
answers
71
views
OLS model assumptions for ML [closed]
hey i'm a beginner in data science, started working on the familiar "Boston" dataset ,and i tried to fit a multiple linear regression model to predict the "median house price " , ...
- 18.2k
1
vote
1
answer
122
views
Period cutoff in Chow test
I am using the Chow test to analyze structural change in time series data potentially caused by a significant event. Since each data point represents a day, the data is broken into two chunks:
Chunk 1:...
- 722
0
votes
0
answers
71
views
Is there an issue doing EFA first, followed by a cluster analysis (with the newly found factors) with the same sample?
We want to compute a cluster analysis in a not well-researched field and therefore created about 60 questions. Those questions are supposedly aspects of factors, but we are not sure yet, to which ...
- 25
9
votes
3
answers
377
views
Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?
In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
- 125k
3
votes
1
answer
25
views
Methods for conducting hypothesis tests on frequencies/modes in a time-series?
I have time series data over an 8hour period of animals displaying behaviors.
The behavior being displayed by a particular animal is notated as numbers 1-9, with a number corresponding to a given ...
- 2,206
56
votes
9
answers
10k
views
Modern successor to Exploratory Data Analysis by Tukey?
I've been reading Tukey's book "Exploratory Data Analysis". Being written in 1977, the book emphasizes paper/pencil methods. Is there a more 'modern' successor which takes into account that we can ...
- 3,814
0
votes
0
answers
29
views
Do imbalanced datasets make removing poor target predictors easier?
I have an imbalanced fraud dataset with ~$1.4$% fraudulent samples across 50,000 rows with 600 columns. I'm performing a binary classification task on this dataset.
I've performed an EDA; some columns ...
- 625
29
votes
2
answers
3k
views
How to cope with exploratory data analysis and data dredging in small-sample studies?
Exploratory data analysis (EDA) often leads to explore other "tracks" that do not necessarily belong to the initial set of hypotheses. I face such a situation in the case of studies with a limited ...
- 1
0
votes
1
answer
50
views
Exploratory Analysis with Small Sample
I'm using data (secondary analysis) to conduct an exploratory analysis. I have a sample size of $n = 67$, and the exploratory analysis includes two independent variables and four dependent variables. ...
- 11
4
votes
1
answer
77
views
Which statistical method to use to compare reinforced colour and speed of learning?
I'm working on my master's thesis where I'm trying to determine whether sea turtles are capable of discriminating between colour. One subcategory I'm looking into is seeing whether the speed they ...
- 9,900
8
votes
2
answers
150
views
Choosing $m$ value when using multiple imputation (MI)
Multiple imputation creates $m$ new imputed datasets by taking each missing value and replacing it by analyzing the $m$ imputed values (for example: using the mean). Is there a rule of thumb or a ...
- 4,243
1
vote
0
answers
24
views
Exploratory factor analysis (EFA) for transcriptomic data
I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness.
My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
- 115
0
votes
0
answers
24
views
Why should exploratory analysis not be followed up with a confirmatory analysis in the same dataset? [duplicate]
On the Wikipedia page for data analysis, the following claim is made. "...one should not follow up an exploratory analysis with a confirmatory analysis in the same dataset. An exploratory ...
- 121
1
vote
1
answer
177
views
Standardized vs. unstandardized PCA
I performed PCA using two different R functions fviz() and ggbiplot() to improve the visualization of data. The sum of the first two PC is equal between the two methods, but there is a different value ...
- 23
1
vote
1
answer
18
views
Non uniform relationship between predictors and response variable
I aim to determine the relation between Y (a dichotomous variable, Y/N) and multiple Xs (Continuous variables). In the following charts, X axis represents the binned value of an X, while Y axis ...
- 124k
10
votes
4
answers
2k
views
Tips and tricks to get started with statistical modeling?
I work in the field of data mining and have had very little formal schooling in statistics. Lately I have been reading a lot of work that focuses on Bayesian paradigms for learning and mining, which ...
- 8,292
1
vote
1
answer
85
views
Why do our devices fail?
I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
- 134
1
vote
1
answer
84
views
What is the R function for EDA plots like this? [closed]
I am searching for the R function that generated this Exploratory Data Analysis plot. Anyone know what it is?
- 205
9
votes
3
answers
8k
views
Should exploratory data analysis include validation set?
I know that EDA should be performed on the training set but not on the test set.
But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
2
votes
2
answers
159
views
How to analyse a data set with more than 500 variables
I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview.
My problem is not that ...
- 4,317
1
vote
0
answers
94
views
How to reconcile number of factors found with parallel factor analysis and network plotting in exploratory factor analysis?
I am trying to apply exploratory factor analysis and I got the following output when using parallel factor analysis (fa.parallel of the R package ...
- 211
3
votes
2
answers
495
views
Pilot study EFA, 50 participants, can I use this sample to explore factors structure?
I’ve run a pilot study with only 50 participants and I was wondering what I could actually do with a small sample like that, how can I best use this data before collecting the final sample of 600 ...
- 17.9k
2
votes
0
answers
37
views
Is there an improved canonical polyadic decomposition for symmetric tensors?
Let us suppose I want to find a CP decomposition of a $n$-mode tensor $\mathcal{A}$. Fortunately the tensor has the permutation symmetry
$$\mathcal{A}[i_1, \cdots, i_n] = \mathcal{A}[\sigma (i_1), \...
- 8,502
1
vote
0
answers
50
views
On dealing with missing data for longitudinal exploratory data analysis
I have a longitudinal dataset. I wish to do some exploratory data analysis on it. However, it has the problem of missing observations/non-response.
Q: How does one do exploratory data analysis on this ...
- 1,416
0
votes
1
answer
172
views
When and how to perform eda
I have some doubts on when to perform eda and the preprocessing the dataset should be put through.
Should EDA be performed before or after balancing the
dataset?
Should one hot encoding and label ...
Tim♦
- 138k