Questions tagged [exploratory-data-analysis]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Vocabulary in EDA

I've found myself straggling to find appropriate vocabulary to describe a scatterplot as part of exploratory data analysis. I've found this paper about the graph theory and although it's interesting I'...
Lefty's user avatar
  • 417
0 votes
0 answers
30 views

Valid forms of exploratory data analysis for time series that don't assume stationarity?

Lets say we are given a time series sample and want to try to create a model to forecast future values of said time series When trying to build a model to forecast time series data, many statistics ...
QMath's user avatar
  • 327
3 votes
1 answer
82 views

Exploratory analysis to find out characteristics of low scorers

I'm currently looking at three specific questions of a feedback survey and have been tasked with finding out the characteristics of the lowest scorers, to see if there are any patterns or common ...
sixfortyseven's user avatar
0 votes
0 answers
41 views

Outliers in EDA - With or without?

I'm trying to carry out my first EDA on a Student Performance dataset. The dataset has 395 samples and consists of 33 attributes. After drawing the boxplots and doing some tests I detected outliers in ...
Christina Kataki's user avatar
2 votes
0 answers
44 views

Variable Selection for Longitudinal Data with a Binary Outomce

I have a large longitudinal dataset (100,000 observations) with firm IDs and Years with about 1000 features (most numeric and ...
thatsroughbuddy's user avatar
1 vote
1 answer
48 views

EDA and Model Selection for Forecasting while avoiding Data Leakage

How to do EDA and model selection for time series forecasting without data leakage? Im assuming just checking for missing values is ok. But is graphing the entire time series considered data leakage? ...
pandashelp's user avatar
0 votes
0 answers
15 views

Calculate when a time series will reach s specific value

I have a time series at day level granularity: Date. Value 2023-10-01 78945 2023-10-02 78990 2023-10-03 79005 2023-10-04 78999 ... While there are some ...
Ricky's user avatar
  • 101
0 votes
0 answers
17 views

multivariate analysis for seasonal patterns among rivers

I sampled three different rivers over one year during the four seasons. In addition, in the autumn and winter seasons I sampled both during an event of high (drought period) and low water (flood ...
Rudy Benetti's user avatar
1 vote
0 answers
27 views

What kind of machine learning model could I use on this dataset?

I am a beginner to data science. I found this dataset that covers natural disaster incidents in Afghanistan from 2016 - present. Here are the 13 columns: REGION (South West, North, etc), PROV_CODE (...
Mas's user avatar
  • 11
0 votes
0 answers
28 views

How do i prioritize which features to use in my machine learning model before the feature engineering stage?

I am encountering a probably fairly common problem where I have too many features, lets say 500 possible features. I only want to pick the top 10-50 features that would be the most predictive of y, or ...
Katsu's user avatar
  • 861
0 votes
0 answers
18 views

Finding functional forms and state space in a state-space model

Let's say that we have a time series: $$\{y_i\}_{i=0}^{n-1}, y_i \in \mathcal{Y} \subseteq \mathbb{R}^m, m \in \mathbb{N}_1$$ that we would like to model with some sort of state-space model: $$x_i = f(...
QMath's user avatar
  • 327
0 votes
0 answers
89 views

exploratory data analysis on a large dataset with hundreds of variables

I'm working on longitudinal data with repeated measures for each subject and hundreds of variables. I would like to use linear mixed model to look at the mean response of each dependent variable at ...
Ed9012's user avatar
  • 161
0 votes
0 answers
29 views

Best way to perform a reliability growth analysis and answering some questions like "the reliability is getting better?"

everyone! I'm a reliability engineer and a project was given to me two weeks ago. I need to answer some questions about the reliability of a specific item in a train. There are 189 items, one for each ...
Caio Rodrigues's user avatar
1 vote
1 answer
24 views

How can I test whether the oblimin exploratory factor analysis function in r has produced the most simple structure?

would anyone know what command could be used to test whether the oblimin exploratory factor analysis function in r has produced the most simple structure? (as my understanding is oblimin does not ...
izzi3880's user avatar
0 votes
0 answers
31 views

Questions related to running an exploratory factor analysis on skewed data

I want to test overall skewness / normality in a large data set of ordinal data from survey questions. (I couldn’t use Shapiro wilk as I received an error saying the dataset is too large as it has ...
izzi3880's user avatar
2 votes
2 answers
173 views

Which set to use for confirmatory data analysis

I am exploring data analysis and want to perform confirmatory data analysis CDA on a dataset which I have splitted into ...
winter's user avatar
  • 123
0 votes
0 answers
15 views

Fixed effects, random effects and mixed effect modelling

I performed an incubation study to evaluate the effect of 4 fertilizer types on available P content in soil. I applied the fertilizers in 3 different soil types. I also had 4 replicates for each ...
Amrit's user avatar
  • 1
0 votes
0 answers
30 views

Is there a way to estimate dependence between an observed variable and an unobserved variable?

I'm attempting to do some open-ended exploratory analysis/model building on real-valued time series data. I am explicitly not assuming that all elements of my class of models are linear in the lagged ...
QMath's user avatar
  • 327
1 vote
1 answer
120 views

Period cutoff in Chow test

I am using the Chow test to analyze structural change in time series data potentially caused by a significant event. Since each data point represents a day, the data is broken into two chunks: Chunk 1:...
Jacob Z's user avatar
  • 11
0 votes
0 answers
63 views

Is there an issue doing EFA first, followed by a cluster analysis (with the newly found factors) with the same sample?

We want to compute a cluster analysis in a not well-researched field and therefore created about 60 questions. Those questions are supposedly aspects of factors, but we are not sure yet, to which ...
Tim's user avatar
  • 25
3 votes
1 answer
25 views

Methods for conducting hypothesis tests on frequencies/modes in a time-series?

I have time series data over an 8hour period of animals displaying behaviors. The behavior being displayed by a particular animal is notated as numbers 1-9, with a number corresponding to a given ...
Daniel K's user avatar
0 votes
0 answers
29 views

Do imbalanced datasets make removing poor target predictors easier?

I have an imbalanced fraud dataset with ~$1.4$% fraudulent samples across 50,000 rows with 600 columns. I'm performing a binary classification task on this dataset. I've performed an EDA; some columns ...
Connor's user avatar
  • 595
0 votes
1 answer
47 views

Exploratory Analysis with Small Sample

I'm using data (secondary analysis) to conduct an exploratory analysis. I have a sample size of $n = 67$, and the exploratory analysis includes two independent variables and four dependent variables. ...
suma suswaram's user avatar
0 votes
0 answers
27 views

How to interpet Exploratory Factor analysis (Network analysis)?

I applied exploratory factor analysis with network analysis to data from healthy and diseased patients. The analysis shows different clusters of parameters; some are similar in both groups, and some ...
Gigiux's user avatar
  • 211
4 votes
1 answer
77 views

Which statistical method to use to compare reinforced colour and speed of learning?

I'm working on my master's thesis where I'm trying to determine whether sea turtles are capable of discriminating between colour. One subcategory I'm looking into is seeing whether the speed they ...
Anamarija Ceglar's user avatar
8 votes
2 answers
140 views

Choosing $m$ value when using multiple imputation (MI)

Multiple imputation creates $m$ new imputed datasets by taking each missing value and replacing it by analyzing the $m$ imputed values (for example: using the mean). Is there a rule of thumb or a ...
Amit S's user avatar
  • 37
1 vote
0 answers
21 views

Exploratory factor analysis (EFA) for transcriptomic data

I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness. My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
Shehab's user avatar
  • 11
0 votes
0 answers
24 views

Why should exploratory analysis not be followed up with a confirmatory analysis in the same dataset? [duplicate]

On the Wikipedia page for data analysis, the following claim is made. "...one should not follow up an exploratory analysis with a confirmatory analysis in the same dataset. An exploratory ...
anna6931's user avatar
  • 111
0 votes
0 answers
71 views

OLS model assumptions for ML [closed]

hey i'm a beginner in data science, started working on the familiar "Boston" dataset ,and i tried to fit a multiple linear regression model to predict the "median house price " , ...
Yahya Sghiouri's user avatar
1 vote
1 answer
159 views

Standardized vs. unstandardized PCA

I performed PCA using two different R functions fviz() and ggbiplot() to improve the visualization of data. The sum of the first two PC is equal between the two methods, but there is a different value ...
bobo's user avatar
  • 23
1 vote
1 answer
18 views

Non uniform relationship between predictors and response variable

I aim to determine the relation between Y (a dichotomous variable, Y/N) and multiple Xs (Continuous variables). In the following charts, X axis represents the binned value of an X, while Y axis ...
Nibbles's user avatar
  • 25
1 vote
1 answer
78 views

Principal component vs. Exploratory Factor Analysis [duplicate]

Problem: Which method should I use to observe whether there is one common factor/component explaining six different measures (i.e., face and objects)? Study: In this study, we have four measures of ...
Bryan Leong's user avatar
1 vote
1 answer
75 views

What is the R function for EDA plots like this? [closed]

I am searching for the R function that generated this Exploratory Data Analysis plot. Anyone know what it is?
iluvmath's user avatar
  • 205
2 votes
2 answers
151 views

How to analyse a data set with more than 500 variables

I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview. My problem is not that ...
Concetta's user avatar
1 vote
0 answers
94 views

How to reconcile number of factors found with parallel factor analysis and network plotting in exploratory factor analysis?

I am trying to apply exploratory factor analysis and I got the following output when using parallel factor analysis (fa.parallel of the R package ...
Gigiux's user avatar
  • 211
2 votes
0 answers
37 views

Is there an improved canonical polyadic decomposition for symmetric tensors?

Let us suppose I want to find a CP decomposition of a $n$-mode tensor $\mathcal{A}$. Fortunately the tensor has the permutation symmetry $$\mathcal{A}[i_1, \cdots, i_n] = \mathcal{A}[\sigma (i_1), \...
Galen's user avatar
  • 8,125
1 vote
0 answers
49 views

On dealing with missing data for longitudinal exploratory data analysis

I have a longitudinal dataset. I wish to do some exploratory data analysis on it. However, it has the problem of missing observations/non-response. Q: How does one do exploratory data analysis on this ...
user45765's user avatar
  • 1,406
0 votes
1 answer
149 views

When and how to perform eda

I have some doubts on when to perform eda and the preprocessing the dataset should be put through. Should EDA be performed before or after balancing the dataset? Should one hot encoding and label ...
IDK's user avatar
  • 125
3 votes
1 answer
831 views

Are coordinates (e.g. Point(123, -123)) considered interval data?

I understand that latitude and longitude are interval data, but are coordinates (e.g. Point(123, -123)) also considered interval data? If so, how can the standard deviation, mean etc. be calculated?
Verum's user avatar
  • 31
2 votes
0 answers
214 views

How do I do an EDA on boolean/binary data without ML?

I'm looking to get some insight on what methods I can use to help identify key factors in my dataset that influence my dependent variable and how to prove that they are statically significant. There ...
yasalghuul's user avatar
-2 votes
1 answer
103 views

what does it means for p value = 8.780e-13 [duplicate]

I would like to ask if 8.780e-13 is greater than 0.05 for significance? I am not sure what is the calculation of this. Thanks a lot!
Tammy's user avatar
  • 9
0 votes
0 answers
34 views

Is it any meaningful taking the histogram of many histograms?

So I’m working on pixel-level assessment of chest x-ray images from the NIH Chest X-ray dataset. The diseases labels on the images are a set of z different categories (i.e. Pneumonia, effusion, etc…). ...
Walid Abdelal's user avatar
2 votes
1 answer
99 views

Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?

Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the ...
Fanfoué's user avatar
  • 621
2 votes
0 answers
84 views

Which transformation technique should we use for 2 bell shape distribution? [closed]

I'm doing exploratory data analysis and trying to transform the feature with log transformation, but I'm not sure which transformation to use for the two bell shaped distributions. Is it required to ...
Paul's user avatar
  • 31
0 votes
0 answers
131 views

Correlation between 2 columns after doing percentage change between consecutive rows for both columns

I am checking whether two columns in my pandas dataframe are correlated, but then wondering if it makes a difference if I do percentage change first (using pandas pct_change().mul(100) function) to ...
Tot's user avatar
  • 1
1 vote
1 answer
13 views

Do differences between ordinations prove a relationship between variables?

I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
Therocmorton's user avatar
9 votes
3 answers
361 views

Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
Fanfoué's user avatar
  • 621
2 votes
1 answer
198 views

In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?

I have generated a multiple-regression equation/model from the data. Parameter A in the data varies from $0.1$ - $10$. Parameter B in the data varies from $100$ - $1800$. Parameter C in the data ...
Pragnesh Rana's user avatar
1 vote
1 answer
85 views

Why do our devices fail?

I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
Ben's user avatar
  • 3,353
1 vote
1 answer
603 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
super_ask's user avatar
  • 225

1
2 3 4 5
7