Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
295
questions
-2
votes
0
answers
9
views
How to make a two-line mediator analysis in R? [closed]
How to make a two-line mediator analysis in R?
Please tell me briefly how to do this. What function should I use? I don't know much about data analysis, but I really need to figure out how to do this ...
- 1
1
vote
1
answer
8
views
Do differences between ordinations prove a relationship between variables?
I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
- 11
0
votes
0
answers
14
views
Multiple Comparisons Exploratory Analysis
I am junior analyst working with a group of physicians on a paper looking at healthcare utilization for people with a certain DX across between COVID and non-COVID years. We have several groups to ...
- 1
0
votes
0
answers
10
views
EDA with Data Modeling
After I read R for DataScience and ggplot2: elegant graphics for data analysis, I am learning how use modeling techniques to improve my EDA. I applied this on two notebooks (https://www.kaggle.com/...
3
votes
1
answer
70
views
Should stepwise regressions or overfitting also be avoided for exploratory (hypothesis generating) modelling?
In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
- 383
0
votes
0
answers
8
views
Cluster confidence scoring
I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info.
e....
2
votes
1
answer
22
views
In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?
I have generated a multiple-regression equation/model from the data.
Parameter A in the data varies from $0.1$ - $10$.
Parameter B in the data varies from $100$ - $1800$.
Parameter C in the data ...
0
votes
1
answer
62
views
Why do our devices fail?
I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
- 2,197
1
vote
1
answer
46
views
Class separability and Overlap
I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average.
This is a classification task and ...
- 215
2
votes
2
answers
92
views
Models for describing cluster change over time
Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
- 2,059
1
vote
1
answer
24
views
Does multicollinearity affect EDA?
I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
0
votes
0
answers
20
views
How to obtain the KMO index from a correlation matrix?
I have a correlation matrix that does the KMO index. In my research, I have only a correlation matrix of 7 variables. I need to do the KMO index but most softwares do not get input a correlation ...
- 1
4
votes
3
answers
1k
views
Which are outliers?
I am in the process of solving a Machine Learning challenge, and I want to do it the right way.
I did some exploratory data analysisand I wanted to check the distribution of the data.
As displayed in ...
- 51
0
votes
1
answer
21
views
Describing normal volume change with age: use a linear model and the standard deviation?
I would like to use a large dataset with tissue volume measures of 500 volunteers over a given ageframe to compare it to a small dataset with volunteers and patients (20-30 datapoints).
The tissue ...
- 169
0
votes
0
answers
32
views
Correlation between Ordinal Data and Binary Data
Hello this is for a personal project of mine regarding surveys. I wanted to compare two responses with each other with some sort of statistic.
-
A
B
1
600
196
2
276
466
3
36
304
4
20
150
5
22
44
...
- 1
2
votes
0
answers
65
views
Is there a reason to look at histogram of residuals of linear regression fit in order to determine if Poisson regression should be used?
Introduction
When reading lecture slides from a lecture I missed, one slide seemed to suggest that when presented with count data (the response being a count of something) one should try to fit a ...
- 171
4
votes
2
answers
535
views
How can I know If LASSO logistic regression model is good enough to be feature selection tool?
It is known that LASSO can be used for feature selection.
How can I know if the model is reliable for that purpose?
In general the model's accuracy, R squared and etc, don't bother me because I don't ...
- 67
0
votes
1
answer
27
views
What is the difference between EDA (Explorative Data Analysis) and Data Profiling?
I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same??
Can someone clarify? Thanks
- 1
1
vote
2
answers
54
views
Feature selection- feature that most of its values are equal. How can I know if to drop it?
If I have a feature that most of its values are the same, how can I know if to drop it or to keep it?
First of all, most of the model's features have low variance. Secondly, maybe the observations ...
- 67
1
vote
1
answer
18
views
What is good consolidated/normalized metric with funnel page views data?
I am currently working on a best way to represent product interest score based on page views.
Working on an ecommerce use case having homepage, search page, and product page interaction data available....
- 11
0
votes
0
answers
9
views
How do I handle underidentification in Bifactor ESEMs?
I hope you can help me to find some answers to my questions.
Following Morin, Arens, & Marsh (2016; references below), I’m trying to conduct a bifactor exploratory structural equation model (...
- 1
0
votes
0
answers
19
views
Identifying outliers in data collection with binary variables
I have a bunch of data collected on a set of engines. Most of my data is a flag if a particular feature works/doesn't or Issue_X exists/doesn't exist and then a final engine rating is given.
I want to ...
- 450
0
votes
1
answer
78
views
At first if I fail, ....should I keep trying
The final goal is to predict the effect of intervention(A) on outcome(Y) in the presence of some confounders (x). Before running the model to evaluate the effect of A on Y, I am doing some Exploratory ...
- 385
1
vote
0
answers
21
views
How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?
We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...
- 65
0
votes
0
answers
43
views
PCA: is linearity important?
I have a 6 dimensions dataset where I want to apply PCA to remove one dimension.
I did a small analysis to check for relationships in my data and concluded that there is very low linear correlation ...
- 16
0
votes
0
answers
22
views
PCA: does outlier detection make sense with low linear correlation? [duplicate]
I am experimenting PCA to detect outliers based on the reconstruction error.
What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and ...
- 16
0
votes
0
answers
16
views
Holdout and the Multiple Comparisons Problem
I apologize in advance if this is an extremely naive question.
I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
- 101
4
votes
0
answers
52
views
How can orthogonal factors explain more than 100% of the variance? [duplicate]
I'm using Stata's exploratory factor analysis command ("factor"). When I ask for a rotated solution (using the varimax orthogonal method, which is the default) it gives me (aside from the ...
- 1,579
0
votes
0
answers
43
views
Outliers for N=4
If you have 4 observations, can you have an outlier? Consider any value outside (Lower fourth-1.5(Fourth Spread), upper fourth+1.5(fourth spread)) as an outlier.
- 1
0
votes
0
answers
26
views
How to conduct exploratory analysis of repeated measures?
Exploratory analysis of repeated measures--without running statistical tests (told not to because study is underpowered and exploratory...?), what is the best way to visualize and/or quantify trends?
...
1
vote
1
answer
34
views
Confirmatory Factor Analysis identifying
I have been explained this technique in class. However, I did not understand some stuff.
The professor said that a model to be estimated needs to be at least identified. Identified meaning to be when ...
3
votes
1
answer
51
views
Need data preparation cheatsheet / guidelines / first principles to train team members! [closed]
Really ran out of ideas and hence such a basic question to the community -
Despite the repeated emphasis on ensuring data accuracy/validity, team members just do not spend enough time on it because ...
0
votes
0
answers
27
views
How to analyse this bimodal gamma distribution?
I have collected physiological data with multiple observations from 35 people, across four conditions. In planning the experiment, I had been hoping to perform inferential statistics comparing between ...
- 65
1
vote
0
answers
16
views
How should I identify the most important motivations to my respondents - factor means or variance explained by factors? (detected by EFA)
I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained 62% of the ...
- 23
1
vote
1
answer
36
views
Can I use construct/factor (revealed by EFA) mean scores as explanatory variables in a GLM (binomial family)?
I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained about 58% of the ...
- 23
0
votes
0
answers
18
views
Efficient metrics for measuring interaction and Building features
Long time user of tree models, rediscovered the benefits of Linear models recently. Now I want to transfer as much non linearity as possible into my features so to mimic as much as possible the ...
- 209
1
vote
0
answers
32
views
Many Preprocessing steps can cause data leakage, then how should we perform EDA?
For the past week, I have been constantly checking with people on this sub on how to avoid data leakage during preprocessing like feature selection and/or scaling etc here and here.
I understand most ...
- 533
0
votes
0
answers
15
views
Determining the range of input feature values
I have an application (numerical simulation) in hand, for which I am generating my own data.
Initially, I wanted to generate a small subset of data and perform initial ML model training. I was ...
0
votes
1
answer
63
views
On the misuse of two sample test: How to report difference in entire "populations"
I have the following understanding: Two (or more) sample tests are only required if conclusion should be drawn about a population based on observed samples (typically much smaller than the population)....
- 112
1
vote
0
answers
28
views
Differences in the factor - variable relationship in EFA and component - variable relationship in PCA?
When I read about exploratory factor analyses, I saw equations showing that each variable is a linear composite of different factors - with loadings correspond to the coefficient in front of each ...
- 69
0
votes
0
answers
52
views
Are there two types of Principal Component Analyses? [duplicate]
At first, I learned that PCA is a different type of analysis than factor analysis: PCA is only used for data reduction and factor analysis is used for detecting latent variables that underlie observed ...
- 69
0
votes
1
answer
50
views
Exploratory Factor Analysis for Big Data: Smaller Loading Threshold and Variance Accounted For?
I saw several pieces of literature mentioned that the larger the sample size, the lower the threshold to be set for loadings to be considered significant, and the lower the variance accounted for. I ...
- 69
0
votes
0
answers
21
views
Differences in Model Fit Evaluation Between Exploratory Factor Analysis and Confirmatory Factor Analysis
For the same dataset, I tried both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). For evaluating EFA model, the cumulative explanation of all latent factors adds up to ...
- 69
0
votes
0
answers
27
views
What data cleaning method is best to be adopted for price=$0?
Suppose I have an AirBnb dataset. In EDA, i noticed that some room price are 0 (as shown in the example below). Is removing them a best approach since its not possible that a room price is 0? Or ...
0
votes
0
answers
21
views
Methodology for exploratory analysis on dataset
This question is about finding the right methodology:
A population of cash machine is being analyzed. The variable we are measuring is the money income produced.
We are analyzing the variable over a ...
0
votes
2
answers
61
views
How to deal with missing values?
I have a question regarding exploratory data analysis, specifically, how to deal with missing values. Here is a toy dataset to describe my question:
In exploring each variable (a frequency table ...
- 33
1
vote
1
answer
21
views
EFA with "found" data
The textbook procedure for EFA is to start by designing the data collection, then you run the EFA to locate (we hope) the latent variables, therein.
Does that mean it is not ever valid to conduct and ...
- 831
0
votes
0
answers
39
views
Exploring relation between x and y [duplicate]
While there are numerous methods in exploring relation between x and y upon receiving a new dataset. Yet it seems I can't find any conclusive guide as of the sequence of analysis, e.g. do correlation ...
- 47
0
votes
0
answers
62
views
Standardization for using two similar data sets
I need your advice regarding standardization:
I have collected two similar datasets from the same population:
Error count and time for a group that was tested in person.
Error count and time for a ...
0
votes
0
answers
26
views
EFA and CFA with a lot of non-random "missing data"
This may be silly question, but I am only newly learning factor analysis. I am working on a project on social media. One of the problems is of the following form: We have circulated a form about ...
