Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
315
questions
0
votes
0
answers
10
views
Top 5 Things you look for in sns.pairplot() for a regression problem? [closed]
Where can I find a laundry list of "What I Should Be Looking For" when reading a sns.pairplot()?
There are a ton of resources that show you how cool a ...
- 422
0
votes
0
answers
14
views
Interpretation of Exploratory Factor Analysis
I am conducting research in cognition, where we are reducing a large dataset. An Exploratory Factor Analaysis with a 3-factor solution (based on a scree plot) reveals factors that appear to correspond ...
- 1
1
vote
0
answers
15
views
Exploratory factor analysis (EFA) for transcriptomic data
I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness.
My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
- 11
0
votes
0
answers
22
views
Why should exploratory analysis not be followed up with a confirmatory analysis in the same dataset? [duplicate]
On the Wikipedia page for data analysis, the following claim is made. "...one should not follow up an exploratory analysis with a confirmatory analysis in the same dataset. An exploratory ...
- 67
0
votes
0
answers
6
views
Help with interpreting unusual EFA result
I was hoping to receive some advice on how to properly interpret the results of my principal axis factoring analysis. The results seem to load well onto one factor except for one item which has a ...
0
votes
0
answers
36
views
OLS model assumptions for ML
hey i'm a beginner in data science, started working on the familiar "Boston" dataset ,and i tried to fit a multiple linear regression model to predict the "median house price " , ...
1
vote
1
answer
33
views
Standardized vs. unstandardized PCA
I performed PCA using two different R functions fviz() and ggbiplot() to improve the visualization of data. The sum of the first two PC is equal between the two methods, but there is a different value ...
- 23
0
votes
0
answers
12
views
What is the best way to identify demographic characteristics from within pooled survey data?
I have conducted a survey looking at individual exercise habits and their preferences regarding technology when measuring exercise and providing feedback. I am in the process of analysing the data ...
- 1
1
vote
1
answer
13
views
Non uniform relationship between predictors and response variable
I aim to determine the relation between Y (a dichotomous variable, Y/N) and multiple Xs (Continuous variables). In the following charts, X axis represents the binned value of an X, while Y axis ...
- 25
0
votes
0
answers
25
views
Scree plot depicting the proportion of variance?
I'm reading the 2nd edition of the book "An Introduction to Statistical Learning" from Springer. In page 507, they have a scree plot with y-axis is the proportion of variance. And the ...
- 1
0
votes
0
answers
20
views
Should I remove/cap "Extreme Values" from Variables and at which percentile should capping be done?
In my dataset, I have a variable corresponding to the monthly average playtime hours of the players. The dataset has 200K records. I estimated various percentiles of the variable - from 0 to 1 for ...
- 143
0
votes
0
answers
33
views
PCA: Does the sum of all the features (not the components) is always linearly correlated to F1?
I'm trying to figure out if we always have:
$$pearson\_corr(F_1, \sum_{k=1}^{n}X_i) = 1$$
with:
$F_1$ the principal component,
$X_i$ the numerical features we used for the PCA,
$n$ the number of ...
1
vote
1
answer
30
views
Principal component vs. Exploratory Factor Analysis [duplicate]
Problem: Which method should I use to observe whether there is one common factor/component explaining six different measures (i.e., face and objects)?
Study: In this study, we have four measures of ...
- 13
0
votes
0
answers
13
views
The actual unit of sample in the data set is higher than the planned unit of sample
the population is all the fish from a lake, sampling frame is all the predator fish, the cased study mentioned that up to 4 fish from the hierarchical order of preferred predator species were obtained'...
1
vote
1
answer
41
views
What is the R function for EDA plots like this? [closed]
I am searching for the R function that generated this Exploratory Data Analysis plot. Anyone know what it is?
- 205
2
votes
2
answers
73
views
How to analyse a data set with more than 500 variables
I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview.
My problem is not that ...
- 59
1
vote
0
answers
25
views
How to reconcile number of factors found with parallel factor analysis and network plotting in exploratory factor analysis?
I am trying to apply exploratory factor analysis and I got the following output when using parallel factor analysis (fa.parallel of the R package ...
- 149
1
vote
0
answers
24
views
Is there an improved canonical polyadic decomposition for symmetric tensors?
Let us suppose I want to find a CP decomposition of a $n$-mode tensor $\mathcal{A}$. Fortunately the tensor has the permutation symmetry
$$\mathcal{A}[i_1, \cdots, i_n] = \mathcal{A}[\sigma (i_1), \...
- 5,626
1
vote
0
answers
20
views
On dealing with missing data for longitudinal exploratory data analysis
I have a longitudinal dataset. I wish to do some exploratory data analysis on it. However, it has the problem of missing observations/non-response.
Q: How does one do exploratory data analysis on this ...
- 1,315
0
votes
0
answers
15
views
Incremental value of a categorical variable
The question is to calculate the incremental value of running an advertisement for a client.
I have provided a sample of my dataset.
...
0
votes
1
answer
28
views
When and how to perform eda
I have some doubts on when to perform eda and the preprocessing the dataset should be put through.
Should EDA be performed before or after balancing the
dataset?
Should one hot encoding and label ...
- 105
0
votes
0
answers
42
views
How to handle multiple time series data with covariates in R
I have time series data on traffic flow in different locations. I want to look at how weather (time series) and location properties (e.g., accessibility calculated by road network and keep unchanged ...
3
votes
1
answer
321
views
Are coordinates (e.g. Point(123, -123)) considered interval data?
I understand that latitude and longitude are interval data, but are coordinates (e.g. Point(123, -123)) also considered interval data? If so, how can the standard deviation, mean etc. be calculated?
- 31
0
votes
0
answers
52
views
How to do an MCA analysis on a binary dataset?
I'm looking to do an EDA with some binary data as seen below and I was recommended to use MCA. However, I'm not sure about how to go about doing this in a meaningful way. I'm looking to find key ...
- 21
2
votes
0
answers
122
views
How do I do an EDA on boolean/binary data without ML?
I'm looking to get some insight on what methods I can use to help identify key factors in my dataset that influence my dependent variable and how to prove that they are statically significant. There ...
- 21
0
votes
0
answers
28
views
Can exploratory data analysis data overlap with that of confirmatory data analysis?
Sometimes data is expensive to obtain. I will do a random draw of small subset of data to do exploratory data analysis.
Q: Can I do confirmatory data analysis on full data if data is limited? Or can ...
- 1,315
-2
votes
1
answer
61
views
what does it means for p value = 8.780e-13 [duplicate]
I would like to ask if 8.780e-13 is greater than 0.05 for significance? I am not sure what is the calculation of this. Thanks a lot!
- 9
0
votes
0
answers
24
views
Is it any meaningful taking the histogram of many histograms?
So I’m working on pixel-level assessment of chest x-ray images from the NIH Chest X-ray dataset. The diseases labels on the images are a set of z different categories (i.e. Pneumonia, effusion, etc…). ...
1
vote
1
answer
61
views
Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?
Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the ...
- 531
2
votes
0
answers
44
views
Which transformation technique should we use for 2 bell shape distribution?
I'm doing exploratory data analysis and trying to transform the feature with log transformation, but I'm not sure which transformation to use for the two bell shaped distributions. Is it required to ...
- 31
0
votes
0
answers
94
views
Correlation between 2 columns after doing percentage change between consecutive rows for both columns
I am checking whether two columns in my pandas dataframe are correlated, but then wondering if it makes a difference if I do percentage change first (using pandas pct_change().mul(100) function) to ...
- 1
1
vote
1
answer
12
views
Do differences between ordinations prove a relationship between variables?
I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
- 11
0
votes
0
answers
26
views
Multiple Comparisons Exploratory Analysis
I am junior analyst working with a group of physicians on a paper looking at healthcare utilization for people with a certain DX across between COVID and non-COVID years. We have several groups to ...
- 1
0
votes
0
answers
21
views
EDA with Data Modeling
After I read R for DataScience and ggplot2: elegant graphics for data analysis, I am learning how use modeling techniques to improve my EDA. I applied this on two notebooks (https://www.kaggle.com/...
- 33
6
votes
3
answers
162
views
Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?
In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
- 531
0
votes
0
answers
49
views
Cluster confidence scoring
I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info.
e....
2
votes
1
answer
49
views
In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?
I have generated a multiple-regression equation/model from the data.
Parameter A in the data varies from $0.1$ - $10$.
Parameter B in the data varies from $100$ - $1800$.
Parameter C in the data ...
0
votes
1
answer
78
views
Why do our devices fail?
I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
- 2,849
1
vote
1
answer
357
views
Class separability and Overlap
I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average.
This is a classification task and ...
- 215
2
votes
2
answers
236
views
Models for describing cluster change over time
Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
- 2,164
2
votes
1
answer
38
views
Does multicollinearity affect EDA?
I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
- 143
4
votes
3
answers
1k
views
Which are outliers?
I am in the process of solving a Machine Learning challenge, and I want to do it the right way.
I did some exploratory data analysisand I wanted to check the distribution of the data.
As displayed in ...
- 51
0
votes
1
answer
25
views
Describing normal volume change with age: use a linear model and the standard deviation?
I would like to use a large dataset with tissue volume measures of 500 volunteers over a given ageframe to compare it to a small dataset with volunteers and patients (20-30 datapoints).
The tissue ...
- 309
2
votes
0
answers
140
views
Is there a reason to look at histogram of residuals of linear regression fit in order to determine if Poisson regression should be used?
Introduction
When reading lecture slides from a lecture I missed, one slide seemed to suggest that when presented with count data (the response being a count of something) one should try to fit a ...
- 171
5
votes
2
answers
1k
views
How can I know If LASSO logistic regression model is good enough to be feature selection tool?
It is known that LASSO can be used for feature selection.
How can I know if the model is reliable for that purpose?
In general the model's accuracy, R squared and etc, don't bother me because I don't ...
- 121
0
votes
1
answer
155
views
What is the difference between EDA (Explorative Data Analysis) and Data Profiling?
I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same??
Can someone clarify? Thanks
- 1
1
vote
2
answers
148
views
Feature selection- feature that most of its values are equal. How can I know if to drop it?
If I have a feature that most of its values are the same, how can I know if to drop it or to keep it?
First of all, most of the model's features have low variance. Secondly, maybe the observations ...
- 121
1
vote
1
answer
19
views
What is good consolidated/normalized metric with funnel page views data?
I am currently working on a best way to represent product interest score based on page views.
Working on an ecommerce use case having homepage, search page, and product page interaction data available....
- 11
0
votes
1
answer
82
views
At first if I fail, ....should I keep trying
The final goal is to predict the effect of intervention(A) on outcome(Y) in the presence of some confounders (x). Before running the model to evaluate the effect of A on Y, I am doing some Exploratory ...
- 409
1
vote
0
answers
24
views
How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?
We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...
- 55