Questions tagged [exploratory-data-analysis]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Top 5 Things you look for in sns.pairplot() for a regression problem? [closed]

Where can I find a laundry list of "What I Should Be Looking For" when reading a sns.pairplot()? There are a ton of resources that show you how cool a ...
  • 422
0 votes
0 answers
14 views

Interpretation of Exploratory Factor Analysis

I am conducting research in cognition, where we are reducing a large dataset. An Exploratory Factor Analaysis with a 3-factor solution (based on a scree plot) reveals factors that appear to correspond ...
  • 1
1 vote
0 answers
15 views

Exploratory factor analysis (EFA) for transcriptomic data

I am doing an EFA for transcriptomic data (n=202, p=190). I did log-transformed the data because of skewness. My question is, do I have to do false discovery rate (FDR) analysis at all prior to my EFA....
  • 11
0 votes
0 answers
22 views

Why should exploratory analysis not be followed up with a confirmatory analysis in the same dataset? [duplicate]

On the Wikipedia page for data analysis, the following claim is made. "...one should not follow up an exploratory analysis with a confirmatory analysis in the same dataset. An exploratory ...
0 votes
0 answers
6 views

Help with interpreting unusual EFA result

I was hoping to receive some advice on how to properly interpret the results of my principal axis factoring analysis. The results seem to load well onto one factor except for one item which has a ...
0 votes
0 answers
36 views

OLS model assumptions for ML

hey i'm a beginner in data science, started working on the familiar "Boston" dataset ,and i tried to fit a multiple linear regression model to predict the "median house price " , ...
1 vote
1 answer
33 views

Standardized vs. unstandardized PCA

I performed PCA using two different R functions fviz() and ggbiplot() to improve the visualization of data. The sum of the first two PC is equal between the two methods, but there is a different value ...
  • 23
0 votes
0 answers
12 views

What is the best way to identify demographic characteristics from within pooled survey data?

I have conducted a survey looking at individual exercise habits and their preferences regarding technology when measuring exercise and providing feedback. I am in the process of analysing the data ...
  • 1
1 vote
1 answer
13 views

Non uniform relationship between predictors and response variable

I aim to determine the relation between Y (a dichotomous variable, Y/N) and multiple Xs (Continuous variables). In the following charts, X axis represents the binned value of an X, while Y axis ...
  • 25
0 votes
0 answers
25 views

Scree plot depicting the proportion of variance?

I'm reading the 2nd edition of the book "An Introduction to Statistical Learning" from Springer. In page 507, they have a scree plot with y-axis is the proportion of variance. And the ...
  • 1
0 votes
0 answers
20 views

Should I remove/cap "Extreme Values" from Variables and at which percentile should capping be done?

In my dataset, I have a variable corresponding to the monthly average playtime hours of the players. The dataset has 200K records. I estimated various percentiles of the variable - from 0 to 1 for ...
0 votes
0 answers
33 views

PCA: Does the sum of all the features (not the components) is always linearly correlated to F1?

I'm trying to figure out if we always have: $$pearson\_corr(F_1, \sum_{k=1}^{n}X_i) = 1$$ with: $F_1$ the principal component, $X_i$ the numerical features we used for the PCA, $n$ the number of ...
1 vote
1 answer
30 views

Principal component vs. Exploratory Factor Analysis [duplicate]

Problem: Which method should I use to observe whether there is one common factor/component explaining six different measures (i.e., face and objects)? Study: In this study, we have four measures of ...
0 votes
0 answers
13 views

The actual unit of sample in the data set is higher than the planned unit of sample

the population is all the fish from a lake, sampling frame is all the predator fish, the cased study mentioned that up to 4 fish from the hierarchical order of preferred predator species were obtained'...
1 vote
1 answer
41 views

What is the R function for EDA plots like this? [closed]

I am searching for the R function that generated this Exploratory Data Analysis plot. Anyone know what it is?
  • 205
2 votes
2 answers
73 views

How to analyse a data set with more than 500 variables

I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview. My problem is not that ...
1 vote
0 answers
25 views

How to reconcile number of factors found with parallel factor analysis and network plotting in exploratory factor analysis?

I am trying to apply exploratory factor analysis and I got the following output when using parallel factor analysis (fa.parallel of the R package ...
  • 149
1 vote
0 answers
24 views

Is there an improved canonical polyadic decomposition for symmetric tensors?

Let us suppose I want to find a CP decomposition of a $n$-mode tensor $\mathcal{A}$. Fortunately the tensor has the permutation symmetry $$\mathcal{A}[i_1, \cdots, i_n] = \mathcal{A}[\sigma (i_1), \...
  • 5,626
1 vote
0 answers
20 views

On dealing with missing data for longitudinal exploratory data analysis

I have a longitudinal dataset. I wish to do some exploratory data analysis on it. However, it has the problem of missing observations/non-response. Q: How does one do exploratory data analysis on this ...
  • 1,315
0 votes
0 answers
15 views

Incremental value of a categorical variable

The question is to calculate the incremental value of running an advertisement for a client. I have provided a sample of my dataset. ...
0 votes
1 answer
28 views

When and how to perform eda

I have some doubts on when to perform eda and the preprocessing the dataset should be put through. Should EDA be performed before or after balancing the dataset? Should one hot encoding and label ...
  • 105
0 votes
0 answers
42 views

How to handle multiple time series data with covariates in R

I have time series data on traffic flow in different locations. I want to look at how weather (time series) and location properties (e.g., accessibility calculated by road network and keep unchanged ...
3 votes
1 answer
321 views

Are coordinates (e.g. Point(123, -123)) considered interval data?

I understand that latitude and longitude are interval data, but are coordinates (e.g. Point(123, -123)) also considered interval data? If so, how can the standard deviation, mean etc. be calculated?
  • 31
0 votes
0 answers
52 views

How to do an MCA analysis on a binary dataset?

I'm looking to do an EDA with some binary data as seen below and I was recommended to use MCA. However, I'm not sure about how to go about doing this in a meaningful way. I'm looking to find key ...
2 votes
0 answers
122 views

How do I do an EDA on boolean/binary data without ML?

I'm looking to get some insight on what methods I can use to help identify key factors in my dataset that influence my dependent variable and how to prove that they are statically significant. There ...
0 votes
0 answers
28 views

Can exploratory data analysis data overlap with that of confirmatory data analysis?

Sometimes data is expensive to obtain. I will do a random draw of small subset of data to do exploratory data analysis. Q: Can I do confirmatory data analysis on full data if data is limited? Or can ...
  • 1,315
-2 votes
1 answer
61 views

what does it means for p value = 8.780e-13 [duplicate]

I would like to ask if 8.780e-13 is greater than 0.05 for significance? I am not sure what is the calculation of this. Thanks a lot!
  • 9
0 votes
0 answers
24 views

Is it any meaningful taking the histogram of many histograms?

So I’m working on pixel-level assessment of chest x-ray images from the NIH Chest X-ray dataset. The diseases labels on the images are a set of z different categories (i.e. Pneumonia, effusion, etc…). ...
1 vote
1 answer
61 views

Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?

Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the ...
  • 531
2 votes
0 answers
44 views

Which transformation technique should we use for 2 bell shape distribution?

I'm doing exploratory data analysis and trying to transform the feature with log transformation, but I'm not sure which transformation to use for the two bell shaped distributions. Is it required to ...
  • 31
0 votes
0 answers
94 views

Correlation between 2 columns after doing percentage change between consecutive rows for both columns

I am checking whether two columns in my pandas dataframe are correlated, but then wondering if it makes a difference if I do percentage change first (using pandas pct_change().mul(100) function) to ...
  • 1
1 vote
1 answer
12 views

Do differences between ordinations prove a relationship between variables?

I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
0 votes
0 answers
26 views

Multiple Comparisons Exploratory Analysis

I am junior analyst working with a group of physicians on a paper looking at healthcare utilization for people with a certain DX across between COVID and non-COVID years. We have several groups to ...
  • 1
0 votes
0 answers
21 views

EDA with Data Modeling

After I read R for DataScience and ggplot2: elegant graphics for data analysis, I am learning how use modeling techniques to improve my EDA. I applied this on two notebooks (https://www.kaggle.com/...
6 votes
3 answers
162 views

Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
  • 531
0 votes
0 answers
49 views

Cluster confidence scoring

I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info. e....
2 votes
1 answer
49 views

In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?

I have generated a multiple-regression equation/model from the data. Parameter A in the data varies from $0.1$ - $10$. Parameter B in the data varies from $100$ - $1800$. Parameter C in the data ...
0 votes
1 answer
78 views

Why do our devices fail?

I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
  • 2,849
1 vote
1 answer
357 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
  • 215
2 votes
2 answers
236 views

Models for describing cluster change over time

Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
  • 2,164
2 votes
1 answer
38 views

Does multicollinearity affect EDA?

I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
4 votes
3 answers
1k views

Which are outliers?

I am in the process of solving a Machine Learning challenge, and I want to do it the right way. I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in ...
0 votes
1 answer
25 views

Describing normal volume change with age: use a linear model and the standard deviation?

I would like to use a large dataset with tissue volume measures of 500 volunteers over a given ageframe to compare it to a small dataset with volunteers and patients (20-30 datapoints). The tissue ...
  • 309
2 votes
0 answers
140 views

Is there a reason to look at histogram of residuals of linear regression fit in order to determine if Poisson regression should be used?

Introduction When reading lecture slides from a lecture I missed, one slide seemed to suggest that when presented with count data (the response being a count of something) one should try to fit a ...
  • 171
5 votes
2 answers
1k views

How can I know If LASSO logistic regression model is good enough to be feature selection tool?

It is known that LASSO can be used for feature selection. How can I know if the model is reliable for that purpose? In general the model's accuracy, R squared and etc, don't bother me because I don't ...
  • 121
0 votes
1 answer
155 views

What is the difference between EDA (Explorative Data Analysis) and Data Profiling?

I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same?? Can someone clarify? Thanks
  • 1
1 vote
2 answers
148 views

Feature selection- feature that most of its values are equal. How can I know if to drop it?

If I have a feature that most of its values are the same, how can I know if to drop it or to keep it? First of all, most of the model's features have low variance. Secondly, maybe the observations ...
  • 121
1 vote
1 answer
19 views

What is good consolidated/normalized metric with funnel page views data?

I am currently working on a best way to represent product interest score based on page views. Working on an ecommerce use case having homepage, search page, and product page interaction data available....
  • 11
0 votes
1 answer
82 views

At first if I fail, ....should I keep trying

The final goal is to predict the effect of intervention(A) on outcome(Y) in the presence of some confounders (x). Before running the model to evaluate the effect of A on Y, I am doing some Exploratory ...
  • 409
1 vote
0 answers
24 views

How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?

We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...

1
2 3 4 5
7