Questions tagged [exploratory-data-analysis]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
-2 votes
0 answers
9 views

How to make a two-line mediator analysis in R? [closed]

How to make a two-line mediator analysis in R? Please tell me briefly how to do this. What function should I use? I don't know much about data analysis, but I really need to figure out how to do this ...
user avatar
  • 1
1 vote
1 answer
8 views

Do differences between ordinations prove a relationship between variables?

I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
user avatar
0 votes
0 answers
14 views

Multiple Comparisons Exploratory Analysis

I am junior analyst working with a group of physicians on a paper looking at healthcare utilization for people with a certain DX across between COVID and non-COVID years. We have several groups to ...
user avatar
  • 1
0 votes
0 answers
10 views

EDA with Data Modeling

After I read R for DataScience and ggplot2: elegant graphics for data analysis, I am learning how use modeling techniques to improve my EDA. I applied this on two notebooks (https://www.kaggle.com/...
user avatar
3 votes
1 answer
70 views

Should stepwise regressions or overfitting also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
user avatar
  • 383
0 votes
0 answers
8 views

Cluster confidence scoring

I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info. e....
user avatar
2 votes
1 answer
22 views

In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?

I have generated a multiple-regression equation/model from the data. Parameter A in the data varies from $0.1$ - $10$. Parameter B in the data varies from $100$ - $1800$. Parameter C in the data ...
user avatar
0 votes
1 answer
62 views

Why do our devices fail?

I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
user avatar
  • 2,197
1 vote
1 answer
46 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
user avatar
  • 215
2 votes
2 answers
92 views

Models for describing cluster change over time

Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
user avatar
  • 2,059
1 vote
1 answer
24 views

Does multicollinearity affect EDA?

I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
user avatar
0 votes
0 answers
20 views

How to obtain the KMO index from a correlation matrix?

I have a correlation matrix that does the KMO index. In my research, I have only a correlation matrix of 7 variables. I need to do the KMO index but most softwares do not get input a correlation ...
user avatar
  • 1
4 votes
3 answers
1k views

Which are outliers?

I am in the process of solving a Machine Learning challenge, and I want to do it the right way. I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in ...
user avatar
0 votes
1 answer
21 views

Describing normal volume change with age: use a linear model and the standard deviation?

I would like to use a large dataset with tissue volume measures of 500 volunteers over a given ageframe to compare it to a small dataset with volunteers and patients (20-30 datapoints). The tissue ...
user avatar
  • 169
0 votes
0 answers
32 views

Correlation between Ordinal Data and Binary Data

Hello this is for a personal project of mine regarding surveys. I wanted to compare two responses with each other with some sort of statistic. - A B 1 600 196 2 276 466 3 36 304 4 20 150 5 22 44 ...
user avatar
2 votes
0 answers
65 views

Is there a reason to look at histogram of residuals of linear regression fit in order to determine if Poisson regression should be used?

Introduction When reading lecture slides from a lecture I missed, one slide seemed to suggest that when presented with count data (the response being a count of something) one should try to fit a ...
user avatar
  • 171
4 votes
2 answers
535 views

How can I know If LASSO logistic regression model is good enough to be feature selection tool?

It is known that LASSO can be used for feature selection. How can I know if the model is reliable for that purpose? In general the model's accuracy, R squared and etc, don't bother me because I don't ...
user avatar
  • 67
0 votes
1 answer
27 views

What is the difference between EDA (Explorative Data Analysis) and Data Profiling?

I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same?? Can someone clarify? Thanks
user avatar
  • 1
1 vote
2 answers
54 views

Feature selection- feature that most of its values are equal. How can I know if to drop it?

If I have a feature that most of its values are the same, how can I know if to drop it or to keep it? First of all, most of the model's features have low variance. Secondly, maybe the observations ...
user avatar
  • 67
1 vote
1 answer
18 views

What is good consolidated/normalized metric with funnel page views data?

I am currently working on a best way to represent product interest score based on page views. Working on an ecommerce use case having homepage, search page, and product page interaction data available....
user avatar
  • 11
0 votes
0 answers
9 views

How do I handle underidentification in Bifactor ESEMs?

I hope you can help me to find some answers to my questions. Following Morin, Arens, & Marsh (2016; references below), I’m trying to conduct a bifactor exploratory structural equation model (...
user avatar
  • 1
0 votes
0 answers
19 views

Identifying outliers in data collection with binary variables

I have a bunch of data collected on a set of engines. Most of my data is a flag if a particular feature works/doesn't or Issue_X exists/doesn't exist and then a final engine rating is given. I want to ...
user avatar
  • 450
0 votes
1 answer
78 views

At first if I fail, ....should I keep trying

The final goal is to predict the effect of intervention(A) on outcome(Y) in the presence of some confounders (x). Before running the model to evaluate the effect of A on Y, I am doing some Exploratory ...
user avatar
  • 385
1 vote
0 answers
21 views

How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?

We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...
user avatar
0 votes
0 answers
43 views

PCA: is linearity important?

I have a 6 dimensions dataset where I want to apply PCA to remove one dimension. I did a small analysis to check for relationships in my data and concluded that there is very low linear correlation ...
user avatar
  • 16
0 votes
0 answers
22 views

PCA: does outlier detection make sense with low linear correlation? [duplicate]

I am experimenting PCA to detect outliers based on the reconstruction error. What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and ...
user avatar
  • 16
0 votes
0 answers
16 views

Holdout and the Multiple Comparisons Problem

I apologize in advance if this is an extremely naive question. I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
user avatar
4 votes
0 answers
52 views

How can orthogonal factors explain more than 100% of the variance? [duplicate]

I'm using Stata's exploratory factor analysis command ("factor"). When I ask for a rotated solution (using the varimax orthogonal method, which is the default) it gives me (aside from the ...
user avatar
0 votes
0 answers
43 views

Outliers for N=4

If you have 4 observations, can you have an outlier? Consider any value outside (Lower fourth-1.5(Fourth Spread), upper fourth+1.5(fourth spread)) as an outlier.
user avatar
  • 1
0 votes
0 answers
26 views

How to conduct exploratory analysis of repeated measures?

Exploratory analysis of repeated measures--without running statistical tests (told not to because study is underpowered and exploratory...?), what is the best way to visualize and/or quantify trends? ...
user avatar
1 vote
1 answer
34 views

Confirmatory Factor Analysis identifying

I have been explained this technique in class. However, I did not understand some stuff. The professor said that a model to be estimated needs to be at least identified. Identified meaning to be when ...
user avatar
3 votes
1 answer
51 views

Need data preparation cheatsheet / guidelines / first principles to train team members! [closed]

Really ran out of ideas and hence such a basic question to the community - Despite the repeated emphasis on ensuring data accuracy/validity, team members just do not spend enough time on it because ...
0 votes
0 answers
27 views

How to analyse this bimodal gamma distribution?

I have collected physiological data with multiple observations from 35 people, across four conditions. In planning the experiment, I had been hoping to perform inferential statistics comparing between ...
user avatar
1 vote
0 answers
16 views

How should I identify the most important motivations to my respondents - factor means or variance explained by factors? (detected by EFA)

I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained 62% of the ...
user avatar
1 vote
1 answer
36 views

Can I use construct/factor (revealed by EFA) mean scores as explanatory variables in a GLM (binomial family)?

I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained about 58% of the ...
user avatar
0 votes
0 answers
18 views

Efficient metrics for measuring interaction and Building features

Long time user of tree models, rediscovered the benefits of Linear models recently. Now I want to transfer as much non linearity as possible into my features so to mimic as much as possible the ...
user avatar
  • 209
1 vote
0 answers
32 views

Many Preprocessing steps can cause data leakage, then how should we perform EDA?

For the past week, I have been constantly checking with people on this sub on how to avoid data leakage during preprocessing like feature selection and/or scaling etc here and here. I understand most ...
user avatar
  • 533
0 votes
0 answers
15 views

Determining the range of input feature values

I have an application (numerical simulation) in hand, for which I am generating my own data. Initially, I wanted to generate a small subset of data and perform initial ML model training. I was ...
user avatar
0 votes
1 answer
63 views

On the misuse of two sample test: How to report difference in entire "populations"

I have the following understanding: Two (or more) sample tests are only required if conclusion should be drawn about a population based on observed samples (typically much smaller than the population)....
user avatar
  • 112
1 vote
0 answers
28 views

Differences in the factor - variable relationship in EFA and component - variable relationship in PCA?

When I read about exploratory factor analyses, I saw equations showing that each variable is a linear composite of different factors - with loadings correspond to the coefficient in front of each ...
user avatar
  • 69
0 votes
0 answers
52 views

Are there two types of Principal Component Analyses? [duplicate]

At first, I learned that PCA is a different type of analysis than factor analysis: PCA is only used for data reduction and factor analysis is used for detecting latent variables that underlie observed ...
user avatar
  • 69
0 votes
1 answer
50 views

Exploratory Factor Analysis for Big Data: Smaller Loading Threshold and Variance Accounted For?

I saw several pieces of literature mentioned that the larger the sample size, the lower the threshold to be set for loadings to be considered significant, and the lower the variance accounted for. I ...
user avatar
  • 69
0 votes
0 answers
21 views

Differences in Model Fit Evaluation Between Exploratory Factor Analysis and Confirmatory Factor Analysis

For the same dataset, I tried both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). For evaluating EFA model, the cumulative explanation of all latent factors adds up to ...
user avatar
  • 69
0 votes
0 answers
27 views

What data cleaning method is best to be adopted for price=$0?

Suppose I have an AirBnb dataset. In EDA, i noticed that some room price are 0 (as shown in the example below). Is removing them a best approach since its not possible that a room price is 0? Or ...
user avatar
0 votes
0 answers
21 views

Methodology for exploratory analysis on dataset

This question is about finding the right methodology: A population of cash machine is being analyzed. The variable we are measuring is the money income produced. We are analyzing the variable over a ...
user avatar
0 votes
2 answers
61 views

How to deal with missing values?

I have a question regarding exploratory data analysis, specifically, how to deal with missing values. Here is a toy dataset to describe my question: In exploring each variable (a frequency table ...
user avatar
1 vote
1 answer
21 views

EFA with "found" data

The textbook procedure for EFA is to start by designing the data collection, then you run the EFA to locate (we hope) the latent variables, therein. Does that mean it is not ever valid to conduct and ...
user avatar
  • 831
0 votes
0 answers
39 views

Exploring relation between x and y [duplicate]

While there are numerous methods in exploring relation between x and y upon receiving a new dataset. Yet it seems I can't find any conclusive guide as of the sequence of analysis, e.g. do correlation ...
user avatar
0 votes
0 answers
62 views

Standardization for using two similar data sets

I need your advice regarding standardization: I have collected two similar datasets from the same population: Error count and time for a group that was tested in person. Error count and time for a ...
user avatar
0 votes
0 answers
26 views

EFA and CFA with a lot of non-random "missing data"

This may be silly question, but I am only newly learning factor analysis. I am working on a project on social media. One of the problems is of the following form: We have circulated a form about ...
user avatar

1
2 3 4 5 6