Questions tagged [exploratory-data-analysis]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

The actual unit of sample in the data set is higher than the planned unit of sample

the population is all the fish from a lake, sampling frame is all the predator fish, the cased study mentioned that up to 4 fish from the hierarchical order of preferred predator species were obtained'...
1 vote
1 answer
30 views

What is the R function for EDA plots like this? [closed]

I am searching for the R function that generated this Exploratory Data Analysis plot. Anyone know what it is?
  • 105
2 votes
2 answers
63 views

How to analyse a data set with more than 500 variables

I have a data set with 1000 entities. For each entity, more than 500 dichotomous variables (0,1) are recorded. I have no idea how to analyse this data or how to get an overview. My problem is not that ...
1 vote
0 answers
9 views

How to reconcile number of factors found with parallel factor analysis and network plotting in exploratory factor analysis?

I am trying to apply exploratory factor analysis and I got the following output when using parallel factor analysis (fa.parallel of the R package ...
  • 149
1 vote
0 answers
20 views

Is there an improved canonical polyadic decomposition for symmetric tensors?

Let us suppose I want to find a CP decomposition of a $n$-mode tensor $\mathcal{A}$. Fortunately the tensor has the permutation symmetry $$\mathcal{A}[i_1, \cdots, i_n] = \mathcal{A}[\sigma (i_1), \...
  • 5,102
1 vote
0 answers
12 views

On dealing with missing data for longitudinal exploratory data analysis

I have a longitudinal dataset. I wish to do some exploratory data analysis on it. However, it has the problem of missing observations/non-response. Q: How does one do exploratory data analysis on this ...
  • 1,169
0 votes
0 answers
15 views

Incremental value of a categorical variable

The question is to calculate the incremental value of running an advertisement for a client. I have provided a sample of my dataset. ...
0 votes
1 answer
24 views

When and how to perform eda

I have some doubts on when to perform eda and the preprocessing the dataset should be put through. Should EDA be performed before or after balancing the dataset? Should one hot encoding and label ...
  • 105
0 votes
0 answers
26 views

How to handle multiple time series data with covariates in R

I have time series data on traffic flow in different locations. I want to look at how weather (time series) and location properties (e.g., accessibility calculated by road network and keep unchanged ...
3 votes
1 answer
191 views

Are coordinates (e.g. Point(123, -123)) considered interval data?

I understand that latitude and longitude are interval data, but are coordinates (e.g. Point(123, -123)) also considered interval data? If so, how can the standard deviation, mean etc. be calculated?
  • 31
0 votes
0 answers
35 views

How to do an MCA analysis on a binary dataset?

I'm looking to do an EDA with some binary data as seen below and I was recommended to use MCA. However, I'm not sure about how to go about doing this in a meaningful way. I'm looking to find key ...
2 votes
0 answers
56 views

How do I do an EDA on boolean/binary data without ML?

I'm looking to get some insight on what methods I can use to help identify key factors in my dataset that influence my dependent variable and how to prove that they are statically significant. There ...
0 votes
0 answers
23 views

Can exploratory data analysis data overlap with that of confirmatory data analysis?

Sometimes data is expensive to obtain. I will do a random draw of small subset of data to do exploratory data analysis. Q: Can I do confirmatory data analysis on full data if data is limited? Or can ...
  • 1,169
-2 votes
1 answer
44 views

what does it means for p value = 8.780e-13 [duplicate]

I would like to ask if 8.780e-13 is greater than 0.05 for significance? I am not sure what is the calculation of this. Thanks a lot!
  • 9
0 votes
0 answers
23 views

Is it any meaningful taking the histogram of many histograms?

So I’m working on pixel-level assessment of chest x-ray images from the NIH Chest X-ray dataset. The diseases labels on the images are a set of z different categories (i.e. Pneumonia, effusion, etc…). ...
1 vote
1 answer
44 views

Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?

Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the ...
  • 507
2 votes
0 answers
32 views

Which transformation technique should we use for 2 bell shape distribution?

I'm doing exploratory data analysis and trying to transform the feature with log transformation, but I'm not sure which transformation to use for the two bell shaped distributions. Is it required to ...
  • 31
0 votes
0 answers
57 views

Correlation between 2 columns after doing percentage change between consecutive rows for both columns

I am checking whether two columns in my pandas dataframe are correlated, but then wondering if it makes a difference if I do percentage change first (using pandas pct_change().mul(100) function) to ...
  • 1
1 vote
1 answer
10 views

Do differences between ordinations prove a relationship between variables?

I have a large dataset that contains a variety of environmental characteristics (air temperature, tree cover, insect counts) as well as bird abundance data for each point. I was talking with my friend ...
0 votes
0 answers
20 views

Multiple Comparisons Exploratory Analysis

I am junior analyst working with a group of physicians on a paper looking at healthcare utilization for people with a certain DX across between COVID and non-COVID years. We have several groups to ...
  • 1
0 votes
0 answers
19 views

EDA with Data Modeling

After I read R for DataScience and ggplot2: elegant graphics for data analysis, I am learning how use modeling techniques to improve my EDA. I applied this on two notebooks (https://www.kaggle.com/...
6 votes
3 answers
132 views

Should stepwise regressions also be avoided for exploratory (hypothesis generating) modelling?

In a recent paper, Andrew Tredennick and colleagues (2021) suggested to use the drop1() function in R for exploratory modelling (that is to generate new hypotheses ...
  • 507
0 votes
0 answers
24 views

Cluster confidence scoring

I have a scenario where I am provided a list of clusters and pairwise distance only between items in same cluster. I need to rank these clusters based on some kind of relative score from this info. e....
2 votes
1 answer
34 views

In multiple regression, How to find the range of independent parameter for which regression equation is applicable/valid?

I have generated a multiple-regression equation/model from the data. Parameter A in the data varies from $0.1$ - $10$. Parameter B in the data varies from $100$ - $1800$. Parameter C in the data ...
0 votes
1 answer
63 views

Why do our devices fail?

I would like to figure out why some devices of my company fail. Therefore, I'm able to use a list in which around 300 devices are listed together with about 70 parameters while only half of it is ...
  • 2,441
1 vote
1 answer
178 views

Class separability and Overlap

I have a dataset for five different classes with 40 features. This dataset is somehow imbalanced with 2 majority & 2 monirity classes, the other somehow average. This is a classification task and ...
  • 215
2 votes
2 answers
148 views

Models for describing cluster change over time

Suppose I have a set of observations encoded as a matrix $X$ whose entries $X_{it} \in \mathcal{C} $ correspond to cluster labels of experimental unit $i$ with repeated measures at time $t$, where $\...
  • 2,141
1 vote
1 answer
31 views

Does multicollinearity affect EDA?

I have been working on a dataset pertaining to 'churn analysis'. I have been trying to demonstrate whether the customers that are being charged more are also the ones that churn more or not. My ...
0 votes
0 answers
27 views

How to obtain the KMO index from a correlation matrix?

I have a correlation matrix that does the KMO index. In my research, I have only a correlation matrix of 7 variables. I need to do the KMO index but most softwares do not get input a correlation ...
  • 1
4 votes
3 answers
1k views

Which are outliers?

I am in the process of solving a Machine Learning challenge, and I want to do it the right way. I did some exploratory data analysisand I wanted to check the distribution of the data. As displayed in ...
0 votes
1 answer
24 views

Describing normal volume change with age: use a linear model and the standard deviation?

I would like to use a large dataset with tissue volume measures of 500 volunteers over a given ageframe to compare it to a small dataset with volunteers and patients (20-30 datapoints). The tissue ...
  • 189
0 votes
0 answers
50 views

Correlation between Ordinal Data and Binary Data

Hello this is for a personal project of mine regarding surveys. I wanted to compare two responses with each other with some sort of statistic. - A B 1 600 196 2 276 466 3 36 304 4 20 150 5 22 44 ...
2 votes
0 answers
99 views

Is there a reason to look at histogram of residuals of linear regression fit in order to determine if Poisson regression should be used?

Introduction When reading lecture slides from a lecture I missed, one slide seemed to suggest that when presented with count data (the response being a count of something) one should try to fit a ...
  • 171
4 votes
2 answers
822 views

How can I know If LASSO logistic regression model is good enough to be feature selection tool?

It is known that LASSO can be used for feature selection. How can I know if the model is reliable for that purpose? In general the model's accuracy, R squared and etc, don't bother me because I don't ...
  • 77
0 votes
1 answer
72 views

What is the difference between EDA (Explorative Data Analysis) and Data Profiling?

I am a little confused here, seems like both use same/similar techniques. Is here a difference at all or it is the same?? Can someone clarify? Thanks
  • 1
1 vote
2 answers
107 views

Feature selection- feature that most of its values are equal. How can I know if to drop it?

If I have a feature that most of its values are the same, how can I know if to drop it or to keep it? First of all, most of the model's features have low variance. Secondly, maybe the observations ...
  • 77
1 vote
1 answer
19 views

What is good consolidated/normalized metric with funnel page views data?

I am currently working on a best way to represent product interest score based on page views. Working on an ecommerce use case having homepage, search page, and product page interaction data available....
  • 11
0 votes
0 answers
10 views

How do I handle underidentification in Bifactor ESEMs?

I hope you can help me to find some answers to my questions. Following Morin, Arens, & Marsh (2016; references below), I’m trying to conduct a bifactor exploratory structural equation model (...
  • 1
0 votes
0 answers
28 views

Identifying outliers in data collection with binary variables

I have a bunch of data collected on a set of engines. Most of my data is a flag if a particular feature works/doesn't or Issue_X exists/doesn't exist and then a final engine rating is given. I want to ...
  • 450
0 votes
1 answer
79 views

At first if I fail, ....should I keep trying

The final goal is to predict the effect of intervention(A) on outcome(Y) in the presence of some confounders (x). Before running the model to evaluate the effect of A on Y, I am doing some Exploratory ...
  • 375
1 vote
0 answers
23 views

How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?

We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...
0 votes
0 answers
47 views

PCA: is linearity important?

I have a 6 dimensions dataset where I want to apply PCA to remove one dimension. I did a small analysis to check for relationships in my data and concluded that there is very low linear correlation ...
  • 16
0 votes
0 answers
22 views

PCA: does outlier detection make sense with low linear correlation? [duplicate]

I am experimenting PCA to detect outliers based on the reconstruction error. What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and ...
  • 16
0 votes
0 answers
16 views

Holdout and the Multiple Comparisons Problem

I apologize in advance if this is an extremely naive question. I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
4 votes
0 answers
55 views

How can orthogonal factors explain more than 100% of the variance? [duplicate]

I'm using Stata's exploratory factor analysis command ("factor"). When I ask for a rotated solution (using the varimax orthogonal method, which is the default) it gives me (aside from the ...
0 votes
0 answers
53 views

Outliers for N=4

If you have 4 observations, can you have an outlier? Consider any value outside (Lower fourth-1.5(Fourth Spread), upper fourth+1.5(fourth spread)) as an outlier.
  • 1
0 votes
0 answers
38 views

How to conduct exploratory analysis of repeated measures?

Exploratory analysis of repeated measures--without running statistical tests (told not to because study is underpowered and exploratory...?), what is the best way to visualize and/or quantify trends? ...
1 vote
1 answer
43 views

Confirmatory Factor Analysis identifying

I have been explained this technique in class. However, I did not understand some stuff. The professor said that a model to be estimated needs to be at least identified. Identified meaning to be when ...
3 votes
1 answer
51 views

Need data preparation cheatsheet / guidelines / first principles to train team members! [closed]

Really ran out of ideas and hence such a basic question to the community - Despite the repeated emphasis on ensuring data accuracy/validity, team members just do not spend enough time on it because ...
0 votes
0 answers
47 views

How to analyse this bimodal gamma distribution?

I have collected physiological data with multiple observations from 35 people, across four conditions. In planning the experiment, I had been hoping to perform inferential statistics comparing between ...

1
2 3 4 5
7