Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0
votes
0answers
9 views

Assessing CFI in Exploratory factor analysis (EFA) in R?

I'm currently conducting EFAs to analyse the internal structure of a construct. I've found a nine factor model with acceptable fit, but only regarding to RMSEA and SRMR. I'm using the program R to ...
0
votes
0answers
11 views

Does Outliers in Categorical Feature varibales exist?

So, I was working on an Exploratory Data analysis project, after dealing with all the preprocessing of numerical features of the dataset when I started analyzing categorical data(nominal) there I ...
0
votes
0answers
15 views

Finding intervals of non-NA:s in huge number of time series [closed]

I have around 20 000 time series during a 5-year period and many of the series contain NA:s during this time. I need to find a time period, say 18 months, that contain as little NA:s as possible, in ...
0
votes
0answers
15 views

Shapiro-Wilk normality test on paired data

I'm going to perform Shapiro-Wilk normality test before Paired Samples T-test. There are a ...
1
vote
1answer
26 views

How to find the upper outlier threshold in a right skewed distribution?

Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers? One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
0
votes
0answers
10 views

How to conduct item selection via exploratory factor analysis?

I want to reduce my item pool via exploratory factor analysis. I couldn't find any advices on the right order in doing so. What I am asking is: Which is the right order in selecting the items? Shall ...
0
votes
0answers
11 views

How to compute factor scores after second-order EFA analysis

I would like to know what are the steps to computing factor scores after a second-order EFA (1 overall factor). I am considering two ways: Ideally, I would like to compute factor scores using a ...
0
votes
1answer
13 views

Handle various shapes of distribution in data preprocessing

Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
11 views

Tests of the effects of variables on imbalanced categorical variable [duplicate]

I have survey data that include a binary response variable,say y, and various theoretically relevant independent continuous variables. y is binary, which includes two options, say Option A and Option ...
0
votes
1answer
18 views

How to carry out EDA on data consisting of 2D variables

I have a temporal dataset that consists of 2D variables (angle, length) I want to investigate: whether the data has any noticeable structure (e.g. clusters) Whether the sequence of variables (a time ...
0
votes
0answers
28 views

Checking Normality of Numerical and Categorical Data [duplicate]

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
2answers
58 views

Checking Normality of Numerical, and Categorical Data

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
1answer
15 views

How much can one rely on the mean or median to infer a difference?

When performing an explonatory data analysis, one tries to visualize patterns of behavior of your target variable against the variables that you're assuming could explain the target variable. For ...
0
votes
1answer
15 views

How do you interpret your features when you standardize your data?

Let's say I have built a boosting tree or neural network and I standardized my features beforehand. When I built my model, I split my data into training, validation, and test sets - each with their ...
0
votes
0answers
34 views

How do you code missing values if 0 is meaningful?

Per this deep learning book I am reading: In general, with neural networks, it’s safe to input missing values as 0, with the condition that 0 isn’t already a meaningful value. The network will ...
0
votes
1answer
18 views

At what point does a training-validation-test split need to be made?

If I have a set of data and I am not sure about the data at all or the meaning I would try to investigate the dataset a bit first. I understand the importance of having a training-validation-test ...
0
votes
0answers
13 views

Family members in factor analysis

I'm carrying out an exploratory PCA, followed by an CFA, to test for dimensionality of a questionnaire. I am wondering about the effect of having family members in the same sample. I have a final ...
1
vote
1answer
18 views

The use of Zero Value as Missing Values and Outliers

I am not sure what is the best title for this question. Sorry for that. Let's say I analyzing a dataset of prices of houses. And one of the columns (features) is ...
7
votes
1answer
599 views

Is exploratory data analysis (EDA) actually needed / useful

There are many guides prevalent on the internet about EDA and how everyone should do it and how useful it is however I rarely see it in practice and often times (in said tutorials) it sticks to very ...
0
votes
1answer
35 views

Understanding How Many Components to Include for PCA

I've conducted a PCA on survey data and I'm confused about the concept of 'choosing how many principal components to include'. When interpreting PC1, I found that it successfully captures the ...
1
vote
1answer
53 views

PCA Interpreting strong negative and positive loadings

I ran a PCA on survey data asking whether respondents trust different levels of government, where they rank their level of trust from 0-7. Whilst I can interpret PC1 as a weighted average of ...
0
votes
1answer
55 views

Correlation between heavy metals and community data

This is a preliminary study. I want to test whether the presence of heavy metals in surface water negatively affects the benthic macroinvertebrate community. If it's the case, I'll continue the ...
0
votes
0answers
50 views

QQ plot distribution check

I want to check if a distribution is log normal or not by using qq plot. So for convenience I am creating a lognormal distribution using stats and checking it in ...
1
vote
0answers
10 views

Sources on simulations of the null hypothesis [closed]

I'm looking for two types of sources. Studies on any topic in any field that simulate fake data that are supposed to represent some null distribution to test the validity of some model.I'm just ...
1
vote
0answers
24 views

Finding Patterns In a 3D Plot

I am starting to use 3D graphs to visualize relationships in my data. The reason I am doing this is to try and maximize my predictor variables. Since I am working with financial data the relationships ...
0
votes
0answers
33 views

The utter futility of bivariate analysis

Consider the following quote from here: Moreover, univariable prefiltering, sometimes also referred to as “bivariable analysis,” does not add stability to the selection process as it is based ...
0
votes
0answers
44 views

Should items with low intercorrelation be removed before EFA to avoid many items with low communalities?

I am attempting Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. Background: 221 participants indicated the extent of their agreement on 61 statements, ...
0
votes
0answers
14 views

Effect of feature interaction in classification model

I have a binary classification problem. I have checked below things as EDA- 1) Distribution of features to see any outlier, linear model can be fit? 2) Box plot to see which variables will be ...
1
vote
0answers
35 views

Approaches to analyzing data from forum

Working with self written example data that mimic a chat-forum with different forums (i.e. cars, politics, sports) and different threads (i.e. "why is porsche popular?", "thoughts on andrew yang?") in ...
0
votes
0answers
6 views

Making Parallel Coordinates In Cylindrical Complete Graph Visually Tractable?

In "Addressing the unmet need for visualizing conditional random fields in biological data", a visually intractable, complete parallel coordinate graph appears: By "complete" I mean that a top-down ...
0
votes
0answers
40 views

Book recommendation for analyzing and preparing data for machine learning

I have a school project in which I need to apply several supervised learning algorithms over a data set. Is there a good book that you recommend that focuses on data preparation, like: zero values, <...
0
votes
0answers
24 views

EDA and relationship with ML?

I am starting to learn Exploratory Data Analysis for ML and I have some doubts that I hope to find some help here. First of all, I have been testing some classifiers such as Logistic Regression and ...
1
vote
1answer
18 views

Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
0
votes
2answers
62 views

How exactly does knowing a variable's probability distribution help you when learning about data?

I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize ...
1
vote
1answer
26 views

PCA explained variance that increases linearly

I'm wondering what it says about the data when, instead of dropping off dramatically the variance explained by the number of components continues increasing in a reasonably linear manner. For ...
0
votes
1answer
39 views

Time series plot with more granularity at present time [closed]

How can i plot realtime timeseries data in a exponential interval.. I mean older data with less granular time scale and present data at higher granular time scale? Any inputs plz
2
votes
1answer
16 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
0
votes
1answer
20 views

Can someone give a concrete example of exploitation in the context of Exploratory Data Analysis?

This post says Exploratory Data Analysis (EDA) consists of 2 steps exploration and exploitation. I know a little about exploration which uses some techniques such as data visualization to understand ...
0
votes
0answers
55 views

Is it possible to compute Kaiser-Meyer Olkin (KMO) for all scale items and how?

In another post, what are the assumptions of factor analysis, someone mentioned that it is possible to compute Kaiser-Meyer Olkin (KMO) not just for the whole correlation matrix but for each scale ...
1
vote
1answer
488 views

Should exploratory data analysis include validation set?

I know that EDA should be performed on the training set but not on the test set. But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
2
votes
0answers
64 views

Exploratory data analysis before pefrorming Canonical Correspondence Analysis (CCA)

I want to perform CCA, but I read that, remember that observations (for example, species abundances) have to present unimodal distributions along gradients (for example, environmental variables). ...
2
votes
2answers
47 views

Is manual pattern recognition a valid scientific tool if followed up by statistical analysis?

Say I have a dataset of some kind, and I am looking through some graphs I have made from it. I notice a pattern, maybe a possible correlation, so I build a computer script or do some spreadsheet-magic ...
1
vote
0answers
35 views

Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...
0
votes
1answer
18 views

Quantifying volatility with limited data?

I have ~400 time series's that I need to quantify volatility over. The only information I have is mean, min, and max values for each time stamp. Here is a sample series for your convenience: My ...
0
votes
1answer
36 views

Exploring a loan data set when I'm interested in change in number of loans over time, not acceptance/rejection, defaulting, etc

This is a simple question but I am having difficulty wrapping my head around it. I have a dataset listing a bunch of individual loans. There are tens of variables describing aspects of the loans, mix ...
1
vote
0answers
55 views

MCA in FactoMineR: all variables are the same on Dim 1?

My data is ordinal (1, 2, 3, 4, 5, NA from a likert scale) and doesn't have any obvious pattern looking at the raw data. My code looks like this: library(FactoMineR) df <- data.frame(Var1 ...
1
vote
0answers
201 views

How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
1
vote
0answers
54 views

Data exploration methods for noisy time series

I have time series data from NanoPore sequencing (Attached is a illustrative figure and short explanation) which I'd ultimately like to use in order to find various methylation patterns in the input ...
0
votes
1answer
23 views

Creating a custom metric based on diverse feature scalse

I would like to create a custom metric that takes into account multiple features and produces a certain score. However, the issue is that I have data that has many different data types, such as ...
0
votes
0answers
38 views

Too many factors extracted from EFA!

I am constructing a scale. I have 80 items and 300 sample data. I ran EFA - Principal Axis Method of extraction and direct oblimin rotation in SPSS. I have 23 factors (using both Eigen value and ...

1
2 3 4 5