Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
2
votes
0answers
21 views

Circular confidence intervals about scatter points [closed]

I'm wondering if this is a "thing", I quite like the visual and feel it would be nice to know how much the points are likely to move from the location given by the data. The following plot ...
3
votes
0answers
112 views

Is there any consensus or literature on the proportion of font parameters within plots?

This is now about how to make a plot, though that is the information that I'm finding when trying to search for an answer to my question. What I'm wondering, is whether there are any thoughts/...
5
votes
2answers
65 views

Exploratory vs. Descriptive Statistical Analysis

Descriptive statistics definition is pretty clear to say that it summarizes data using statistical methods like mean, mode, median, and spread. However, I came across the term 'exploratory' today ...
0
votes
0answers
17 views

Assessing CFI in Exploratory factor analysis (EFA) in R?

I'm currently conducting EFAs to analyse the internal structure of a construct. I've found a nine factor model with acceptable fit, but only regarding to RMSEA and SRMR. I'm using the program R to ...
0
votes
0answers
11 views

Does Outliers in Categorical Feature varibales exist?

So, I was working on an Exploratory Data analysis project, after dealing with all the preprocessing of numerical features of the dataset when I started analyzing categorical data(nominal) there I ...
0
votes
0answers
16 views

Shapiro-Wilk normality test on paired data

I'm going to perform Shapiro-Wilk normality test before Paired Samples T-test. There are a ...
1
vote
1answer
30 views

How to find the upper outlier threshold in a right skewed distribution?

Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers? One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
0
votes
0answers
10 views

How to conduct item selection via exploratory factor analysis?

I want to reduce my item pool via exploratory factor analysis. I couldn't find any advices on the right order in doing so. What I am asking is: Which is the right order in selecting the items? Shall ...
0
votes
1answer
13 views

Handle various shapes of distribution in data preprocessing

Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
11 views

How to compute factor scores after second-order EFA analysis

I would like to know what are the steps to computing factor scores after a second-order EFA (1 overall factor). I am considering two ways: Ideally, I would like to compute factor scores using a ...
0
votes
0answers
11 views

Tests of the effects of variables on imbalanced categorical variable [duplicate]

I have survey data that include a binary response variable,say y, and various theoretically relevant independent continuous variables. y is binary, which includes two options, say Option A and Option ...
0
votes
1answer
20 views

How to carry out EDA on data consisting of 2D variables

I have a temporal dataset that consists of 2D variables (angle, length) I want to investigate: whether the data has any noticeable structure (e.g. clusters) Whether the sequence of variables (a time ...
1
vote
1answer
229 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
0
votes
2answers
82 views

Checking Normality of Numerical, and Categorical Data

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
0answers
28 views

Checking Normality of Numerical and Categorical Data [duplicate]

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
1answer
15 views

How much can one rely on the mean or median to infer a difference?

When performing an explonatory data analysis, one tries to visualize patterns of behavior of your target variable against the variables that you're assuming could explain the target variable. For ...
0
votes
0answers
35 views

How do you code missing values if 0 is meaningful?

Per this deep learning book I am reading: In general, with neural networks, it’s safe to input missing values as 0, with the condition that 0 isn’t already a meaningful value. The network will ...
0
votes
1answer
16 views

How do you interpret your features when you standardize your data?

Let's say I have built a boosting tree or neural network and I standardized my features beforehand. When I built my model, I split my data into training, validation, and test sets - each with their ...
0
votes
1answer
19 views

At what point does a training-validation-test split need to be made?

If I have a set of data and I am not sure about the data at all or the meaning I would try to investigate the dataset a bit first. I understand the importance of having a training-validation-test ...
0
votes
0answers
13 views

Family members in factor analysis

I'm carrying out an exploratory PCA, followed by an CFA, to test for dimensionality of a questionnaire. I am wondering about the effect of having family members in the same sample. I have a final ...
1
vote
1answer
21 views

The use of Zero Value as Missing Values and Outliers

I am not sure what is the best title for this question. Sorry for that. Let's say I analyzing a dataset of prices of houses. And one of the columns (features) is ...
7
votes
1answer
617 views

Is exploratory data analysis (EDA) actually needed / useful

There are many guides prevalent on the internet about EDA and how everyone should do it and how useful it is however I rarely see it in practice and often times (in said tutorials) it sticks to very ...
11
votes
1answer
10k views

How to interpret notched box plots

While doing some EDA I decided to use a box plot to illustrate the difference between two levels of a factor. The way ggplot rendered the box plot was satisfactory, but slightly simplistic (first ...
0
votes
1answer
36 views

Understanding How Many Components to Include for PCA

I've conducted a PCA on survey data and I'm confused about the concept of 'choosing how many principal components to include'. When interpreting PC1, I found that it successfully captures the ...
1
vote
1answer
53 views

PCA Interpreting strong negative and positive loadings

I ran a PCA on survey data asking whether respondents trust different levels of government, where they rank their level of trust from 0-7. Whilst I can interpret PC1 as a weighted average of ...
1
vote
1answer
552 views

Should exploratory data analysis include validation set?

I know that EDA should be performed on the training set but not on the test set. But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
0
votes
1answer
55 views

Correlation between heavy metals and community data

This is a preliminary study. I want to test whether the presence of heavy metals in surface water negatively affects the benthic macroinvertebrate community. If it's the case, I'll continue the ...
0
votes
0answers
768 views

Does confirmatory = inferential, and exploratory = descriptive analysis / statistics?

Reading this webpage, I wonder: Are confirmatory analysis / statistics and inferential analysis / statistics the same concept? Are exploratory analysis / statistics and descriptive analysis / ...
0
votes
0answers
53 views

QQ plot distribution check

I want to check if a distribution is log normal or not by using qq plot. So for convenience I am creating a lognormal distribution using stats and checking it in ...
4
votes
2answers
165 views

Analysis of binary variables

I have a data set consisting of about a quarter-million objects, each of which may have any of 30 particular features. So I might have Object 1: feature 3, feature 7 Object 2: feature 3, feature 29, ...
1
vote
0answers
10 views

Sources on simulations of the null hypothesis [closed]

I'm looking for two types of sources. Studies on any topic in any field that simulate fake data that are supposed to represent some null distribution to test the validity of some model.I'm just ...
1
vote
0answers
24 views

Finding Patterns In a 3D Plot

I am starting to use 3D graphs to visualize relationships in my data. The reason I am doing this is to try and maximize my predictor variables. Since I am working with financial data the relationships ...
0
votes
0answers
36 views

The utter futility of bivariate analysis

Consider the following quote from here: Moreover, univariable prefiltering, sometimes also referred to as “bivariable analysis,” does not add stability to the selection process as it is based ...
0
votes
0answers
48 views

Should items with low intercorrelation be removed before EFA to avoid many items with low communalities?

I am attempting Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. Background: 221 participants indicated the extent of their agreement on 61 statements, ...
0
votes
0answers
14 views

Effect of feature interaction in classification model

I have a binary classification problem. I have checked below things as EDA- 1) Distribution of features to see any outlier, linear model can be fit? 2) Box plot to see which variables will be ...
15
votes
5answers
5k views

Is it better to do exploratory data analysis on the training dataset only?

I'm doing exploratory data analysis (EDA) on a dataset. Then I will select some features to predict a dependent variable. The question is: Should I do the EDA on my training dataset only? Or ...
1
vote
0answers
35 views

Approaches to analyzing data from forum

Working with self written example data that mimic a chat-forum with different forums (i.e. cars, politics, sports) and different threads (i.e. "why is porsche popular?", "thoughts on andrew yang?") in ...
0
votes
0answers
6 views

Making Parallel Coordinates In Cylindrical Complete Graph Visually Tractable?

In "Addressing the unmet need for visualizing conditional random fields in biological data", a visually intractable, complete parallel coordinate graph appears: By "complete" I mean that a top-down ...
0
votes
0answers
41 views

Book recommendation for analyzing and preparing data for machine learning

I have a school project in which I need to apply several supervised learning algorithms over a data set. Is there a good book that you recommend that focuses on data preparation, like: zero values, <...
0
votes
0answers
25 views

EDA and relationship with ML?

I am starting to learn Exploratory Data Analysis for ML and I have some doubts that I hope to find some help here. First of all, I have been testing some classifiers such as Logistic Regression and ...
1
vote
1answer
19 views

Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
0
votes
2answers
63 views

How exactly does knowing a variable's probability distribution help you when learning about data?

I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize ...
1
vote
1answer
27 views

PCA explained variance that increases linearly

I'm wondering what it says about the data when, instead of dropping off dramatically the variance explained by the number of components continues increasing in a reasonably linear manner. For ...
2
votes
1answer
16 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
0
votes
1answer
41 views

Time series plot with more granularity at present time [closed]

How can i plot realtime timeseries data in a exponential interval.. I mean older data with less granular time scale and present data at higher granular time scale? Any inputs plz
0
votes
1answer
20 views

Can someone give a concrete example of exploitation in the context of Exploratory Data Analysis?

This post says Exploratory Data Analysis (EDA) consists of 2 steps exploration and exploitation. I know a little about exploration which uses some techniques such as data visualization to understand ...
0
votes
0answers
60 views

Is it possible to compute Kaiser-Meyer Olkin (KMO) for all scale items and how?

In another post, what are the assumptions of factor analysis, someone mentioned that it is possible to compute Kaiser-Meyer Olkin (KMO) not just for the whole correlation matrix but for each scale ...
2
votes
0answers
66 views

Exploratory data analysis before pefrorming Canonical Correspondence Analysis (CCA)

I want to perform CCA, but I read that, remember that observations (for example, species abundances) have to present unimodal distributions along gradients (for example, environmental variables). ...
2
votes
2answers
47 views

Is manual pattern recognition a valid scientific tool if followed up by statistical analysis?

Say I have a dataset of some kind, and I am looking through some graphs I have made from it. I notice a pattern, maybe a possible correlation, so I build a computer script or do some spreadsheet-magic ...
1
vote
0answers
38 views

Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...

1
2 3 4 5