Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0
votes
0answers
18 views

How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
2
votes
2answers
31 views

Comparing yearly data that has different $n$ values per year?

I have been collecting soil moisture data at my research site for almost 20 years. However, I have not been consistent with the number of times that I have been taken these measurements. The problem ...
0
votes
0answers
20 views

What is the use Additive and Multiplicative decomposition

I am trying to do an Exploratory Data Analysis over the time series based data. While doing some research on it and came up with the term Additive and Multiplicative Decomposition, but i am not able ...
0
votes
0answers
6 views

Exclusion of Spuriousness With More Dimensions?

The usual way of excluding spuriousness in multivariate correlation is to increase the number of samples. However, if increasing the dimensions of measurements produces an unusual magnitude of ...
5
votes
1answer
88 views

What is the defintion of variation in a box plot?

Variation is a term I have always been confused with. The first time I heard it, it was defined as "the different between the max and min values in a distribution". Lately, I've been hearing ...
1
vote
1answer
24 views

Interpreting contour plots for two variables

This contour plot shows the relationship between the finished square feet of a home and the tax-assessed value for homes in a County. Things I couldn't understand: what does it mean when lots of ...
1
vote
1answer
40 views

Interpreting Violin Plots

The plot shown below, shows the percent of flights delayed by each Airline at an airport. From the above violin plot, all I could understand is $25^{th}$, $50^{th}$, $75^{th}$ percentiles are shown ...
0
votes
2answers
21 views

What kind of analysis can I make in order to understand which variable impact the most my result?

Here's the thing, I have data from my products in a dataset just like this: ...
2
votes
0answers
35 views

When should we split the data into train, valid and test datasets? [duplicate]

In EDA we actually get insights about the data. That is completely fine but what i cannot understand is on which dataset should we do the EDA? Should we do the EDA on train dataset or train + valid ...
0
votes
1answer
49 views

Counting and percentages in EDA, analysis in binary classification

Let's asume we have a dataframe with 100 clients, 70 males and 30 females, where 10% of them buys the product and 90% doesn't. Case 1: 7 males and 3 females buys the product = Same distributions If ...
1
vote
2answers
45 views

Linearity assumption for Pearson correlation

It's known that values must have a linear relationship to count Pearson correlation between them. I'm wondering if there are any ...
0
votes
0answers
31 views

Best way to analyse categorical data?

I'm fairly new to data science so forgive the newb question, but: what's the best way to work with categorical data when doing exploratory data analysis? Is there a dedicated field for that? In this ...
3
votes
2answers
77 views

How do you spot errors in data?

I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
1
vote
1answer
123 views

What's the meaning of “frequency represents area of bars” in histogram?

In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers: [1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10] The ...
3
votes
2answers
61 views

Categorizing river discharge data

I'm struggling with figuring out the best way to break up my river discharge data in a way that I can use it as a factor for further analysis. I'm currently manipulating the data in R but plan to pull ...
1
vote
0answers
12 views

How do you cope with the risk of false-positives in exploratory analysis?

Let's say that I'm running exploratory analysis on a dataset. For instance, let's say that the dataset consists of several features and two groups and I want to see which features are significantly ...
2
votes
0answers
22 views

Circular confidence intervals about scatter points [closed]

I'm wondering if this is a "thing", I quite like the visual and feel it would be nice to know how much the points are likely to move from the location given by the data. The following plot ...
3
votes
0answers
114 views

Is there any consensus or literature on the proportion of font parameters within plots?

This is now about how to make a plot, though that is the information that I'm finding when trying to search for an answer to my question. What I'm wondering, is whether there are any thoughts/...
4
votes
2answers
241 views

Exploratory vs. Descriptive Statistical Analysis

Descriptive statistics definition is pretty clear to say that it summarizes data using statistical methods like mean, mode, median, and spread. However, I came across the term 'exploratory' today ...
0
votes
0answers
31 views

Assessing CFI in Exploratory factor analysis (EFA) in R?

I'm currently conducting EFAs to analyse the internal structure of a construct. I've found a nine factor model with acceptable fit, but only regarding to RMSEA and SRMR. I'm using the program R to ...
0
votes
0answers
15 views

Does Outliers in Categorical Feature varibales exist?

So, I was working on an Exploratory Data analysis project, after dealing with all the preprocessing of numerical features of the dataset when I started analyzing categorical data(nominal) there I ...
0
votes
0answers
19 views

Shapiro-Wilk normality test on paired data

I'm going to perform Shapiro-Wilk normality test before Paired Samples T-test. There are a ...
1
vote
1answer
76 views

How to find the upper outlier threshold in a right skewed distribution?

Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers? One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
0
votes
0answers
12 views

How to conduct item selection via exploratory factor analysis?

I want to reduce my item pool via exploratory factor analysis. I couldn't find any advices on the right order in doing so. What I am asking is: Which is the right order in selecting the items? Shall ...
0
votes
0answers
16 views

How to compute factor scores after second-order EFA analysis

I would like to know what are the steps to computing factor scores after a second-order EFA (1 overall factor). I am considering two ways: Ideally, I would like to compute factor scores using a ...
0
votes
1answer
16 views

Handle various shapes of distribution in data preprocessing

Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
11 views

Tests of the effects of variables on imbalanced categorical variable [duplicate]

I have survey data that include a binary response variable,say y, and various theoretically relevant independent continuous variables. y is binary, which includes two options, say Option A and Option ...
0
votes
1answer
24 views

How to carry out EDA on data consisting of 2D variables

I have a temporal dataset that consists of 2D variables (angle, length) I want to investigate: whether the data has any noticeable structure (e.g. clusters) Whether the sequence of variables (a time ...
0
votes
0answers
28 views

Checking Normality of Numerical and Categorical Data [duplicate]

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
2answers
250 views

Checking Normality of Numerical, and Categorical Data

I have come across 3 questions on the title subject. Why is it necessary to do a normality test? To check if data is imbalanced or not? Are these 4 methods of checking if the data follows normal ...
0
votes
1answer
16 views

How much can one rely on the mean or median to infer a difference?

When performing an explonatory data analysis, one tries to visualize patterns of behavior of your target variable against the variables that you're assuming could explain the target variable. For ...
0
votes
1answer
16 views

How do you interpret your features when you standardize your data?

Let's say I have built a boosting tree or neural network and I standardized my features beforehand. When I built my model, I split my data into training, validation, and test sets - each with their ...
0
votes
0answers
46 views

How do you code missing values if 0 is meaningful?

Per this deep learning book I am reading: In general, with neural networks, it’s safe to input missing values as 0, with the condition that 0 isn’t already a meaningful value. The network will ...
0
votes
1answer
23 views

At what point does a training-validation-test split need to be made?

If I have a set of data and I am not sure about the data at all or the meaning I would try to investigate the dataset a bit first. I understand the importance of having a training-validation-test ...
0
votes
0answers
13 views

Family members in factor analysis

I'm carrying out an exploratory PCA, followed by an CFA, to test for dimensionality of a questionnaire. I am wondering about the effect of having family members in the same sample. I have a final ...
1
vote
1answer
69 views

The use of Zero Value as Missing Values and Outliers

I am not sure what is the best title for this question. Sorry for that. Let's say I analyzing a dataset of prices of houses. And one of the columns (features) is ...
7
votes
1answer
669 views

Is exploratory data analysis (EDA) actually needed / useful

There are many guides prevalent on the internet about EDA and how everyone should do it and how useful it is however I rarely see it in practice and often times (in said tutorials) it sticks to very ...
0
votes
1answer
56 views

Understanding How Many Components to Include for PCA

I've conducted a PCA on survey data and I'm confused about the concept of 'choosing how many principal components to include'. When interpreting PC1, I found that it successfully captures the ...
1
vote
1answer
61 views

PCA Interpreting strong negative and positive loadings

I ran a PCA on survey data asking whether respondents trust different levels of government, where they rank their level of trust from 0-7. Whilst I can interpret PC1 as a weighted average of ...
0
votes
1answer
55 views

Correlation between heavy metals and community data

This is a preliminary study. I want to test whether the presence of heavy metals in surface water negatively affects the benthic macroinvertebrate community. If it's the case, I'll continue the ...
0
votes
0answers
55 views

QQ plot distribution check

I want to check if a distribution is log normal or not by using qq plot. So for convenience I am creating a lognormal distribution using stats and checking it in ...
1
vote
0answers
11 views

Sources on simulations of the null hypothesis [closed]

I'm looking for two types of sources. Studies on any topic in any field that simulate fake data that are supposed to represent some null distribution to test the validity of some model.I'm just ...
1
vote
0answers
25 views

Finding Patterns In a 3D Plot

I am starting to use 3D graphs to visualize relationships in my data. The reason I am doing this is to try and maximize my predictor variables. Since I am working with financial data the relationships ...
0
votes
0answers
46 views

The utter futility of bivariate analysis

Consider the following quote from here: Moreover, univariable prefiltering, sometimes also referred to as “bivariable analysis,” does not add stability to the selection process as it is based ...
0
votes
0answers
71 views

Should items with low intercorrelation be removed before EFA to avoid many items with low communalities?

I am attempting Exploratory Factor Analysis (EFA) in R, using principal axis factoring in the psych package. Background: 221 participants indicated the extent of their agreement on 61 statements, ...
0
votes
0answers
14 views

Effect of feature interaction in classification model

I have a binary classification problem. I have checked below things as EDA- 1) Distribution of features to see any outlier, linear model can be fit? 2) Box plot to see which variables will be ...
1
vote
0answers
37 views

Approaches to analyzing data from forum

Working with self written example data that mimic a chat-forum with different forums (i.e. cars, politics, sports) and different threads (i.e. "why is porsche popular?", "thoughts on andrew yang?") in ...
0
votes
0answers
6 views

Making Parallel Coordinates In Cylindrical Complete Graph Visually Tractable?

In "Addressing the unmet need for visualizing conditional random fields in biological data", a visually intractable, complete parallel coordinate graph appears: By "complete" I mean that a top-down ...
0
votes
0answers
43 views

Book recommendation for analyzing and preparing data for machine learning

I have a school project in which I need to apply several supervised learning algorithms over a data set. Is there a good book that you recommend that focuses on data preparation, like: zero values, <...
0
votes
0answers
28 views

EDA and relationship with ML?

I am starting to learn Exploratory Data Analysis for ML and I have some doubts that I hope to find some help here. First of all, I have been testing some classifiers such as Logistic Regression and ...

1
2 3 4 5