Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
236
questions
2
votes
1answer
92 views
Is GAM appropriate for these data
I am strangling to find the right modeling method for my data.
Short explain of the dataset : I have a variable called alpha diversity ( Alpha diversity refers to the average species diversity in a ...
0
votes
0answers
12 views
Handling time data
I'm trying to solve a classification problem that has records of customers of a company seeking some service. It has different attributes and one of the attributes is the time the customer has been ...
1
vote
0answers
28 views
Dominant variables and percentage of explained variance in PCA analysis
A long time i'm using PCA for exploratory data analysis and i was sure that it is Ok if the first ...
0
votes
0answers
11 views
Bayes weighted average (which imdb uses) for my datatable in R
I'm very new in R and I'm working on my first project, I have searched for hours but I can't find a detailed example in the whole internet.
I want to use Bayes weighted average (which imdb uses) for ...
0
votes
0answers
12 views
What method do you use to find rise/jump on a graph?
Backdrop..
Bucketed X axis values (of previous dataset) and found slope (linear regression) for each bucket. Below is the dict of the resulting bucketed data. Note: since I have named a variable '...
1
vote
1answer
20 views
Pilot study EFA, 50 participants, can I use this sample to explore factors structure?
I’ve run a pilot study with only 50 participants and I was wondering what I could actually do with a small sample like that, how can I best use this data before collecting the final sample of 600 ...
1
vote
1answer
34 views
Differencing Time Series
I am trying to remove the trends by differencing this logarithmically transformed time series. It contains two columns about COVID-19 Cases in the United States: one column being the number of cases ...
3
votes
1answer
45 views
Accuracy over different sample sizes from dataset
What I'm trying to do is predict how much more data would help in a classification task.
So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-...
0
votes
1answer
33 views
Analyse relationship between categorical feature vs continuous target
In a regression problem, I am trying to analyse the relationship between categorical predictors vs continuous target variable, therefore I opted for plotting with a box plot, but can not infer ...
0
votes
0answers
12 views
Is jittering variables beneficaial before building a scatter plot?
I am given a dataset with features X and Y and need to learn to classify objects into 2 classes. The corresponding targets for the objects from the dataset are denoted as y:
Top left plot shows X vs ...
1
vote
0answers
10 views
Why do Mosteller & Tukey use 1/6 to start their “flog” (logit) transformation
Mosteller & Tukey (Data Analysis and Regression) recommend transforming counted fractions with the flog transformation:
$$
\frac{\log{(n)}}{\log{(m - n)}}.
$$
...
1
vote
1answer
40 views
Exploratory factor analysis with factor evolving over the time
Drawing an analogy, my task can be phrased like this: Evaluate a person's IQ based on several tests and examine how IQ changes over time.
I do $K$ different intelligence tests for each person several ...
0
votes
0answers
9 views
What should I do after determine correlation between variables?
I was doing a correlation analysis during my machine learning project. Among all the features, I had a categorical variable, "weather" and a numerical variable, "humidity". After ...
0
votes
0answers
8 views
Help with brainstorming for data analytics portfolio
I am new to the field of data analytics and am currently brainstorming project ideas to kickstart my data analytics portfolio. The idea is to use datasets from my previous research projects as the ...
0
votes
2answers
48 views
Performing regression on a dataset with lots of categories
I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
0answers
13 views
Incorporating quantity of points in median
I have multiple datasets with a variable amount of measure points.
I want to sort these datasets after the median of their points, but also incorporate how many points were measured. Medians from just ...
0
votes
0answers
21 views
How should one transform a variable in which the further away from 0, the more significant it is?
Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
2
votes
2answers
41 views
Comparing yearly data that has different $n$ values per year?
I have been collecting soil moisture data at my research site for almost 20 years. However, I have not been consistent with the number of times that I have been taken these measurements. The problem ...
0
votes
0answers
26 views
What is the use Additive and Multiplicative decomposition
I am trying to do an Exploratory Data Analysis over the time series based data. While doing some research on it and came up with the term Additive and Multiplicative Decomposition, but i am not able ...
0
votes
0answers
7 views
Exclusion of Spuriousness With More Dimensions?
The usual way of excluding spuriousness in multivariate correlation is to increase the number of samples. However, if increasing the dimensions of measurements produces an unusual magnitude of ...
5
votes
1answer
191 views
What is the defintion of variation in a box plot?
Variation is a term I have always been confused with. The first time I heard it, it was defined as "the different between the max and min values in a distribution". Lately, I've been hearing ...
1
vote
1answer
30 views
Interpreting contour plots for two variables
This contour plot shows the relationship between the finished square feet of a home and the tax-assessed value for homes in a County.
Things I couldn't understand:
what does it mean when lots of ...
2
votes
1answer
90 views
Interpreting Violin Plots
The plot shown below, shows the percent of flights delayed by each Airline at an airport.
From the above violin plot, all I could understand is $25^{th}$, $50^{th}$, $75^{th}$ percentiles are shown ...
0
votes
2answers
22 views
What kind of analysis can I make in order to understand which variable impact the most my result?
Here's the thing, I have data from my products in a dataset just like this:
...
2
votes
0answers
40 views
When should we split the data into train, valid and test datasets? [duplicate]
In EDA we actually get insights about the data. That is completely fine but what i cannot understand is on which dataset should we do the EDA?
Should we do the EDA on train dataset or train + valid ...
0
votes
1answer
67 views
Counting and percentages in EDA, analysis in binary classification
Let's asume we have a dataframe with 100 clients, 70 males and 30 females, where 10% of them buys the product and 90% doesn't.
Case 1: 7 males and 3 females buys the product = Same distributions
If ...
2
votes
2answers
50 views
Linearity assumption for Pearson correlation
It's known that values must have a linear relationship to count Pearson correlation between them.
I'm wondering if there are any ...
0
votes
0answers
32 views
Best way to analyse categorical data?
I'm fairly new to data science so forgive the newb question, but: what's the best way to work with categorical data when doing exploratory data analysis? Is there a dedicated field for that?
In this ...
3
votes
2answers
98 views
How do you spot errors in data?
I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
1
vote
1answer
278 views
What's the meaning of “frequency represents area of bars” in histogram?
In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers:
[1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10]
The ...
3
votes
2answers
65 views
Categorizing river discharge data
I'm struggling with figuring out the best way to break up my river discharge data in a way that I can use it as a factor for further analysis. I'm currently manipulating the data in R but plan to pull ...
1
vote
0answers
16 views
How do you cope with the risk of false-positives in exploratory analysis?
Let's say that I'm running exploratory analysis on a dataset.
For instance, let's say that the dataset consists of several features and two groups and I want to see which features are significantly ...
2
votes
0answers
25 views
Circular confidence intervals about scatter points [closed]
I'm wondering if this is a "thing", I quite like the visual and feel it would be nice to know how much the points are likely to move from the location given by the data.
The following plot ...
3
votes
0answers
114 views
Is there any consensus or literature on the proportion of font parameters within plots?
This is now about how to make a plot, though that is the information that I'm finding when trying to search for an answer to my question. What I'm wondering, is whether there are any thoughts/...
4
votes
2answers
1k views
Exploratory vs. Descriptive Statistical Analysis
Descriptive statistics definition is pretty clear to say that it summarizes data using statistical methods like mean, mode, median, and spread.
However, I came across the term 'exploratory' today ...
0
votes
0answers
68 views
Assessing CFI in Exploratory factor analysis (EFA) in R?
I'm currently conducting EFAs to analyse the internal structure of a construct. I've found a nine factor model with acceptable fit, but only regarding to RMSEA and SRMR. I'm using the program R to ...
0
votes
0answers
20 views
Does Outliers in Categorical Feature varibales exist?
So, I was working on an Exploratory Data analysis project, after dealing with all the preprocessing of numerical features of the dataset when I started analyzing categorical data(nominal) there I ...
0
votes
0answers
33 views
Shapiro-Wilk normality test on paired data
I'm going to perform Shapiro-Wilk normality test before Paired Samples T-test. There are a ...
1
vote
1answer
169 views
How to find the upper outlier threshold in a right skewed distribution?
Say we have a right skewed distribution (tail on the right side) like income. Above what value can I consider data points to be outliers?
One way could be Q3 (75th percentile) + 1.5 * Interquartile ...
0
votes
0answers
18 views
How to conduct item selection via exploratory factor analysis?
I want to reduce my item pool via exploratory factor analysis. I couldn't find any advices on the right order in doing so. What I am asking is: Which is the right order in selecting the items?
Shall ...
0
votes
0answers
29 views
How to compute factor scores after second-order EFA analysis
I would like to know what are the steps to computing factor scores after a second-order EFA (1 overall factor).
I am considering two ways:
Ideally, I would like to compute factor scores using a ...
0
votes
1answer
34 views
Handle various shapes of distribution in data preprocessing
Lately i have just learned about EDA, and through the distplot i saw that the distributions of my features and target the distributions are various. I barely ...
0
votes
0answers
11 views
Tests of the effects of variables on imbalanced categorical variable [duplicate]
I have survey data that include a binary response variable,say y, and various theoretically relevant independent continuous variables. y is binary, which includes two options, say Option A and Option ...
0
votes
1answer
26 views
How to carry out EDA on data consisting of 2D variables
I have a temporal dataset that consists of 2D variables (angle, length)
I want to investigate:
whether the data has any noticeable structure (e.g. clusters)
Whether the sequence of variables (a time ...
0
votes
0answers
33 views
Checking Normality of Numerical and Categorical Data [duplicate]
I have come across 3 questions on the title subject.
Why is it necessary to do a normality test? To check if data is imbalanced or not?
Are these 4 methods of checking if the data follows normal ...
0
votes
2answers
698 views
Checking Normality of Numerical, and Categorical Data
I have come across 3 questions on the title subject.
Why is it necessary to do a normality test? To check if data is imbalanced or not?
Are these 4 methods of checking if the data follows normal ...
0
votes
1answer
17 views
How much can one rely on the mean or median to infer a difference?
When performing an explonatory data analysis, one tries to visualize patterns of behavior of your target variable against the variables that you're assuming could explain the target variable. For ...
0
votes
1answer
19 views
How do you interpret your features when you standardize your data?
Let's say I have built a boosting tree or neural network and I standardized my features beforehand. When I built my model, I split my data into training, validation, and test sets - each with their ...
0
votes
0answers
69 views
How do you code missing values if 0 is meaningful?
Per this deep learning book I am reading:
In general, with neural networks, it’s safe to input missing values as 0, with the condition that 0 isn’t already a meaningful value. The network will ...
1
vote
1answer
28 views
At what point does a training-validation-test split need to be made?
If I have a set of data and I am not sure about the data at all or the meaning I would try to investigate the dataset a bit first. I understand the importance of having a training-validation-test ...
