Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
251
questions
0
votes
0answers
7 views
SPSS changing count totals [closed]
I am running crosstabs from a survey in SPSS - Layered 2 independent variables on one question (the dependent variable). There are 1000 responses total, but whenever I run the crosstab it returns the ...
1
vote
0answers
24 views
Applying ML to Diagnostic Analytics on Financial Report
Background:
We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks.
The task is this: they review the financial report monthly and ...
2
votes
1answer
28 views
Analysis Course Survey Results
I have data from ~3500 students regarding their experiences taking an online class last year. The class consisted of a unified on-demand component and several sessions on Zoom with instructors. I ...
0
votes
0answers
7 views
Identify regime shift for given data
For a certain process two discrete regimes are obtained by analytical solution. The regimes are identified by a response variable ($x$). If $x = 1$, it is said to be Regime A and if $x = 2$ it is ...
0
votes
1answer
37 views
what should be the minimum age of sharks that fall in different grades
I have data on sharks. that data contains basic spec. about sharks like length, breadth, age, etc.
Sharks have assigned ratings A,B,C,D,E based upon a combination of features.
I want to estimate what ...
0
votes
1answer
26 views
Which significance test is the best for measuring the response to two survey questions are significantly different?
I have a survey where there are two questions designed in a Likert Scale (1-7). Let's say:
Q1: Your ratings for apples.
Q2: Your ratings for pears.
By counting the frequency of the likert response. I ...
1
vote
1answer
20 views
Exploratory search for interactions in linear regression with normal distr. of residuals but non-normally distr. DV
I am conducting a very exploratory analysis (I.e., few to no specific predictions about which predictors/interactions are most relevant) assessing whether ten IVs (all binary) predict one DV (...
0
votes
0answers
8 views
Structural Equation modeling (SEM) with EFA
I'm picturing the SEM as it shows. But, each variable (WFB, JS, FS, LS) has 18 items from the survey.
Question) In this case, should I reduce the dimension of a variable by using EFA? (because some of ...
0
votes
0answers
13 views
Where can I find source for learning EDA?
In the post (What is the relationship between $Y$ and $X$ in this plot?), I find it's quite interesting to know more deeply about the data rather than just choosing model. In that post, the scatter ...
0
votes
1answer
39 views
A variable that affects the dependent variable and is affected by another independent variable
I was analyzing happiness records data. One of the variables was trust in government and I hypothesized that this variable does not have an actual impact on the happiness score. I believe that this ...
0
votes
1answer
20 views
EFA: variables load on different components as hypothesis
I am doing undegrad thesis sand I stumble on some major problems when doing factors analysis:
My research topic is to compare the attitude of people towards 2 types of products (namely A and B).
My ...
0
votes
1answer
18 views
Remove dependency from a dependent variable
I am tasked with doing some exploratory data analysis on the Diamond dataset from the library(Ecdat) in R. The assignment states:...
0
votes
0answers
15 views
German electricity market: Do I use a paired or an unpaired t-Test for my data?
In Germany, Frequency Containment Reserve is auctioned daily. Participants bid the amount they wish to receive for being able to react in the event of a frequency deviation. They can bid on different ...
1
vote
0answers
12 views
EDA and Data cleaning can be done before selecting data samples from population?
I have a huge dataset from which I need to select a sample for machine learning. The data set has high NAN and noise. Hence is it good approach to do EDA, cleaning before taking samples?
0
votes
0answers
16 views
How to best visualize time interval difference and compute a single measure
I have a table like as shown below
...
0
votes
0answers
22 views
Checking the distribution of input variables with missing data imputed in exploratory data analysis
Suppose there are missing data in input variables and the missing rates are relatively high, we use some certain value to impute the missing info. When we check the distribution for input variables in ...
1
vote
1answer
19 views
Factor Analysis - multiple answers from same respondents
i am conducting an Exploratory Factor Analysis. I asked 100 subjects to rate the credibility of 4 marketing tactics. I examined 8 tactics in total and 4 tactics were allocated randomly to every ...
0
votes
0answers
7 views
When does median polish fail to converge?
As mentioned here https://stats.stackexchange.com/a/46267/122459 the median polish may fail to converge under some circumstances. It is not clear to me why this can happen. I would like to get a more ...
0
votes
0answers
11 views
Various methods to determining the important features for total purchases and how a feature impacts the purchases
I have data for unique items in a store and their corresponding purchases -> 10 features of the item and number of times a corresponding item is purchased. There are about 900 of these items and ...
2
votes
1answer
155 views
Is GAM appropriate for these data
I am strangling to find the right modeling method for my data.
Short explain of the dataset : I have a variable called alpha diversity ( Alpha diversity refers to the average species diversity in a ...
0
votes
0answers
15 views
Handling time data
I'm trying to solve a classification problem that has records of customers of a company seeking some service. It has different attributes and one of the attributes is the time the customer has been ...
1
vote
0answers
32 views
Dominant variables and percentage of explained variance in PCA analysis
A long time i'm using PCA for exploratory data analysis and i was sure that it is Ok if the first ...
0
votes
0answers
13 views
Bayes weighted average (which imdb uses) for my datatable in R
I'm very new in R and I'm working on my first project, I have searched for hours but I can't find a detailed example in the whole internet.
I want to use Bayes weighted average (which imdb uses) for ...
0
votes
0answers
14 views
What method do you use to find rise/jump on a graph?
Backdrop..
Bucketed X axis values (of previous dataset) and found slope (linear regression) for each bucket. Below is the dict of the resulting bucketed data. Note: since I have named a variable '...
1
vote
1answer
52 views
Pilot study EFA, 50 participants, can I use this sample to explore factors structure?
I’ve run a pilot study with only 50 participants and I was wondering what I could actually do with a small sample like that, how can I best use this data before collecting the final sample of 600 ...
2
votes
1answer
40 views
Differencing Time Series
I am trying to remove the trends by differencing this logarithmically transformed time series. It contains two columns about COVID-19 Cases in the United States: one column being the number of cases ...
3
votes
1answer
54 views
Accuracy over different sample sizes from dataset
What I'm trying to do is predict how much more data would help in a classification task.
So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-...
0
votes
1answer
38 views
Analyse relationship between categorical feature vs continuous target
In a regression problem, I am trying to analyse the relationship between categorical predictors vs continuous target variable, therefore I opted for plotting with a box plot, but can not infer ...
0
votes
0answers
15 views
Is jittering variables beneficaial before building a scatter plot?
I am given a dataset with features X and Y and need to learn to classify objects into 2 classes. The corresponding targets for the objects from the dataset are denoted as y:
Top left plot shows X vs ...
1
vote
0answers
14 views
Why do Mosteller & Tukey use 1/6 to start their “flog” (logit) transformation
Mosteller & Tukey (Data Analysis and Regression) recommend transforming counted fractions with the flog transformation:
$$
\frac{\log{(n)}}{\log{(m - n)}}.
$$
...
1
vote
1answer
48 views
Exploratory factor analysis with factor evolving over the time
Drawing an analogy, my task can be phrased like this: Evaluate a person's IQ based on several tests and examine how IQ changes over time.
I do $K$ different intelligence tests for each person several ...
0
votes
0answers
11 views
What should I do after determine correlation between variables?
I was doing a correlation analysis during my machine learning project. Among all the features, I had a categorical variable, "weather" and a numerical variable, "humidity". After ...
0
votes
0answers
9 views
Help with brainstorming for data analytics portfolio
I am new to the field of data analytics and am currently brainstorming project ideas to kickstart my data analytics portfolio. The idea is to use datasets from my previous research projects as the ...
1
vote
0answers
31 views
Detecting Non-linearity in large data sets
I understand that the best way to test for non-linearity is to look at the residual plots. However, I have 20,000 or more points and any pattern in the residuals is not easy to spot. Are there ...
0
votes
2answers
57 views
Performing regression on a dataset with lots of categories
I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
0answers
13 views
Incorporating quantity of points in median
I have multiple datasets with a variable amount of measure points.
I want to sort these datasets after the median of their points, but also incorporate how many points were measured. Medians from just ...
0
votes
0answers
23 views
How should one transform a variable in which the further away from 0, the more significant it is?
Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
2
votes
2answers
59 views
Comparing yearly data that has different $n$ values per year?
I have been collecting soil moisture data at my research site for almost 20 years. However, I have not been consistent with the number of times that I have been taken these measurements. The problem ...
0
votes
0answers
28 views
What is the use Additive and Multiplicative decomposition
I am trying to do an Exploratory Data Analysis over the time series based data. While doing some research on it and came up with the term Additive and Multiplicative Decomposition, but i am not able ...
0
votes
0answers
7 views
Exclusion of Spuriousness With More Dimensions?
The usual way of excluding spuriousness in multivariate correlation is to increase the number of samples. However, if increasing the dimensions of measurements produces an unusual magnitude of ...
5
votes
1answer
998 views
What is the defintion of variation in a box plot?
Variation is a term I have always been confused with. The first time I heard it, it was defined as "the different between the max and min values in a distribution". Lately, I've been hearing ...
1
vote
1answer
43 views
Interpreting contour plots for two variables
This contour plot shows the relationship between the finished square feet of a home and the tax-assessed value for homes in a County.
Things I couldn't understand:
what does it mean when lots of ...
2
votes
1answer
139 views
Interpreting Violin Plots
The plot shown below, shows the percent of flights delayed by each Airline at an airport.
From the above violin plot, all I could understand is $25^{th}$, $50^{th}$, $75^{th}$ percentiles are shown ...
0
votes
2answers
25 views
What kind of analysis can I make in order to understand which variable impact the most my result?
Here's the thing, I have data from my products in a dataset just like this:
...
2
votes
0answers
46 views
When should we split the data into train, valid and test datasets? [duplicate]
In EDA we actually get insights about the data. That is completely fine but what i cannot understand is on which dataset should we do the EDA?
Should we do the EDA on train dataset or train + valid ...
0
votes
1answer
84 views
Counting and percentages in EDA, analysis in binary classification
Let's asume we have a dataframe with 100 clients, 70 males and 30 females, where 10% of them buys the product and 90% doesn't.
Case 1: 7 males and 3 females buys the product = Same distributions
If ...
2
votes
2answers
56 views
Linearity assumption for Pearson correlation
It's known that values must have a linear relationship to count Pearson correlation between them.
I'm wondering if there are any ...
0
votes
0answers
32 views
Best way to analyse categorical data?
I'm fairly new to data science so forgive the newb question, but: what's the best way to work with categorical data when doing exploratory data analysis? Is there a dedicated field for that?
In this ...
3
votes
2answers
126 views
How do you spot errors in data?
I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
1
vote
1answer
430 views
What's the meaning of “frequency represents area of bars” in histogram?
In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers:
[1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10]
The ...
