Questions tagged [exploratory-data-analysis]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0
votes
0answers
7 views

SPSS changing count totals [closed]

I am running crosstabs from a survey in SPSS - Layered 2 independent variables on one question (the dependent variable). There are 1000 responses total, but whenever I run the crosstab it returns the ...
1
vote
0answers
24 views

Applying ML to Diagnostic Analytics on Financial Report

Background: We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks. The task is this: they review the financial report monthly and ...
2
votes
1answer
28 views

Analysis Course Survey Results

I have data from ~3500 students regarding their experiences taking an online class last year. The class consisted of a unified on-demand component and several sessions on Zoom with instructors. I ...
0
votes
0answers
7 views

Identify regime shift for given data

For a certain process two discrete regimes are obtained by analytical solution. The regimes are identified by a response variable ($x$). If $x = 1$, it is said to be Regime A and if $x = 2$ it is ...
0
votes
1answer
37 views

what should be the minimum age of sharks that fall in different grades

I have data on sharks. that data contains basic spec. about sharks like length, breadth, age, etc. Sharks have assigned ratings A,B,C,D,E based upon a combination of features. I want to estimate what ...
0
votes
1answer
26 views

Which significance test is the best for measuring the response to two survey questions are significantly different?

I have a survey where there are two questions designed in a Likert Scale (1-7). Let's say: Q1: Your ratings for apples. Q2: Your ratings for pears. By counting the frequency of the likert response. I ...
1
vote
1answer
20 views

Exploratory search for interactions in linear regression with normal distr. of residuals but non-normally distr. DV

I am conducting a very exploratory analysis (I.e., few to no specific predictions about which predictors/interactions are most relevant) assessing whether ten IVs (all binary) predict one DV (...
0
votes
0answers
8 views

Structural Equation modeling (SEM) with EFA

I'm picturing the SEM as it shows. But, each variable (WFB, JS, FS, LS) has 18 items from the survey. Question) In this case, should I reduce the dimension of a variable by using EFA? (because some of ...
0
votes
0answers
13 views

Where can I find source for learning EDA?

In the post (What is the relationship between $Y$ and $X$ in this plot?), I find it's quite interesting to know more deeply about the data rather than just choosing model. In that post, the scatter ...
0
votes
1answer
39 views

A variable that affects the dependent variable and is affected by another independent variable

I was analyzing happiness records data. One of the variables was trust in government and I hypothesized that this variable does not have an actual impact on the happiness score. I believe that this ...
0
votes
1answer
20 views

EFA: variables load on different components as hypothesis

I am doing undegrad thesis sand I stumble on some major problems when doing factors analysis: My research topic is to compare the attitude of people towards 2 types of products (namely A and B). My ...
0
votes
1answer
18 views

Remove dependency from a dependent variable

I am tasked with doing some exploratory data analysis on the Diamond dataset from the library(Ecdat) in R. The assignment states:...
0
votes
0answers
15 views

German electricity market: Do I use a paired or an unpaired t-Test for my data?

In Germany, Frequency Containment Reserve is auctioned daily. Participants bid the amount they wish to receive for being able to react in the event of a frequency deviation. They can bid on different ...
1
vote
0answers
12 views

EDA and Data cleaning can be done before selecting data samples from population?

I have a huge dataset from which I need to select a sample for machine learning. The data set has high NAN and noise. Hence is it good approach to do EDA, cleaning before taking samples?
0
votes
0answers
16 views
0
votes
0answers
22 views

Checking the distribution of input variables with missing data imputed in exploratory data analysis

Suppose there are missing data in input variables and the missing rates are relatively high, we use some certain value to impute the missing info. When we check the distribution for input variables in ...
1
vote
1answer
19 views

Factor Analysis - multiple answers from same respondents

i am conducting an Exploratory Factor Analysis. I asked 100 subjects to rate the credibility of 4 marketing tactics. I examined 8 tactics in total and 4 tactics were allocated randomly to every ...
0
votes
0answers
7 views

When does median polish fail to converge?

As mentioned here https://stats.stackexchange.com/a/46267/122459 the median polish may fail to converge under some circumstances. It is not clear to me why this can happen. I would like to get a more ...
0
votes
0answers
11 views

Various methods to determining the important features for total purchases and how a feature impacts the purchases

I have data for unique items in a store and their corresponding purchases -> 10 features of the item and number of times a corresponding item is purchased. There are about 900 of these items and ...
2
votes
1answer
155 views

Is GAM appropriate for these data

I am strangling to find the right modeling method for my data. Short explain of the dataset : I have a variable called alpha diversity ( Alpha diversity refers to the average species diversity in a ...
0
votes
0answers
15 views

Handling time data

I'm trying to solve a classification problem that has records of customers of a company seeking some service. It has different attributes and one of the attributes is the time the customer has been ...
1
vote
0answers
32 views

Dominant variables and percentage of explained variance in PCA analysis

A long time i'm using PCA for exploratory data analysis and i was sure that it is Ok if the first ...
0
votes
0answers
13 views

Bayes weighted average (which imdb uses) for my datatable in R

I'm very new in R and I'm working on my first project, I have searched for hours but I can't find a detailed example in the whole internet. I want to use Bayes weighted average (which imdb uses) for ...
0
votes
0answers
14 views

What method do you use to find rise/jump on a graph?

Backdrop.. Bucketed X axis values (of previous dataset) and found slope (linear regression) for each bucket. Below is the dict of the resulting bucketed data. Note: since I have named a variable '...
1
vote
1answer
52 views

Pilot study EFA, 50 participants, can I use this sample to explore factors structure?

I’ve run a pilot study with only 50 participants and I was wondering what I could actually do with a small sample like that, how can I best use this data before collecting the final sample of 600 ...
2
votes
1answer
40 views

Differencing Time Series

I am trying to remove the trends by differencing this logarithmically transformed time series. It contains two columns about COVID-19 Cases in the United States: one column being the number of cases ...
3
votes
1answer
54 views

Accuracy over different sample sizes from dataset

What I'm trying to do is predict how much more data would help in a classification task. So, what I'm doing is bootstrapping entries in my dataset to get a sample, with a specified size. Then, I fine-...
0
votes
1answer
38 views

Analyse relationship between categorical feature vs continuous target

In a regression problem, I am trying to analyse the relationship between categorical predictors vs continuous target variable, therefore I opted for plotting with a box plot, but can not infer ...
0
votes
0answers
15 views

Is jittering variables beneficaial before building a scatter plot?

I am given a dataset with features X and Y and need to learn to classify objects into 2 classes. The corresponding targets for the objects from the dataset are denoted as y: Top left plot shows X vs ...
1
vote
0answers
14 views

Why do Mosteller & Tukey use 1/6 to start their “flog” (logit) transformation

Mosteller & Tukey (Data Analysis and Regression) recommend transforming counted fractions with the flog transformation: $$ \frac{\log{(n)}}{\log{(m - n)}}. $$ ...
1
vote
1answer
48 views

Exploratory factor analysis with factor evolving over the time

Drawing an analogy, my task can be phrased like this: Evaluate a person's IQ based on several tests and examine how IQ changes over time. I do $K$ different intelligence tests for each person several ...
0
votes
0answers
11 views

What should I do after determine correlation between variables?

I was doing a correlation analysis during my machine learning project. Among all the features, I had a categorical variable, "weather" and a numerical variable, "humidity". After ...
0
votes
0answers
9 views

Help with brainstorming for data analytics portfolio

I am new to the field of data analytics and am currently brainstorming project ideas to kickstart my data analytics portfolio. The idea is to use datasets from my previous research projects as the ...
1
vote
0answers
31 views

Detecting Non-linearity in large data sets

I understand that the best way to test for non-linearity is to look at the residual plots. However, I have 20,000 or more points and any pattern in the residuals is not easy to spot. Are there ...
0
votes
2answers
57 views

Performing regression on a dataset with lots of categories

I am trying to work on a price prediction model, the attributes have lots of categories and all these categories are coded as integers. I am assuming if I build a regression model on this, the model ...
0
votes
0answers
13 views

Incorporating quantity of points in median

I have multiple datasets with a variable amount of measure points. I want to sort these datasets after the median of their points, but also incorporate how many points were measured. Medians from just ...
0
votes
0answers
23 views

How should one transform a variable in which the further away from 0, the more significant it is?

Let's say you have a variable that ranges from -inf to +inf. The further you get away from 0, the more effect you think it has on the response. I am doing a logistic regression by the way. And ...
2
votes
2answers
59 views

Comparing yearly data that has different $n$ values per year?

I have been collecting soil moisture data at my research site for almost 20 years. However, I have not been consistent with the number of times that I have been taken these measurements. The problem ...
0
votes
0answers
28 views

What is the use Additive and Multiplicative decomposition

I am trying to do an Exploratory Data Analysis over the time series based data. While doing some research on it and came up with the term Additive and Multiplicative Decomposition, but i am not able ...
0
votes
0answers
7 views

Exclusion of Spuriousness With More Dimensions?

The usual way of excluding spuriousness in multivariate correlation is to increase the number of samples. However, if increasing the dimensions of measurements produces an unusual magnitude of ...
5
votes
1answer
998 views

What is the defintion of variation in a box plot?

Variation is a term I have always been confused with. The first time I heard it, it was defined as "the different between the max and min values in a distribution". Lately, I've been hearing ...
1
vote
1answer
43 views

Interpreting contour plots for two variables

This contour plot shows the relationship between the finished square feet of a home and the tax-assessed value for homes in a County. Things I couldn't understand: what does it mean when lots of ...
2
votes
1answer
139 views

Interpreting Violin Plots

The plot shown below, shows the percent of flights delayed by each Airline at an airport. From the above violin plot, all I could understand is $25^{th}$, $50^{th}$, $75^{th}$ percentiles are shown ...
0
votes
2answers
25 views

What kind of analysis can I make in order to understand which variable impact the most my result?

Here's the thing, I have data from my products in a dataset just like this: ...
2
votes
0answers
46 views

When should we split the data into train, valid and test datasets? [duplicate]

In EDA we actually get insights about the data. That is completely fine but what i cannot understand is on which dataset should we do the EDA? Should we do the EDA on train dataset or train + valid ...
0
votes
1answer
84 views

Counting and percentages in EDA, analysis in binary classification

Let's asume we have a dataframe with 100 clients, 70 males and 30 females, where 10% of them buys the product and 90% doesn't. Case 1: 7 males and 3 females buys the product = Same distributions If ...
2
votes
2answers
56 views

Linearity assumption for Pearson correlation

It's known that values must have a linear relationship to count Pearson correlation between them. I'm wondering if there are any ...
0
votes
0answers
32 views

Best way to analyse categorical data?

I'm fairly new to data science so forgive the newb question, but: what's the best way to work with categorical data when doing exploratory data analysis? Is there a dedicated field for that? In this ...
3
votes
2answers
126 views

How do you spot errors in data?

I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
1
vote
1answer
430 views

What's the meaning of “frequency represents area of bars” in histogram?

In some statistics lessons, I have heard that frequency represents area of bars. So I was curious and plotted these numbers: [1, 2, 1, 3, 3, 4, 5, 1, 4, 6, 7, 3, 7, 5, 7, 2, 8, 9, 10, 8, 10] The ...

1
2 3 4 5 6