Questions tagged [exploratory-data-analysis]
EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.
282
questions
1
vote
0answers
18 views
How to confirm (for reviewers) that possible nuisance variables have not affected my experimental outcome?
We are undergoing review on an auditory perception experiment. We purposely used naturalistic stimuli, which differ in various acoustic features. To deal with this natural variability, we produced 8 ...
0
votes
0answers
24 views
PCA: is linearity important?
I have a 6 dimensions dataset where I want to apply PCA to remove one dimension.
I did a small analysis to check for relationships in my data and concluded that there is very low linear correlation ...
0
votes
0answers
20 views
PCA: does outlier detection make sense with low linear correlation? [duplicate]
I am experimenting PCA to detect outliers based on the reconstruction error.
What I do: I start with a 6 dimensions dataset and reduce it to 5 dimensions. Then, I reconstruct the initial dataset and ...
0
votes
0answers
12 views
Holdout and the Multiple Comparisons Problem
I apologize in advance if this is an extremely naive question.
I am attempting to analyze a large body of discretized event data (of the form "did this event occur on date X"). This data is ...
4
votes
0answers
47 views
How can orthogonal factors explain more than 100% of the variance? [duplicate]
I'm using Stata's exploratory factor analysis command ("factor"). When I ask for a rotated solution (using the varimax orthogonal method, which is the default) it gives me (aside from the ...
0
votes
0answers
34 views
Outliers for N=4
If you have 4 observations, can you have an outlier? Consider any value outside (Lower fourth-1.5(Fourth Spread), upper fourth+1.5(fourth spread)) as an outlier.
0
votes
0answers
12 views
How to conduct exploratory analysis of repeated measures?
Exploratory analysis of repeated measures--without running statistical tests (told not to because study is underpowered and exploratory...?), what is the best way to visualize and/or quantify trends?
...
1
vote
1answer
25 views
Confirmatory Factor Analysis identifying
I have been explained this technique in class. However, I did not understand some stuff.
The professor said that a model to be estimated needs to be at least identified. Identified meaning to be when ...
3
votes
1answer
45 views
Need data preparation cheatsheet / guidelines / first principles to train team members! [closed]
Really ran out of ideas and hence such a basic question to the community -
Despite the repeated emphasis on ensuring data accuracy/validity, team members just do not spend enough time on it because ...
0
votes
0answers
21 views
How to analyse this bimodal gamma distribution?
I have collected physiological data with multiple observations from 35 people, across four conditions. In planning the experiment, I had been hoping to perform inferential statistics comparing between ...
1
vote
0answers
14 views
How should I identify the most important motivations to my respondents - factor means or variance explained by factors? (detected by EFA)
I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained 62% of the ...
1
vote
1answer
32 views
Can I use construct/factor (revealed by EFA) mean scores as explanatory variables in a GLM (binomial family)?
I have 11 motivations (M1-M11) in my questionnaire for ocean swimming. After running an EFA, I detected 4 underlying latent factors (relaxation, socialization, etc.) which explained about 58% of the ...
0
votes
0answers
14 views
Efficient metrics for measuring interaction and Building features
Long time user of tree models, rediscovered the benefits of Linear models recently. Now I want to transfer as much non linearity as possible into my features so to mimic as much as possible the ...
1
vote
0answers
26 views
Many Preprocessing steps can cause data leakage, then how should we perform EDA?
For the past week, I have been constantly checking with people on this sub on how to avoid data leakage during preprocessing like feature selection and/or scaling etc here and here.
I understand most ...
0
votes
0answers
14 views
Determining the range of input feature values
I have an application (numerical simulation) in hand, for which I am generating my own data.
Initially, I wanted to generate a small subset of data and perform initial ML model training. I was ...
0
votes
1answer
59 views
On the misuse of two sample test: How to report difference in entire "populations"
I have the following understanding: Two (or more) sample tests are only required if conclusion should be drawn about a population based on observed samples (typically much smaller than the population)....
1
vote
0answers
20 views
Differences in the factor - variable relationship in EFA and component - variable relationship in PCA?
When I read about exploratory factor analyses, I saw equations showing that each variable is a linear composite of different factors - with loadings correspond to the coefficient in front of each ...
0
votes
0answers
42 views
Are there two types of Principal Component Analyses? [duplicate]
At first, I learned that PCA is a different type of analysis than factor analysis: PCA is only used for data reduction and factor analysis is used for detecting latent variables that underlie observed ...
0
votes
1answer
21 views
Exploratory Factor Analysis for Big Data: Smaller Loading Threshold and Variance Accounted For?
I saw several pieces of literature mentioned that the larger the sample size, the lower the threshold to be set for loadings to be considered significant, and the lower the variance accounted for. I ...
0
votes
0answers
20 views
Differences in Model Fit Evaluation Between Exploratory Factor Analysis and Confirmatory Factor Analysis
For the same dataset, I tried both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). For evaluating EFA model, the cumulative explanation of all latent factors adds up to ...
0
votes
0answers
26 views
What data cleaning method is best to be adopted for price=$0?
Suppose I have an AirBnb dataset. In EDA, i noticed that some room price are 0 (as shown in the example below). Is removing them a best approach since its not possible that a room price is 0? Or ...
0
votes
0answers
20 views
Methodology for exploratory analysis on dataset
This question is about finding the right methodology:
A population of cash machine is being analyzed. The variable we are measuring is the money income produced.
We are analyzing the variable over a ...
0
votes
2answers
45 views
How to deal with missing values?
I have a question regarding exploratory data analysis, specifically, how to deal with missing values. Here is a toy dataset to describe my question:
In exploring each variable (a frequency table ...
0
votes
1answer
18 views
EFA with "found" data
The textbook procedure for EFA is to start by designing the data collection, then you run the EFA to locate (we hope) the latent variables, therein.
Does that mean it is not ever valid to conduct and ...
1
vote
0answers
39 views
Exploring relation between x and y [duplicate]
While there are numerous methods in exploring relation between x and y upon receiving a new dataset. Yet it seems I can't find any conclusive guide as of the sequence of analysis, e.g. do correlation ...
0
votes
0answers
24 views
Standardization for using two similar data sets
I need your advice regarding standardization:
I have collected two similar datasets from the same population:
Error count and time for a group that was tested in person.
Error count and time for a ...
0
votes
0answers
11 views
EFA and CFA with a lot of non-random "missing data"
This may be silly question, but I am only newly learning factor analysis. I am working on a project on social media. One of the problems is of the following form: We have circulated a form about ...
4
votes
2answers
44 views
What does the "mass" of a box plot mean?
By “mass” I mean the size of the two subretangles that make up the box in a standard box plot. I thought the mass quantified the observations below and above the median but if it did so every box plot ...
3
votes
1answer
41 views
How to interpret this scatter plot?
I am simulating a population of binary stars by generating many samples of orbital parameters. I'm investigating the relationship between the inclination of the orbit and the fraction of the ...
1
vote
0answers
15 views
Exploratory factor analysis based on separated constructs before CB-SEM?
I've seen a paper where the researcher developed a modified version of the UTAUT2 theoretical model. This modification means that 5 new constructs have been added (from the literature), so the model ...
0
votes
1answer
40 views
Articles about data analysis workflow
I am a non-statistician. I have to write a non-English article about my data analysis workflow for a particular epidemiological regression analysis that I conducted.
The article will cover my workflow ...
1
vote
0answers
27 views
Comparing two participant groups on multiple likert scale response options in exploratory study
Hi I am new to CrossValidate so wanted to thank anyone in advance for contributing and looking at my questions.
I am presently reviewing data from an exploratory study where two groups of participants ...
1
vote
1answer
22 views
Categorical data treatment courses
Is there a book or online course/mooc related to the treatment of categorical data? I have seen this course as an elective in my university in the Statistics degree, and also as a core course in a ...
0
votes
0answers
6 views
What approach do I use to analyze this F&B sales data?
I work as a data analyst at a theme park (kind of like disney) but we operate only during 4-6 months in an year. I was asked to analyze the F&B sales at the different locations in the park and ...
1
vote
0answers
30 views
Applying ML to Diagnostic Analytics on Financial Report
Background:
We (a team in the IT department) are currently helping the financial team solve their problem, by automating their tasks.
The task is this: they review the financial report monthly and ...
2
votes
1answer
31 views
Analysis Course Survey Results
I have data from ~3500 students regarding their experiences taking an online class last year. The class consisted of a unified on-demand component and several sessions on Zoom with instructors. I ...
0
votes
0answers
7 views
Identify regime shift for given data
For a certain process two discrete regimes are obtained by analytical solution. The regimes are identified by a response variable ($x$). If $x = 1$, it is said to be Regime A and if $x = 2$ it is ...
0
votes
1answer
37 views
what should be the minimum age of sharks that fall in different grades
I have data on sharks. that data contains basic spec. about sharks like length, breadth, age, etc.
Sharks have assigned ratings A,B,C,D,E based upon a combination of features.
I want to estimate what ...
0
votes
1answer
28 views
Which significance test is the best for measuring the response to two survey questions are significantly different?
I have a survey where there are two questions designed in a Likert Scale (1-7). Let's say:
Q1: Your ratings for apples.
Q2: Your ratings for pears.
By counting the frequency of the likert response. I ...
1
vote
1answer
37 views
Exploratory search for interactions in linear regression with normal distr. of residuals but non-normally distr. DV
I am conducting a very exploratory analysis (I.e., few to no specific predictions about which predictors/interactions are most relevant) assessing whether ten IVs (all binary) predict one DV (...
0
votes
0answers
9 views
Structural Equation modeling (SEM) with EFA
I'm picturing the SEM as it shows. But, each variable (WFB, JS, FS, LS) has 18 items from the survey.
Question) In this case, should I reduce the dimension of a variable by using EFA? (because some of ...
0
votes
0answers
16 views
Where can I find source for learning EDA?
In the post (What is the relationship between $Y$ and $X$ in this plot?), I find it's quite interesting to know more deeply about the data rather than just choosing model. In that post, the scatter ...
0
votes
1answer
45 views
A variable that affects the dependent variable and is affected by another independent variable
I was analyzing happiness records data. One of the variables was trust in government and I hypothesized that this variable does not have an actual impact on the happiness score. I believe that this ...
0
votes
1answer
29 views
EFA: variables load on different components as hypothesis
I am doing undegrad thesis sand I stumble on some major problems when doing factors analysis:
My research topic is to compare the attitude of people towards 2 types of products (namely A and B).
My ...
0
votes
1answer
24 views
Remove dependency from a dependent variable
I am tasked with doing some exploratory data analysis on the Diamond dataset from the library(Ecdat) in R. The assignment states:...
0
votes
0answers
15 views
German electricity market: Do I use a paired or an unpaired t-Test for my data?
In Germany, Frequency Containment Reserve is auctioned daily. Participants bid the amount they wish to receive for being able to react in the event of a frequency deviation. They can bid on different ...
1
vote
0answers
20 views
EDA and Data cleaning can be done before selecting data samples from population?
I have a huge dataset from which I need to select a sample for machine learning. The data set has high NAN and noise. Hence is it good approach to do EDA, cleaning before taking samples?
0
votes
0answers
17 views
How to best visualize time interval difference and compute a single measure
I have a table like as shown below
...
0
votes
0answers
44 views
Checking the distribution of input variables with missing data imputed in exploratory data analysis
Suppose there are missing data in input variables and the missing rates are relatively high, we use some certain value to impute the missing info. When we check the distribution for input variables in ...
1
vote
1answer
33 views
Factor Analysis - multiple answers from same respondents
i am conducting an Exploratory Factor Analysis. I asked 100 subjects to rate the credibility of 4 marketing tactics. I examined 8 tactics in total and 4 tactics were allocated randomly to every ...
