The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
0
votes
2answers
38 views

How exactly does knowing a variable's probability distribution help you when learning about data?

I am an elementary/wanna-be statistician/data scientist from South Korea. I have been studying a variety of theories of mathematical statistics and different probability distributions. (I apologize ...
1
vote
1answer
19 views

PCA explained variance that increases linearly

I'm wondering what it says about the data when, instead of dropping off dramatically the variance explained by the number of components continues increasing in a reasonably linear manner. For ...
0
votes
1answer
29 views

Time series plot with more granularity at present time [closed]

How can i plot realtime timeseries data in a exponential interval.. I mean older data with less granular time scale and present data at higher granular time scale? Any inputs plz
2
votes
1answer
12 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
0
votes
1answer
19 views

Can someone give a concrete example of exploitation in the context of Exploratory Data Analysis?

This post says Exploratory Data Analysis (EDA) consists of 2 steps exploration and exploitation. I know a little about exploration which uses some techniques such as data visualization to understand ...
0
votes
0answers
15 views

Is it possible to compute Kaiser-Meyer Olkin (KMO) for all scale items and how?

In another post, what are the assumptions of factor analysis, someone mentioned that it is possible to compute Kaiser-Meyer Olkin (KMO) not just for the whole correlation matrix but for each scale ...
1
vote
1answer
56 views

Should exploratory data analysis include validation set?

I know that EDA should be performed on the training set but not on the test set. But my question is: we usually split the training set into training and validation datasets. Should we perform EDA on ...
0
votes
0answers
12 views

Selecting method for adding a random component to regression estimates during expectation-maximization (goal: EFA)

Edited to update: I now see that I cannot use the 'Regression' option in SPSS to add error variance to data imputed via EM, which was an assumption I made in my original question. However, I am still ...
2
votes
0answers
54 views

Exploratory data analysis before pefrorming Canonical Correspondence Analysis (CCA)

I want to perform CCA, but I read that, remember that observations (for example, species abundances) have to present unimodal distributions along gradients (for example, environmental variables). ...
2
votes
2answers
43 views

Is manual pattern recognition a valid scientific tool if followed up by statistical analysis?

Say I have a dataset of some kind, and I am looking through some graphs I have made from it. I notice a pattern, maybe a possible correlation, so I build a computer script or do some spreadsheet-magic ...
1
vote
0answers
30 views

Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...
0
votes
1answer
18 views

Quantifying volatility with limited data?

I have ~400 time series's that I need to quantify volatility over. The only information I have is mean, min, and max values for each time stamp. Here is a sample series for your convenience: My ...
0
votes
1answer
31 views

Exploring a loan data set when I'm interested in change in number of loans over time, not acceptance/rejection, defaulting, etc

This is a simple question but I am having difficulty wrapping my head around it. I have a dataset listing a bunch of individual loans. There are tens of variables describing aspects of the loans, mix ...
1
vote
0answers
33 views

MCA in FactoMineR: all variables are the same on Dim 1?

My data is ordinal (1, 2, 3, 4, 5, NA from a likert scale) and doesn't have any obvious pattern looking at the raw data. My code looks like this: library(FactoMineR) df <- data.frame(Var1 ...
1
vote
0answers
123 views

How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
0
votes
0answers
14 views

Discriminating Between Spurious and Real Relationships With Algorithmic Information

The standard means of avoiding spurious relationships is to predict them. However, it seems that this can also be accomplished with sufficiently sophisticated exploratory modeling, involving, say, ...
0
votes
0answers
78 views

EDA of Time Series Data (System Logs)

My project is about exploring time series data - system logs of all desktops in a large organisation. The idea is to see whether this data can be used for: (A) diagnostics (B) anomaly detection The ...
1
vote
0answers
45 views

Data exploration methods for noisy time series

I have time series data from NanoPore sequencing (Attached is a illustrative figure and short explanation) which I'd ultimately like to use in order to find various methylation patterns in the input ...
0
votes
1answer
21 views

Creating a custom metric based on diverse feature scalse

I would like to create a custom metric that takes into account multiple features and produces a certain score. However, the issue is that I have data that has many different data types, such as ...
0
votes
0answers
13 views

How do i select the attribute from the correlation and covariance metrics?

I have a values from heatmap which is contain correlation and covarian. for the correlation values, i got with this code : ...
0
votes
0answers
8 views

After conducting an EFA, how to score a factor for new participants using its constituting variables?

I study psychology, and I'm learning EFA (again autodidactically -_- bloody middle east).. So I hope you help me with this naive question Scenario I have a questionnaire made, and I was able to ...
0
votes
0answers
30 views

Too many factors extracted from EFA!

I am constructing a scale. I have 80 items and 300 sample data. I ran EFA - Principal Axis Method of extraction and direct oblimin rotation in SPSS. I have 23 factors (using both Eigen value and ...
1
vote
0answers
76 views

How to Use Exploratory Data Analysis To Criticize Pre-Existing Hypotheses

Obviously, exploratory data analysis is a useful tool that can be used to support the hypothesis generation process in the social sciences. Especially in the age of big data. But, my coursework never ...
1
vote
0answers
53 views

Data exploration to determine if GAMs are appropriate [duplicate]

How to decide, based on data exploration and data visualisation, between a GAM and a GLM. Linear Models (LM) assume that the relationship between the response and the predictor are linear. ...
0
votes
0answers
53 views

Degrees of Freedom in 2k factorial design

I am working on a 2k factorial design experiment, and got stuck at one point. If the suppose there are only 2 factors A and B (K=2) each with only one replicate each. This means n=2. Then according ...
0
votes
0answers
44 views

Modification of Model after EFA and CFA

I have the following problem: for my master thesis I am, or rather was trying to validate a scale on relationship cognitions in a different cultural context (germany/western vs. malaysia/asian) Since ...
0
votes
0answers
48 views

Why would the mean and standard deviation of the first and second derivatives of a signal (e.g. EDA) be useful?

When analysing a signal, e.g. EDA, I intuitively understand why one would want to determine the mean and standard deviation of the signal. The mean would tell us the average value of the EDA signal, ...
1
vote
0answers
18 views

Exploratory Factor Analysis for Smoking cessation RCT?

So I'm planning a RCT where I'm evaluating a smoking cessation application intervention vs usual care among adolescent smokers. For my statistical analysis I'm looking at the point prevalence of ...
1
vote
0answers
86 views

Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
3
votes
1answer
2k views

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
0
votes
0answers
25 views

Conducting a Pilot Study without Large Sample

I want to obtain preliminary evidence that some experimental manipulations I would like to use in research have a good chance of being successful (i.e., successfully manipulating the intended ...
3
votes
0answers
43 views

What plots and summary statistics are usually used for exploratory data analysis of multivariate time series? [closed]

For multivariate cross-sectional data, tools such as scatterplot matrices, the five numbers summary, faceted boxplots and so on, allow an efficient exploration of a new data set, and can suggest how ...
2
votes
1answer
200 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
1
vote
1answer
309 views

Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
23
votes
5answers
3k views

Is exploratory data analysis important when doing purely predictive modeling?

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your ...
10
votes
2answers
787 views

What's intended by “Let the data speak for itself”?

In reading the following paper, I came across the following statement: As mentioned, it is often presented without any reference to probabilistic models, in line with Benzecri [1973]’s idea to “let ...
0
votes
0answers
101 views

Exploratory analysis for multiple regression with a dummy varible

I currently have a model like this in R: weight ~ age + height + gender So for basic exploratory analysis I made scatter plots of weight vs age weight vs ...
0
votes
1answer
32 views

Measuring the level of effect a continuous variable has on a binary outcome

I'm working with auction data, and I'd like to better understand how much the bid amount effects whether or not the bid is won. I have evidence to believe that the bid amount is not a reliable ...
2
votes
1answer
421 views

How to interpret draftsman plot

I've conduct some draftsman plot to find the correlation between variables. And I got this kind of result. But I don't understand how to interpret the data. What is the different between the color? ...
0
votes
1answer
698 views

Heywood Case in Exploratory Factor Analysis (in R)

I am running an EFA in R using the psych package and am getting a warning that there's a Heywood case with the following code: ...
0
votes
1answer
153 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
1
vote
2answers
1k views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
0
votes
1answer
68 views

Feature Selection Prior to Cross Validating

After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X. I get the general idea of ...
1
vote
0answers
174 views

How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
1
vote
1answer
587 views

EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
1
vote
1answer
260 views

Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
1
vote
1answer
36 views

Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty ...
0
votes
1answer
110 views

Missing Value Treatment avoiding bias in category

I have a customer information data .. One of the variable is Income(Numeric) with missing value. I computed the mean (treated outlier first) and then replaced the missing value with the mean value of ...
9
votes
4answers
1k views

Why doesn't the fact that 1 median is lower than another median, mean that most in group 1 are less than most in group 2?

I believed that the boxplots below could be interpreted as "most men are faster than most women" (in this dataset), primarily because the median men's time was lower than the median women's time. But ...
2
votes
1answer
324 views

Is it accepted to use statistical test in EDA for feature selection?

This question has been already asked Can I use statistical test in EDA for feature selection? but nobody has answered. Just posted it again as I've also gone through same situation.