Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Estimating Algorithmic Information Content With EDA

I've compiled over 6000 county-level measures from the US Census into a tsv file and need a high end estimate of its algorithmic information content. By "high end" ...
0
votes
1answer
17 views

Quantifying volatility with limited data?

I have ~400 time series's that I need to quantify volatility over. The only information I have is mean, min, and max values for each time stamp. Here is a sample series for your convenience: My ...
0
votes
1answer
28 views

Exploring a loan data set when I'm interested in change in number of loans over time, not acceptance/rejection, defaulting, etc

This is a simple question but I am having difficulty wrapping my head around it. I have a dataset listing a bunch of individual loans. There are tens of variables describing aspects of the loans, mix ...
1
vote
0answers
18 views

MCA in FactoMineR: all variables are the same on Dim 1?

My data is ordinal (1, 2, 3, 4, 5, NA from a likert scale) and doesn't have any obvious pattern looking at the raw data. My code looks like this: library(FactoMineR) df <- data.frame(Var1 ...
1
vote
0answers
74 views

How to do exploratory analysis on very large dataset?

I have very large dataset stored in Google Cloud BigQuery. (the system dumps a table everyday. A table has approximately 25GB of size, and I have 2 months data = 60 tables = 60x25 GB) It is quite ...
0
votes
0answers
9 views

Discriminating Between Spurious and Real Relationships With Algorithmic Information

The standard means of avoiding spurious relationships is to predict them. However, it seems that this can also be accomplished with sufficiently sophisticated exploratory modeling, involving, say, ...
0
votes
0answers
31 views

EDA of Time Series Data (System Logs)

My project is about exploring time series data - system logs of all desktops in a large organisation. The idea is to see whether this data can be used for: (A) diagnostics (B) anomaly detection The ...
1
vote
0answers
42 views

Data exploration methods for noisy time series

I have time series data from NanoPore sequencing (Attached is a illustrative figure and short explanation) which I'd ultimately like to use in order to find various methylation patterns in the input ...
0
votes
1answer
20 views

Creating a custom metric based on diverse feature scalse

I would like to create a custom metric that takes into account multiple features and produces a certain score. However, the issue is that I have data that has many different data types, such as ...
0
votes
0answers
11 views

How do i select the attribute from the correlation and covariance metrics?

I have a values from heatmap which is contain correlation and covarian. for the correlation values, i got with this code : ...
0
votes
0answers
7 views

After conducting an EFA, how to score a factor for new participants using its constituting variables?

I study psychology, and I'm learning EFA (again autodidactically -_- bloody middle east).. So I hope you help me with this naive question Scenario I have a questionnaire made, and I was able to ...
0
votes
0answers
25 views

Too many factors extracted from EFA!

I am constructing a scale. I have 80 items and 300 sample data. I ran EFA - Principal Axis Method of extraction and direct oblimin rotation in SPSS. I have 23 factors (using both Eigen value and ...
1
vote
0answers
57 views

How to Use Exploratory Data Analysis To Criticize Pre-Existing Hypotheses

Obviously, exploratory data analysis is a useful tool that can be used to support the hypothesis generation process in the social sciences. Especially in the age of big data. But, my coursework never ...
1
vote
0answers
46 views

Data exploration to determine if GAMs are appropriate [duplicate]

How to decide, based on data exploration and data visualisation, between a GAM and a GLM. Linear Models (LM) assume that the relationship between the response and the predictor are linear. ...
0
votes
0answers
46 views

Degrees of Freedom in 2k factorial design

I am working on a 2k factorial design experiment, and got stuck at one point. If the suppose there are only 2 factors A and B (K=2) each with only one replicate each. This means n=2. Then according ...
0
votes
0answers
35 views

Modification of Model after EFA and CFA

I have the following problem: for my master thesis I am, or rather was trying to validate a scale on relationship cognitions in a different cultural context (germany/western vs. malaysia/asian) Since ...
0
votes
0answers
33 views

Why would the mean and standard deviation of the first and second derivatives of a signal (e.g. EDA) be useful?

When analysing a signal, e.g. EDA, I intuitively understand why one would want to determine the mean and standard deviation of the signal. The mean would tell us the average value of the EDA signal, ...
1
vote
0answers
18 views

Exploratory Factor Analysis for Smoking cessation RCT?

So I'm planning a RCT where I'm evaluating a smoking cessation application intervention vs usual care among adolescent smokers. For my statistical analysis I'm looking at the point prevalence of ...
1
vote
0answers
75 views

Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
3
votes
1answer
1k views

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
0
votes
0answers
25 views

Conducting a Pilot Study without Large Sample

I want to obtain preliminary evidence that some experimental manipulations I would like to use in research have a good chance of being successful (i.e., successfully manipulating the intended ...
3
votes
0answers
39 views

What plots and summary statistics are usually used for exploratory data analysis of multivariate time series? [closed]

For multivariate cross-sectional data, tools such as scatterplot matrices, the five numbers summary, faceted boxplots and so on, allow an efficient exploration of a new data set, and can suggest how ...
2
votes
1answer
158 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
1
vote
1answer
270 views

Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
23
votes
5answers
3k views

Is exploratory data analysis important when doing purely predictive modeling?

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your ...
10
votes
2answers
602 views

What's intended by “Let the data speak for itself”?

In reading the following paper, I came across the following statement: As mentioned, it is often presented without any reference to probabilistic models, in line with Benzecri [1973]’s idea to “let ...
0
votes
0answers
93 views

Exploratory analysis for multiple regression with a dummy varible

I currently have a model like this in R: weight ~ age + height + gender So for basic exploratory analysis I made scatter plots of weight vs age weight vs ...
0
votes
1answer
32 views

Measuring the level of effect a continuous variable has on a binary outcome

I'm working with auction data, and I'd like to better understand how much the bid amount effects whether or not the bid is won. I have evidence to believe that the bid amount is not a reliable ...
2
votes
1answer
347 views

How to interpret draftsman plot

I've conduct some draftsman plot to find the correlation between variables. And I got this kind of result. But I don't understand how to interpret the data. What is the different between the color? ...
0
votes
1answer
587 views

Heywood Case in Exploratory Factor Analysis (in R)

I am running an EFA in R using the psych package and am getting a warning that there's a Heywood case with the following code: ...
0
votes
1answer
130 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
1
vote
2answers
934 views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
0
votes
1answer
67 views

Feature Selection Prior to Cross Validating

After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X. I get the general idea of ...
1
vote
0answers
151 views

How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
1
vote
1answer
479 views

EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
1
vote
1answer
219 views

Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
1
vote
1answer
35 views

Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty ...
0
votes
1answer
98 views

Missing Value Treatment avoiding bias in category

I have a customer information data .. One of the variable is Income(Numeric) with missing value. I computed the mean (treated outlier first) and then replaced the missing value with the mean value of ...
9
votes
4answers
1k views

Why doesn't the fact that 1 median is lower than another median, mean that most in group 1 are less than most in group 2?

I believed that the boxplots below could be interpreted as "most men are faster than most women" (in this dataset), primarily because the median men's time was lower than the median women's time. But ...
2
votes
1answer
306 views

Is it accepted to use statistical test in EDA for feature selection?

This question has been already asked Can I use statistical test in EDA for feature selection? but nobody has answered. Just posted it again as I've also gone through same situation.
0
votes
0answers
427 views

Exploratory Data Analysis with Categorical/Dummy Variables

I'm pretty new to the data analysis and predictive modeling process and I had a quick question about EDA. For numerical variables this seems pretty straightforward to do some quick correlations and ...
0
votes
0answers
40 views

Reporting similarity between continuous variables

I have two continuous variables. They have the same shape, but there is some time shift between them. In the following plot you see a line plot:the values (y) and time (X). Which measure can I use to ...
1
vote
1answer
2k views

How to analyze correlation on high dimensional data?

I have a dataset with over 100 features from where I want to know if there is a high correlation between some of those. I'm doing: corr = features_final.corr() ...
0
votes
0answers
41 views

Exploratory Data Analysis using XLMiner

I am new to Analytics, I am trying to understand how to use XLMiner for performing exploratory data analysis. Kindly suggest me any reference to learn more about this. Since I am new to this I am ...
0
votes
0answers
14 views

How to merge two group by attribute

I have a dataset with 10 variables The last variable is a factor variable with levels 1-2-3-4-5 My prof thinks that the units that has the attribute 1 and 3 May be merged in a new factor levels ...
4
votes
1answer
628 views

Is multiple comparisons a problem for exploratory analyses?

I have been struggling with the idea of multiple comparisons for while. Correcting for multiple comparisons is easy to understand and to correct for when conducting for example an ANOVA between ...
4
votes
4answers
139 views

Does this present a linear relationship?

I have created the following graph which contains a great number of datapoints, I can see that there is a very linear scattering of the points, but am not sure since it displays a lot of datapoints ...
1
vote
1answer
108 views

Exploratory data analysis techniques to percentage variable

I'm running an analysis about a dataset with three variables: Income, number of residents per residence and percentage of income spent on rent (response variable). What descriptive statistics should I ...
0
votes
0answers
16 views

Which machine learning algorithm may help to explore the importance of contributions of variables to their sum over time?

I have data in the form of a $n \times t$ matrix $X$ where $n$ is a number of variables and $t$ a (large) number of time points. At any given time point the elements of the matrix can be expressed as ...
1
vote
0answers
80 views

Histograms for severely skewed data [duplicate]

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,...