Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

0
votes
0answers
6 views

Trying to run a hierarchical ESEM in MPlus 6

I've been trying to run a hierarchical ESEM using MPlus 6. Running the first-order ESEM works fine (see below for that example). But when ever I try a different method to add a general factor in to ...
1
vote
0answers
15 views

Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
3
votes
1answer
86 views

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
0
votes
0answers
16 views

Conducting a Pilot Study without Large Sample

I want to obtain preliminary evidence that some experimental manipulations I would like to use in research have a good chance of being successful (i.e., successfully manipulating the intended ...
2
votes
0answers
16 views

What plots and summary statistics are usually used for exploratory data analysis of multivariate time series?

For multivariate cross-sectional data, tools such as scatterplot matrices, the five numbers summary, faceted boxplots and so on, allow an efficient exploration of a new data set, and can suggest how ...
2
votes
1answer
53 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
1
vote
1answer
85 views

Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
23
votes
5answers
2k views

Is exploratory data analysis important when doing purely predictive modeling?

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your ...
0
votes
0answers
11 views

How to find minimum set of variables maximally contributing towards AoC

While analysing dummy data for TV shows viewed by user I came up with this graph where I can clearly spot peak hours. As a next step I would like to know "set of ...
10
votes
2answers
211 views

What's intended by “Let the data speak for itself”?

In reading the following paper, I came across the following statement: As mentioned, it is often presented without any reference to probabilistic models, in line with Benzecri [1973]’s idea to “let ...
0
votes
0answers
40 views

Is it acceptable to use composite indexes as variables in factor analysis (EFA, PCA)?

Is it possible to use composite indexes(built on theoretical foundation) as variables in factor analysis? I have 16 composite indexes that group in 3 factors using EFA, PCA. My sample size seems to ...
0
votes
0answers
33 views

Exploratory analysis for multiple regression with a dummy varible

I currently have a model like this in R: weight ~ age + height + gender So for basic exploratory analysis I made scatter plots of weight vs age weight vs ...
0
votes
1answer
30 views

Measuring the level of effect a continuous variable has on a binary outcome

I'm working with auction data, and I'd like to better understand how much the bid amount effects whether or not the bid is won. I have evidence to believe that the bid amount is not a reliable ...
2
votes
1answer
131 views

How to interpret draftsman plot

I've conduct some draftsman plot to find the correlation between variables. And I got this kind of result. But I don't understand how to interpret the data. What is the different between the color? ...
0
votes
0answers
17 views

What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?

I am solving for a regression use case using tensorflow's DNNRegressor. For EDA purpose, I referred to this post and used pandas ...
0
votes
1answer
83 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
0
votes
0answers
15 views

Exploratory analysis to inform regression process

When using regression to model time series data, are there any specific exploratory analysis steps that should be applied to either the input or target variables, and if so, how do the results inform ...
1
vote
2answers
247 views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
0
votes
1answer
62 views

Feature Selection Prior to Cross Validating

After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X. I get the general idea of ...
1
vote
0answers
96 views

How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
0
votes
1answer
176 views

EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
0
votes
1answer
129 views

Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
1
vote
1answer
32 views

Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty ...
0
votes
1answer
71 views

Missing Value Treatment avoiding bias in category

I have a customer information data .. One of the variable is Income(Numeric) with missing value. I computed the mean (treated outlier first) and then replaced the missing value with the mean value of ...
9
votes
4answers
1k views

Why doesn't the fact that 1 median is lower than another median, mean that most in group 1 are less than most in group 2?

I believed that the boxplots below could be interpreted as "most men are faster than most women" (in this dataset), primarily because the median men's time was lower than the median women's time. But ...
2
votes
1answer
223 views

Is it accepted to use statistical test in EDA for feature selection?

This question has been already asked Can I use statistical test in EDA for feature selection? but nobody has answered. Just posted it again as I've also gone through same situation.
0
votes
0answers
287 views

Exploratory Data Analysis with Categorical/Dummy Variables

I'm pretty new to the data analysis and predictive modeling process and I had a quick question about EDA. For numerical variables this seems pretty straightforward to do some quick correlations and ...
0
votes
0answers
28 views

Reporting similarity between continuous variables

I have two continuous variables. They have the same shape, but there is some time shift between them. In the following plot you see a line plot:the values (y) and time (X). Which measure can I use to ...
1
vote
1answer
869 views

How to analyze correlation on high dimensional data?

I have a dataset with over 100 features from where I want to know if there is a high correlation between some of those. I'm doing: corr = features_final.corr() ...
0
votes
0answers
32 views

Exploratory Data Analysis using XLMiner

I am new to Analytics, I am trying to understand how to use XLMiner for performing exploratory data analysis. Kindly suggest me any reference to learn more about this. Since I am new to this I am ...
0
votes
0answers
14 views

How to merge two group by attribute

I have a dataset with 10 variables The last variable is a factor variable with levels 1-2-3-4-5 My prof thinks that the units that has the attribute 1 and 3 May be merged in a new factor levels ...
4
votes
1answer
404 views

Is multiple comparisons a problem for exploratory analyses?

I have been struggling with the idea of multiple comparisons for while. Correcting for multiple comparisons is easy to understand and to correct for when conducting for example an ANOVA between ...
4
votes
4answers
133 views

Does this present a linear relationship?

I have created the following graph which contains a great number of datapoints, I can see that there is a very linear scattering of the points, but am not sure since it displays a lot of datapoints ...
1
vote
1answer
77 views

Exploratory data analysis techniques to percentage variable

I'm running an analysis about a dataset with three variables: Income, number of residents per residence and percentage of income spent on rent (response variable). What descriptive statistics should I ...
0
votes
0answers
16 views

Which machine learning algorithm may help to explore the importance of contributions of variables to their sum over time?

I have data in the form of a $n \times t$ matrix $X$ where $n$ is a number of variables and $t$ a (large) number of time points. At any given time point the elements of the matrix can be expressed as ...
1
vote
0answers
78 views

Histograms for severely skewed data [duplicate]

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,...
3
votes
0answers
34 views

De-trend or normalize ordinal data across years

I have multiple years of data from students rating their courses (Unacceptable, Adequate, Good, Excellent). The data covers about 10 courses and I have data from the past 6 years. Charting the data, ...
1
vote
0answers
388 views

Can EDA be used for asking a new question?

Data Analyst gets inspiration to ask a new question about the data, so the analyst begins the EDA on the same data set and goes through the 5 phases - Question; EDA; Modeling; Interpretation and ...
3
votes
1answer
74 views

Lowess optimal selection of bandwidth

I'm currently running lowess on a dataset, and am wondering what bandwidth I should use. What are some guidelines for selecting the bandwidth? Or is it largely arbitrary?
7
votes
5answers
162 views

Preserving comments on graphs for exploratory data analysis

In performing exploratory data analysis, I will often print out the graphs and write out comments/annotations etc. Do people have suggestions for a better electronic methodology? I am especially ...
0
votes
0answers
76 views

What is the relationship between two variables that Chi-Squared Test for Independence is testing for?

I am trying to understand the application of Chi-Squared test for independence between predictor and response variables as it applies to feature selection in machine learning and exploratory data ...
2
votes
1answer
51 views

Seek tool for univariate exploratory analysis

Looking for a tool to assist with univariate exploratory analysis as an early step in a model building process. Capabilities I’m looking for are to recode or transform the raw predictor variable, x, ...
1
vote
0answers
19 views

Are there any intuitive/visual ways of detecting interactions between response variables?

Say I have a dataset with ~30 variables and I'm trying to make the best lm or glm. What's the best way to do exploratory data analysis to see if any of the variables would produce a good two-way ...
2
votes
2answers
497 views

How to interpret linear regression in scatter plot

I am trying to understand the relationship between $2$ variables in $4$ different subgroups. I would like to decide if the variables are correlated in a different way depending on the subgroup. For ...
1
vote
1answer
1k views

When to do the split training and test set

I want to split my sample into a training and a testing set in order to cross-validate my findings. However, I am not sure in which point in the process I have to do the split. Here is the order I ...
5
votes
0answers
85 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...
3
votes
2answers
123 views

Websites to learn data analysis and visualization?

I took "business statistics" in college and survived, but I find data analysis and visualization fascinating. I have a decent grasps on the basics (probability, distributions, etc.). What are some ...
0
votes
1answer
29 views

Identifying health profiles from a large datset

I have three concepts that I am trying to relate to each other. Health/Cognition/Lifestyle For each of the concepts I have over twenty variables from about 20,000 participants. Lets use memory as an ...
2
votes
0answers
97 views

time series model selection in exploratory research

I work in the field of behavioural interventions and I use dynamic models to gain better insight into (explain) the process of behaviour change and to help inform future behavioural interventions (...
28
votes
4answers
774 views

Has the journal Science endorsed the Garden of Forking Pathes Analyses?

The idea of adaptive data analysis is that you alter your plan for analyzing the data as you learn more about it. In the case of exploratory data analysis (EDA), this is generally a good idea (you are ...