Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

0
votes
0answers
20 views

Degrees of Freedom in 2k factorial design

I am working on a 2k factorial design experiment, and got stuck at one point. If the suppose there are only 2 factors A and B (K=2) each with only one replicate each. This means n=2. Then according ...
0
votes
0answers
15 views

Modification of Model after EFA and CFA

I have the following problem: for my master thesis I am, or rather was trying to validate a scale on relationship cognitions in a different cultural context (germany/western vs. malaysia/asian) Since ...
0
votes
0answers
13 views

Why would the mean and standard deviation of the first and second derivatives of a signal (e.g. EDA) be useful?

When analysing a signal, e.g. EDA, I intuitively understand why one would want to determine the mean and standard deviation of the signal. The mean would tell us the average value of the EDA signal, ...
-1
votes
0answers
13 views

Normalisation with different kind of variables with different scales

I have a data frame with 5 variables, first three are numerical variable with different ranges i.e. first: (1-100), second: (50-500), third: (0-2000) And other 2 variables having count of some events ...
1
vote
0answers
11 views

Exploratory Factor Analysis for Smoking cessation RCT?

So I'm planning a RCT where I'm evaluating a smoking cessation application intervention vs usual care among adolescent smokers. For my statistical analysis I'm looking at the point prevalence of ...
1
vote
0answers
20 views

Correct Interpretation of copula contour plots

Going into exploratory data analysis with the intention of fitting copula models, I was looking at the famous copula and they mention here that either contour or 3D ...
3
votes
1answer
175 views

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
0
votes
0answers
18 views

Conducting a Pilot Study without Large Sample

I want to obtain preliminary evidence that some experimental manipulations I would like to use in research have a good chance of being successful (i.e., successfully manipulating the intended ...
3
votes
0answers
27 views

What plots and summary statistics are usually used for exploratory data analysis of multivariate time series?

For multivariate cross-sectional data, tools such as scatterplot matrices, the five numbers summary, faceted boxplots and so on, allow an efficient exploration of a new data set, and can suggest how ...
2
votes
1answer
77 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
1
vote
1answer
131 views

Regression algorithm for weak correlated target and features

I have a regression problem in hand. Dataset have 20 predictors and 1 target. Target is continuous and predictors are both categorical and continous. I performed a correlation test between continous ...
23
votes
5answers
3k views

Is exploratory data analysis important when doing purely predictive modeling?

When building a predictive model using machine learning techniques, what is the point of doing an exploratory data analysis (EDA)? Is it okay to jump straight to feature generation and building your ...
0
votes
0answers
11 views

How to find minimum set of variables maximally contributing towards AoC

While analysing dummy data for TV shows viewed by user I came up with this graph where I can clearly spot peak hours. As a next step I would like to know "set of ...
10
votes
2answers
264 views

What's intended by “Let the data speak for itself”?

In reading the following paper, I came across the following statement: As mentioned, it is often presented without any reference to probabilistic models, in line with Benzecri [1973]’s idea to “let ...
0
votes
0answers
49 views

Is it acceptable to use composite indexes as variables in factor analysis (EFA, PCA)?

Is it possible to use composite indexes(built on theoretical foundation) as variables in factor analysis? I have 16 composite indexes that group in 3 factors using EFA, PCA. My sample size seems to ...
0
votes
0answers
49 views

Exploratory analysis for multiple regression with a dummy varible

I currently have a model like this in R: weight ~ age + height + gender So for basic exploratory analysis I made scatter plots of weight vs age weight vs ...
0
votes
1answer
31 views

Measuring the level of effect a continuous variable has on a binary outcome

I'm working with auction data, and I'd like to better understand how much the bid amount effects whether or not the bid is won. I have evidence to believe that the bid amount is not a reliable ...
2
votes
1answer
180 views

How to interpret draftsman plot

I've conduct some draftsman plot to find the correlation between variables. And I got this kind of result. But I don't understand how to interpret the data. What is the different between the color? ...
0
votes
0answers
18 views

What data treatment/transformation should be applied if there are a lot of outliers and features lack normal distribution?

I am solving for a regression use case using tensorflow's DNNRegressor. For EDA purpose, I referred to this post and used pandas ...
0
votes
1answer
94 views

appropriateness of factor analysis using correlation matrix of mixed variable types

I am aware that questions about factor analysis with mixed variable types have already been addressed. My situation is unique: I have datasets from two samples that were administered the exact same ...
0
votes
0answers
16 views

Exploratory analysis to inform regression process

When using regression to model time series data, are there any specific exploratory analysis steps that should be applied to either the input or target variables, and if so, how do the results inform ...
1
vote
2answers
335 views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
0
votes
1answer
63 views

Feature Selection Prior to Cross Validating

After doing some testing and research, I came across a pitfall of performing feature selection and running cross validation on the results of that process for some model X. I get the general idea of ...
1
vote
0answers
114 views

How to interpret pattern in this scatter plot?

How could you interpret the clear pattern that appears in each plot, in which it seems the points are very well aligned? The dataset is Olives, from ...
1
vote
1answer
225 views

EDA, Descriptive statistics, Visual Analytics

Is there any difference between EDA, Descriptive Statistics, and Visual Analytics?
0
votes
1answer
145 views

Using PCA to perform feature selection when variability and correlation are the only selection critera?

I'm running some data exploration and have a large ish number of variables with varying degrees of correlation and covariance. I'd like to start throwing out some variables I don't "need" according ...
1
vote
1answer
32 views

Method for trying out different groupings of data

I have a linguistic dataset consisting of 389 sentences and my aim is to classify these sentences into meaningful groups. If the size of the dataset would be smaller – say, only twenty or thirty ...
0
votes
1answer
77 views

Missing Value Treatment avoiding bias in category

I have a customer information data .. One of the variable is Income(Numeric) with missing value. I computed the mean (treated outlier first) and then replaced the missing value with the mean value of ...
9
votes
4answers
1k views

Why doesn't the fact that 1 median is lower than another median, mean that most in group 1 are less than most in group 2?

I believed that the boxplots below could be interpreted as "most men are faster than most women" (in this dataset), primarily because the median men's time was lower than the median women's time. But ...
2
votes
1answer
246 views

Is it accepted to use statistical test in EDA for feature selection?

This question has been already asked Can I use statistical test in EDA for feature selection? but nobody has answered. Just posted it again as I've also gone through same situation.
0
votes
0answers
311 views

Exploratory Data Analysis with Categorical/Dummy Variables

I'm pretty new to the data analysis and predictive modeling process and I had a quick question about EDA. For numerical variables this seems pretty straightforward to do some quick correlations and ...
0
votes
0answers
30 views

Reporting similarity between continuous variables

I have two continuous variables. They have the same shape, but there is some time shift between them. In the following plot you see a line plot:the values (y) and time (X). Which measure can I use to ...
1
vote
1answer
1k views

How to analyze correlation on high dimensional data?

I have a dataset with over 100 features from where I want to know if there is a high correlation between some of those. I'm doing: corr = features_final.corr() ...
0
votes
0answers
34 views

Exploratory Data Analysis using XLMiner

I am new to Analytics, I am trying to understand how to use XLMiner for performing exploratory data analysis. Kindly suggest me any reference to learn more about this. Since I am new to this I am ...
0
votes
0answers
14 views

How to merge two group by attribute

I have a dataset with 10 variables The last variable is a factor variable with levels 1-2-3-4-5 My prof thinks that the units that has the attribute 1 and 3 May be merged in a new factor levels ...
4
votes
1answer
461 views

Is multiple comparisons a problem for exploratory analyses?

I have been struggling with the idea of multiple comparisons for while. Correcting for multiple comparisons is easy to understand and to correct for when conducting for example an ANOVA between ...
4
votes
4answers
133 views

Does this present a linear relationship?

I have created the following graph which contains a great number of datapoints, I can see that there is a very linear scattering of the points, but am not sure since it displays a lot of datapoints ...
1
vote
1answer
84 views

Exploratory data analysis techniques to percentage variable

I'm running an analysis about a dataset with three variables: Income, number of residents per residence and percentage of income spent on rent (response variable). What descriptive statistics should I ...
0
votes
0answers
16 views

Which machine learning algorithm may help to explore the importance of contributions of variables to their sum over time?

I have data in the form of a $n \times t$ matrix $X$ where $n$ is a number of variables and $t$ a (large) number of time points. At any given time point the elements of the matrix can be expressed as ...
1
vote
0answers
78 views

Histograms for severely skewed data [duplicate]

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,...
3
votes
0answers
35 views

De-trend or normalize ordinal data across years

I have multiple years of data from students rating their courses (Unacceptable, Adequate, Good, Excellent). The data covers about 10 courses and I have data from the past 6 years. Charting the data, ...
1
vote
0answers
423 views

Can EDA be used for asking a new question?

Data Analyst gets inspiration to ask a new question about the data, so the analyst begins the EDA on the same data set and goes through the 5 phases - Question; EDA; Modeling; Interpretation and ...
3
votes
1answer
79 views

Lowess optimal selection of bandwidth

I'm currently running lowess on a dataset, and am wondering what bandwidth I should use. What are some guidelines for selecting the bandwidth? Or is it largely arbitrary?
7
votes
5answers
170 views

Preserving comments on graphs for exploratory data analysis

In performing exploratory data analysis, I will often print out the graphs and write out comments/annotations etc. Do people have suggestions for a better electronic methodology? I am especially ...
0
votes
0answers
77 views

What is the relationship between two variables that Chi-Squared Test for Independence is testing for?

I am trying to understand the application of Chi-Squared test for independence between predictor and response variables as it applies to feature selection in machine learning and exploratory data ...
2
votes
1answer
53 views

Seek tool for univariate exploratory analysis

Looking for a tool to assist with univariate exploratory analysis as an early step in a model building process. Capabilities I’m looking for are to recode or transform the raw predictor variable, x, ...
1
vote
0answers
19 views

Are there any intuitive/visual ways of detecting interactions between response variables?

Say I have a dataset with ~30 variables and I'm trying to make the best lm or glm. What's the best way to do exploratory data analysis to see if any of the variables would produce a good two-way ...
2
votes
2answers
535 views

How to interpret linear regression in scatter plot

I am trying to understand the relationship between $2$ variables in $4$ different subgroups. I would like to decide if the variables are correlated in a different way depending on the subgroup. For ...
1
vote
1answer
1k views

When to do the split training and test set

I want to split my sample into a training and a testing set in order to cross-validate my findings. However, I am not sure in which point in the process I have to do the split. Here is the order I ...
5
votes
0answers
88 views

Censored logit transform for (ad hoc) exploratory data analysis

In my work I commonly have to analyze binary composition data, expressed as a fraction $f\in[0,1]$. The data $f[x]$ is spatially distributed ($x\in\mathbb{R}^n$, $n=1,2,3$), and typically comes in the ...