Questions tagged [eda]

EDA stands for "Exploratory data analysis". Developed by Tukey to contrast with Confirmatory Data Analysis or CDA (the formal testing of hypotheses). EDA is typically concerned with describing data numerically and graphically to make the data easier to understand and to yield new insights.

Filter by
Sorted by
Tagged with
4
votes
3answers
6k views

How to determine moving window size?

I am using moving window technique for data analysis... For example I compute the mean, the standard deviation and etc. for a given window. And I wonder if there's any good criterion to determine ...
4
votes
2answers
2k views

Panel-data exploratory data analysis

I have a dataset for around 40k firms over fiscal years 1950-2011 with about 430k firm-years. If I'm not mistaken I have panel-data. In addition, the firms are nested within 9 industries. I created a ...
4
votes
1answer
10k views

Panel data descriptives, plots and 'feel for the data'

I have a dataset for around 40k firms over fiscal years 1950-2011 with about 430k firm-years. If I'm not mistaken I have panel data. I created a unique identifier ...
3
votes
3answers
5k views

Any necessary EDA before logistic?

I wanted to know if we do EDA before logistic regression. Sure, I will look at the variables and their distributions, but is there anything specific to logistic?
3
votes
3answers
471 views

Which is correct way for regression line?

I have a set of data (some Frequencies per month, Var1 is the month): ...
3
votes
1answer
3k views

Quick Exploratory Analysis of Categorical Data [closed]

Does anyone know of a tool (preferably free) that does quick analysis of exploratory data mainly categorical with date. Using R and Python I can create time series and histograms, perform tests such ...
3
votes
1answer
411 views

Does the normal probability plot systematically underestimate the mean?

A normal probability plot is defined as a plot of $n$ pairs: ($[100(i-0.5)/n]$ th $z$ percentile, $i$th observation). Theoretically the points should fall close to a straight line with slope $\...
3
votes
2answers
55 views

Categorizing river discharge data

I'm struggling with figuring out the best way to break up my river discharge data in a way that I can use it as a factor for further analysis. I'm currently manipulating the data in R but plan to pull ...
3
votes
2answers
140 views

Websites to learn data analysis and visualization?

I took "business statistics" in college and survived, but I find data analysis and visualization fascinating. I have a decent grasps on the basics (probability, distributions, etc.). What are some ...
3
votes
1answer
57 views

How do you spot errors in data?

I was in an interview recently for a job where I'd been given a task relating to some employee data that had obvious errors in it. I've worked with data in jobs for years where I could just look at ...
3
votes
1answer
3k views

R scatterplot matrix with nonparametric density

I normally use MATLAB, or JMP but right now am working with R. I have ~150 dimensional data with a few hundred thousand rows. Some of the columns are non-informative, they only have one value. This ...
3
votes
1answer
4k views

Interpreting non-significant regression coefficients

Out of seven, six of the independent variables (predictors) are not significant ($p>0.05$), but their correlation values are small to moderate. Moreover, the $p$-value of the regression itself is ...
3
votes
1answer
141 views

How can I adjust classifier to the scale of the other

I have 2 classifiers with different scales. How can I adjust one classifier to the scale of the other without loss of quality? On the scatter plot we have 2 solutions plotted (x1, x2) against each ...
3
votes
1answer
113 views

Lowess optimal selection of bandwidth

I'm currently running lowess on a dataset, and am wondering what bandwidth I should use. What are some guidelines for selecting the bandwidth? Or is it largely arbitrary?
3
votes
2answers
3k views

Is it appropriate to do an ANOVA on a feature selected via inspecting PCA results?

I've been given a dataset consisting of 8 dimensional feature vectors for 4 classes of objects. I was asked to find the features that best distinguish the classes, and write up a short report. My ...
3
votes
1answer
262 views

Is there a way to explain and generalise the decision made by random forest?

I saw a similar question was asked a few years ago, maybe there are updates on that. I would like to have a way to explain the decisions generated by random forest, possibly in a single tree. I ...
3
votes
2answers
755 views

Is PCA appropriate for comparing subsets of panel data?

I have a large panel (5000+ subjects, 4 variables over 182 periods), and I've identified particular Granger-causal relationship in a large subset of those subjects (30% or so). I would like to somehow ...
3
votes
2answers
1k views

What are some differences between confirmatory analysis and exploratory analysis?

In confirmatory analysis do you basically just test hypotheses? Then in exploratory analysis you try to generate hypotheses? In general, I know that you can first do exploratory analysis to form ...
3
votes
2answers
2k views

Can I do parallel analysis with any type of exploratory factor analysis/principal component analysis?

I wish to perform parallel analysis to determine how many factors I should extract from my maximum likelihood exploratory factor analysis. I have been referred to a program that calculates the ...
3
votes
2answers
194 views

Whether to exclude a case from two analyses when the case is only an outlier in one?

I have a question regarding the elimination of participant data for two statistical analyses on the same sample. Here is a bit of background information: I collected data from one sample (n = 181 ...
3
votes
2answers
1k views

Choosing predictors in regression analysis and multicollinearity

I would like to run a linear regression analysis and I'm uncertain about including predictors. I have three predictor variables available. One is based on a lot of previous research. Therefore I am ...
3
votes
1answer
392 views

Exploratory data analysis using box plots

How should you make a box plot when the data have an outlier? Must we use the data with the outlier, or use the data without the outlier? If we use the data without the outlier, we will change the ...
3
votes
1answer
813 views

Combining Exploratory Factor Analysis with Random Forest for Binary Logistic Regression Feature Selection

For those of you familiar with Exploratory Factor Analysis (EFA) and Random Forest (RF), I have recently had an idea of combining these two methods to reduce the number of potential predictor ...
3
votes
0answers
112 views

Is there any consensus or literature on the proportion of font parameters within plots?

This is now about how to make a plot, though that is the information that I'm finding when trying to search for an answer to my question. What I'm wondering, is whether there are any thoughts/...
3
votes
0answers
69 views

What plots and summary statistics are usually used for exploratory data analysis of multivariate time series? [closed]

For multivariate cross-sectional data, tools such as scatterplot matrices, the five numbers summary, faceted boxplots and so on, allow an efficient exploration of a new data set, and can suggest how ...
2
votes
2answers
47 views

Is manual pattern recognition a valid scientific tool if followed up by statistical analysis?

Say I have a dataset of some kind, and I am looking through some graphs I have made from it. I notice a pattern, maybe a possible correlation, so I build a computer script or do some spreadsheet-magic ...
2
votes
1answer
632 views

How to interpret draftsman plot

I've conduct some draftsman plot to find the correlation between variables. And I got this kind of result. But I don't understand how to interpret the data. What is the different between the color? ...
2
votes
1answer
2k views

Minimum cumulative variance to extract in exploratory factor analysis to ensure a good fit

As a part of my exploratory factor analysis, I would like to report the cumulative variance % (eigenvalues). I wonder if there are guidelines on the minimum percentage in order to have a good model ...
2
votes
2answers
18k views

Exploratory factor analysis - promax & factor cross-loadings

I have a question regarding the best practice for dealing with cross-loadings on factors after conducting an exploratory factor analysis using a promax rotation. Just to give a bit of background ...
2
votes
1answer
17 views

Visualized Imputation of 2D Sample Space With Saturation Corresponding to Confidence?

I want to visualize a 2D sample space imputed from a 2D scatter of measurements. Something like a 2D hexbin color histogram would work if it were enhanced in the ...
2
votes
1answer
358 views

Choosing appropriate distance metric and algorithm for clustering for any given dataset

I have been looking for an answer/guidance/pointer to this question of mine for a while. After going through many (100s actually) posts and articles, I finally found this question, where this response ...
2
votes
1answer
355 views

Is it accepted to use statistical test in EDA for feature selection?

This question has been already asked Can I use statistical test in EDA for feature selection? but nobody has answered. Just posted it again as I've also gone through same situation.
2
votes
2answers
779 views

How to interpret linear regression in scatter plot

I am trying to understand the relationship between $2$ variables in $4$ different subgroups. I would like to decide if the variables are correlated in a different way depending on the subgroup. For ...
2
votes
2answers
1k views

How to assess seasonality effect influence on time series

Suppose I collected a time-series data (e.g. drug prescription on every month for 12 years). I have no reason to believe that my data is influenced by a seasonal factor (e.g. drug consumption is not ...
2
votes
3answers
2k views

Exploratory data analysis for a dataset with continuous and categorical variables

I have a data set which has DV and around 40 IVs. I want to select best variables out of the existing ones. I can use correlation, but it requires only numeric variables. I would like to see relation ...
2
votes
1answer
7k views

Tukey's Hinges: Grouping Data

I am having trouble figuring out how to group data using the inter-quartile ranges calculated with a box and whiskers plot as well as looking at Tukey's Hinges. I understand the IQR and Tukey's hinges ...
2
votes
0answers
21 views

Circular confidence intervals about scatter points [closed]

I'm wondering if this is a "thing", I quite like the visual and feel it would be nice to know how much the points are likely to move from the location given by the data. The following plot ...
2
votes
0answers
66 views

Exploratory data analysis before pefrorming Canonical Correspondence Analysis (CCA)

I want to perform CCA, but I read that, remember that observations (for example, species abundances) have to present unimodal distributions along gradients (for example, environmental variables). ...
2
votes
0answers
35 views

How do I fit a function for a plot like this

The blue dot is the (0,0) point. So there is clearly a reverse relationship between X and Y. but its not exactly linear. the slope is also deeper on the positive side of X. What kind of ...
2
votes
0answers
40 views

De-trend or normalize ordinal data across years

I have multiple years of data from students rating their courses (Unacceptable, Adequate, Good, Excellent). The data covers about 10 courses and I have data from the past 6 years. Charting the data, ...
2
votes
1answer
60 views

Seek tool for univariate exploratory analysis

Looking for a tool to assist with univariate exploratory analysis as an early step in a model building process. Capabilities I’m looking for are to recode or transform the raw predictor variable, x, ...
2
votes
0answers
143 views

time series model selection in exploratory research

I work in the field of behavioural interventions and I use dynamic models to gain better insight into (explain) the process of behaviour change and to help inform future behavioural interventions (...
2
votes
0answers
278 views

Is a higher correlation coefficient always “better” or “more appropriate”?

I have a question (reproduced below) from an exam. It seems to be presumed that the greater the (product moment) correlation coefficient, the "more appropriate" and the "better" the model. Is such a ...
2
votes
1answer
59 views

Which method can I use to pinpoint features that separates a sub-group from a group

Let's say I have a group (G) of 1000 individuals. I have complete knowledge of 200 demographic properties of each of them, from categorical (favorite drink) to numerical (age). Let's say I invite ...
2
votes
1answer
109 views

Quantitative Location Shifting Detection for Run-Sequence Plot

I am trying to complement a Run-Sequence Plot by some quantitative metric to validate that a dataset has a fixed location. Since the Run-Sequence Plot will be used in the early phases of Exploratory ...
2
votes
0answers
73 views

What to do when exploratory factor analysis results are different for complete-cases and imputed data?

I have a hundred items that I'm performing EFA on, with around 370 complete cases. Using parallel analysis to determine the number of factors to extract, EFA gave 9 factors, all of which make ...
2
votes
0answers
130 views

The (statistically unsound) exploratory study (no particular hypothesis)

It seems like a proposed study gets penalised if it doesn't also predict the direction of an effect. For instance, if I want to see the effect of a certain medication on happiness ratings, but do not ...
2
votes
1answer
702 views

Exploratory Analysis - finding the most important factor

I have a dataset of 113 variables. In exploratory analysis the first thing I want to know is what are the most important factors on a single variable (revenue). I learned that naive Bayes would work,...
1
vote
2answers
4k views

What type of data are dates?

According to Yale: Categorical variables represent types of data which may be divided into groups (Lacey M, 1997) To me, dates do not fit this definition. They are ordinal, as one date is ...
1
vote
2answers
117 views

Implications of trying multiple tests for an analysis

Let's say I have a fixed set of data containing one variable. I'm interested in whether the variable shows a difference between two conditions. Now I perform a test and determine a p-value somewhere ...