102

Disclaimer: @ttnphns is very knowledgeable about both PCA and FA, and I respect his opinion and have learned a lot from many of his great answers on the topic. However, I tend to disagree with his reply here, as well as with other (numerous) posts on this topic here on CV, not only his; or rather, I think they have limited applicability. I think that the ...


52

Not long ago, I had an interview task for a data science position. I was given a data set and asked to build a predictive model to predict a certain binary variable given the others, with a time limit of a few hours. I went through each of the variables in turn, graphing them, calculating summary statistics etc. I also calculated correlations between the ...


36

We usually know it's impossible for a variable to be exactly normally distributed... The normal distribution has infinitely long tails extending out in either direction - it is unlikely for data to lie far out in these extremes, but for a true normal distribution it has to be physically possible. For ages, a normally distributed model will predict there is ...


29

As you said, you are familiar with relevant answers; see also: So, as long as "Factor analysis..." + a couple of last paragraphs; and the bottom list here. In short, PCA is mostly a data reduction technique whereas FA is a modeling-of-latent-traits technique. Sometimes they happen to give similar results; but in your case - because you probably feel like ...


27

I do not think I will be able to give regular time investment to continue learning data analysis I don't think Casella & Berger is a place to learn data much in the way of data analysis. It's a place to learn some of the tools of statistical theory. My experience so far telling me to be a statistican one needs to bear with a lot of tedious ...


20

In my case (second plot), the notches don't meaningfully overlap. But why does the bottom of the box on the right hand side take that strange form? How do I explain that? It indicates that the 25th percentile is about 21, 75th percentile about 30.5. And the lower and upper limits of the notch are about 18 and 27. A common reason is that your ...


20

Q1 Ecologists talk of gradients all the time. There are lots of kinds of gradients, but it may be best to think of them as some combination of whatever variable(s) you want or are important for the response. So a gradient could be time, or space, or soil acidity, or nutrients, or something more complex such as a linear combination of a range of variables ...


19

This is a broad question without a simple answer. At CMU I taught a 3-month course on this topic. It covered issues such as: Using projections to understand correlation between variables and overall distributional structure. How to build up a regression model by successively modelling residuals. Determining when to add nonlinear interaction terms to a ...


18

In this my answer (a second and additional to the other of mine here) I will try to show in pictures that PCA does not restore a covariance any well (whereas it restores - maximizes - variance optimally). As in a number of my answers on PCA or Factor analysis I will turn to vector representation of variables in subject space. In this instance it is but a ...


17

Consider flipping your questions around. Begin with uncorrelated data - I generated this data randomly, so these variables are independent; my y is normal and my x is log(1+X1) where X1 is a mixture of several geometric distributions chosen to give a roughly similar appearance to your plot: The y variable is symmetric and the x-variable is mildly skew, but ...


17

I don't think either the web-page or your statements are correct. I'd rather stick with more straightforward descriptions: Inferential statistics: Given a sample, what can we say about the population from which it was drawn? Descriptive statistics: Given a sample, what can we say about the sample? Both can be used as part of inductive or deductive ...


15

Obviously, yes. The data analysis could lead you to many points that would hurt your predictive model : Incomplete data Assuming we are talking about quantitative data, you'll have to decide whether you want to ignore the column (if there's too much data missing) or figure out what will be your "default" value (Mean, Mode, Etc). You can't do this ...


12

If one views the role of EDA strictly as generating hypotheses, then no the sharpshooter fallacy does not apply. However, it is very important that subsequent confirmatory trials are indeed independent. Many researchers attempt to "reconcile differences" with things like pooled analyses, meta analyses, and Bayesian methods. This means that at least some of ...


11

To add to @Peter Flom's answer, it is worth defining the other terms that were used: Deductive reasoning: Derive conclusions or predictions about specific cases from fundamental rules or theories. Inductive reasoning: Derive universal rules or theories from observation of many cases. Inferential statistics use both inductive and deductive reasoning. ...


11

Age can not be from normal distribution. Think logically: you cannot have negative age, yet normal distribution allows for negative numbers. There are many bell-shaped distributions out there. If something looks bell-shaped it doesn't mean that it has to be normal. There is no way to know for sure anything in statistics, including from which distribution ...


11

This paints a very negative view of exploratory data analysis. While the argument is not wrong, it's really saying "what can go wrong when I use a very important tool in the wrong manner?" Accepting unadjusted p-values from EDA methods will lead to vastly inflated type I error rates. But I think Tukey would not be happy with anyone doing this. The point of ...


11

I come from a traditional biostatistics/epidemiology background, and EDA are definitely useful, although it doesn't mean doing histograms/correlation plots just for the sake of it. With the preeminence of machine learning and prediction, I do feel that it is practiced less and less often these days though. If you are in medical statistics/epidemiology, ...


10

I think that frequently, the tendency to feel like you've gone down a rabbit hole with exploratory analyses is due to losing sight of the substantive question(s) you're asking. I do it myself, occasionally, and then have to remind myself what my goal(s) are. For example, am I trying to build a specific model, or evaluate the adequacy of an existing one? Am I ...


10

I'd recommend having a look at "7.10.2 The Wrong and Right Way to Do Cross-validation" in http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf. The authors give an example in which someone does the following: Screen the predictors: find a subset of “good” predictors that show fairly strong (univariate) correlation with the class ...


9

Here's the smallest counter-example I could find : A ([1, 4, 10]) and B ([0, 6, 9]) have the same average (5) B has a larger median (6) than A (4) There's a 5/9 probability that a random A element is larger than a random B element. Here's another example with 4 elements:


8

Let me add a few points: first of all, hypothesis generation is an important part of science. And non-predictive (exploratory/descriptive) results can be published. IMHO the trouble is not per se that data exploration is used on a data set and only parts of those findings are published. The problems are not describing how much has been tried out then ...


8

The help on lm and line (?lm and ?line) explain what they are. In R, lm fits the least squares regression line. Its description says: ‘lm’ is used to fit linear models. It can be used to carry out regression, single stratum analysis of variance and analysis of covariance ... If you want "the regression line" without any other ...


8

The best thing to do is make facetted barcharts, which is easy enough to do in R, with ggplot2, or with the ggpairs function in the GGally package (for multivariate categorical data). The plot below shows survey responses to liking different subjects by different types of students. You can see that there are roughly two groups, one set tends to like GEO, BIO,...


8

This is just one of the many exploratory data analyses (EDA's) that one could do for logistic regression that may shed some insights to the problem (and hopefully this clarifies my comment above @FrankHarrell). For clarity sake, let's give an understandable example. Imagine you have data for some people and of interest is trying to predict the individual's ...


8

The interpretation depends on context, but there are some common contexts in which this comes up. The statement is often used in Bayesian analysis to stress the fact that we would ideally like the posterior distribution in the analysis to be robust to prior assumptions, so that the effect of the data "dominates" the posterior. More generally, the quote ...


8

One important thing done by EDA is finding data entry errors and other anomalous points. Another is that the distribution of variables can influence the models you try to fit.


8

We used to have a phrase in chemistry: "Two weeks spent in the lab can save you two hours on Scifinder". I'm sure the same applies to machine learning: "Two weeks spent training a neuralnet can save you 2 hours looking at the input data". These are the things I'd go through before starting any ML process. Plot out the density of every (continuous) ...


8

I'm not sure that these are sufficiently well defined anywhere to say definitively what is what in everyday conversation. I think if you look hard enough, you will be able to find something that an author or reviewer calls "descriptive" or "exploratory", but that someone else would say falls within their conception of the other. That said, the idea was ...


7

My advice, coming from the opposite perspective (Stats PhD student) is to work through a regression textbook. This seems a natural starting point for someone with a solid theoretical background without any applied experience. I know many graduate students from outside our department start in a regression course. A good one is Sanford Weisberg's Applied ...


7

I am not sure if there is a pithy title for this entire topic but it is certainly an important issue. Maybe "robust statistics" would be a good place to start? The aptly-named empirical influence function describes how an estimator (e.g., the mean or median) depends on the value of one of the points in its sample. It can also be generalized into the "...


Only top voted, non community-wiki answers of a minimum length are eligible